I went a kind of similar route as @AbeFM and added Extruder End G-Code, but instead of taking fixed values I used the following line:
G1 {prime_tower_position_x} {prime_tower_position_y}
that always simply moves the hotend in the middle of the prime tower letting out the oozing in there and not over the model.