Finally had some time to play with it some more and the behavior is even weirder than I thought. I tried the traffic cone again, so never more than one color per layer. It would start with red PLA at the bottom, prime tower first, then the actual layer. Layer 2, same story. Right up to the point where the first white layer begins. That's when it would first deposit red filament on the prime tower, then swap tools, then print the next layer. It's not purging the nozzle at the critical time when it should...
And even just that wouldn't be an issue, that would just mean a single pink layer between a red and a white one, would probably not even be noticeable. But it keeps doing this: it will print that "white" (pink) layer, swap to red, deposit a layer on the purge tower, swap back to white and print the next layer...in muddy pink.
I have no idea what's going on here but this seems to very much defeat the entire purpose of the prime tower.