@rw3iss
In my own experience I noticed that sometimes Cura will allow the skirt to be printed in material 1 before switching to material 2 to print the object. I suggest switching the model to layer view and checking to see what layers are actually being printed from extruder 1 and extruder 2.
In my case I always stopped the printer when I saw the skirt/brim being printed in the non-desired material thus it took a few hours to realize what was happening.
In Addition, it is possible to right click the Extruder 1 in the right side toolbar and disable it, same goes for extruder 2.
Hope this helps!
Current Release: Cura 3.3.1