No, it didn't. I had the brim printed in PVA, but the first layer of the model itself was still PLA.
I am not certain that the PVA is what is causing the imperfections. I tried to print an arch with PVA support on one side and no support on the other and both sides came out perfectly fine. If anything, it looks like the "no-support"-side has a similar roughness artifact on the non-overhanging side:
That being said, I tried to print my complex model again and it came out worse. This time, instead of just having the PLA "bumps", it also had holes of similar size and shape. I feel like this is where PVA was inside the printed geometry, similar to what I imagine JohnInOttawa was talking about.
I will keep experimenting, but it would be super helpful if someone else has come across this previously