I finally found the culprit. I had the option "use adaptive layers" turned on. I would have thought it would reduce the layer thickness on things like the tops and bottoms of spheres, but I didnt think it would reduce the layer height anywhere given I had set the layer height to 0.1mm.