I did some further investigations and I have to add some corrections:
The X-axis never hit the end stop (the belt pulley), but went very close. It appeared like was hitting the pulley when homing though.
My fan holders were slightly bent, not much though. Nothing close to the photo previously posted, I doubt it could have made the fan sit more than a millimeter closer to the wall when homing.
I was testing it a bit now and fan is still extremely close to the wall when homing, even though the limit switch tab is bent and the fan holder looks like it is supposed to.
If I move the head to the home position by hand, fan screws will hit the wall just a millimeter after the X-switch is clicking.
If the tab of the limit switch was not bent a bit, the fan would go extremely close to the wall, probably hitting it.
The paper squeezed between the fan screws and the wall in the following photo is 0,9mm thick in total.
I put it there at the exact position when the X-switch is clicking.
Here is my X-switch with it's slightly bent tab:
EDIT: Mixed up X and Y-axis