Is there such a big difference between 5MS and 1ms (144hz)?

I wanted to ask if it makes such a difference if I play fortnite, there's an extra charge of 30-40 euro if I get the monitor with 1Ms. Does it make a big difference and would it be worth it?

Yes. 4ms response times in gaming are like light years apart.

At the beginning I had a 1ms monitor and now a 4ms monitor, it makes absolutely no difference to me.

Also for games like fortnite where every single second counts (if you play the game at all)?

I've never touched Fortnite in my life, but CSGO is a LEM myself.