Lint on boiled linseed oil?

I have put on four coats of boiled linseed oil on my m1 stock. I used lint free rags to wipe off the excess and i noticed that when i put the stock in the sun, it shows a lot of lint on it and its really bothering me. Should i be worried?