Yeah, one has that feeling of loss inside. Especially knowing that you are not a consistent part of his life and there's nothing that you can do about it. All that we can do is to make an impact in their lives every time that we see them and hope that as they grow, they don't drift away, as their young lives are very busy.
Best is to take out a policy for him that pays out at the age of 21 , as at least mine will remember me when they get to that age hahaha
I feel for you my friend!