Hey, @samysamy.
I think it's different for fathers than for mothers. In my case, I'd love the silence and the ability to just think and rest and do whatever. My wife, though, thinks things are too empty and lonely like you said.
Fortunately, for her, and, surprisingly, for me, we have my youngest son, his wife and their first child with us, so there's always something going on. I manage to spend most of the time in my home office, so I can stay away from the hubbub or I can come right into the middle of it. The granddaughter is just too much fun not to be around for too long.
I think that as your children grow and go through their different phases to adulthood, you will see things a little differently. That might seem like a lifetime away, but it does go rather quickly, especially if you don't take some time alone and maybe slow life down a little. Do whatever you can to enjoy all of the moments, together, and the fleeting times alone.