Is it selfish for a mother to take care of her own health and well-being so that she can do so for her child? Is it selfish for a fireman to spend hours in the gym working out so he or she is better able to perform the job of saving lives and property? (No is the answer).
You're not selfish by taking some time to read, to retreat from society to feel more settled, centred or at peace. You're not selfish to say no to people because what they ask of you doesn't align with your ethics, morals or time frames. It's not selfish to have your own thoughts and opinions even though they may not align with the general consensus. These things and more are human. We shouldn't be made to feel guilty of selfish for them.