Ah, yes. I know exactly what you mean.
This reminds me of something I came across in my martial arts/metaphysical training. I learned about remote viewing and became intrigued, researching at length. I was gung ho- until I was warned to protect my body while I sent my minds eye "out" as there are many who tried without the protective measures and were "inhabited" if you will by others who had lost their own bodies while traveling in this manner. This chilled me to the bone and I never tried RV.
I am in agreement with you on this, there is no telling what you will open yourself to, or open up that has long been buried within. If it has been buried for generations, there is no way you have the ability or techniques to control, or possibly even understand it...