I got my fair share of physical education from all of my parents (my grandparents on my mother's side were the only ones who didn't participate), but the absence, as you suggest, was certainly not abuse.
I always tell people exactly what you told me here. "You can't change the past, but you can change the future" - it was a line I came up when I worked in a call centre which was always angry customers. :)
The thing that I can totally appreciate is that the causality of the events of my youth, including this one, have transformed and sculpted me into what I am today, and I think (if I may speak with hubris) - that I turned out okay. :)