4 is an awesome age.
Both gymnastics and swimming are great because they do the entire body (there's a surprising number of sports that miss the core). So is dance while I think about it. If you go far enough with gymnastics and dance it is pretty hard on your body so there is also that to consider. If you're doing anything that's hard on muscles like that I recommend balancing it out with tai chi or yoga (which does the ligaments and tendons, the sports I'm familiar with don't develop them along with the muscles).
Whatever decision you make doesn't have to be for life ;D try out a few things and see what you both like if you're not sure :)
Thank you, really. For all the advices. It's helpful. I'll keep it in mind, for sure. I was thinking to start something in spring, maybe summer. And swimming was our first thought because I want him to learn how to swimm. So its kind of 2 in 1 thing. The neccesity of doing a sport and swimming lessons. π