I find LISTENING really difficult with Japanese because of the monotonous rhythm and because there are a limited amount of sounds so every word ends up sounding similar. Was it Ko? To? Mo? Ho? Do? Go? I can read novels but I've yet to fully understand a television show without subtitles. Conversation is easier because people adjust themselves to the individual.
You make a great point. I believe everyone should be exposed to at least 2 languages and cultures growing up. It really helps us to see the world more objectively.