I guess this may matter more to someone with millions of dollars in their account. Bad people may see those users as an easy target. Imagine all somebody evil has to do is get your keys and then they get full access to your account and can take every token without anybody being able to stop them. This will be unlikely for most, but there are always going to be scammers. It would be a lot more difficult to find someone who is anonymous than it would be to find somebody who shows their face and says where they live. After that, all it would take is some torture and most would give up their passwords.
This is all hypothetical of course and probably wouldn't happen until far into the future. I have watched way too many movies and get wild ideas like this in my head. May never be a problem, but I can't help but consider the possibility.
What you are thinking is not crazy and it is normal. I'm sure it happens in real life too. That's why it might even be better if people with the rich accounts you mentioned keep themselves private. Nobody wants to face torture etc.