This issue has been bugging me for a while. Like you said, it's your wallet, so you should have some control over it.
I don't normally post links to my posts in comments, but I wrote an article on this a month ago. If you'd like to check it out, I'd be curious to hear your thoughts. If you'd rather not have the link in your comments section, let me know and I'll remove it.
https://steemit.com/steemit/@themanwithnoname/wallet-fix-stop-the-scams-spam