Yes, the most they could do is vote for you and post things.
Your active key is required to move money.
Honestly, I wouldn't worry about a posting key, unless they did access the account.
The learning curve is steep for hive, they likely have no clue what they have, IF they have your key, nor how to use it.
But, you should change your recovery account for the off chance that somebody did find your master password/owner key.