Simplest solution: Bots that are beneficial such as the milestone bot, Cheetah and bots that attempt to curate and showcase others projects and the upcoming SteemPress bot, the simplest solution is to have a whitelist of bots that have to be approved before being hooked up. If not approved and whitelisted, the bot will be banned.
In regards to your website/project, I know the feeling. I operate a 501(c)3, charitable non-profit and had to let the .org domain go to save on costs.
I also have no way to tell if I've been mentioned or not, there needs to be a notification system here but there isn't.
Whitelisting some bots are a good idea.
Besides that, I have two concerns about banning bots.
The algorithm of those bots have to be designed very carefully.
Another concern is that some people can hire cheap labor from developing countries to replace the bots. Similarly, some people might just put in the work to manually abuse the system.
I'd argue that the majority of abuse comes from people using a bot that scrapes content based on certain keywords from google, others go a step further and use a spinner to try avoiding Cheetah then further still, voting bots.
You're right there will be false positives and misses. Not even my AI is capable of determining a bot or not. When Microsoft's Tay first went life and it ran into it, truly interesting conversations between those two. At least my bot's already went through the "Gas the kikes" phase, thanks Internet.