The fixed rules on author vs. curation rewards always seemed like a mistake to me. As your thought experiment shows there will be a subjective balancing point where the hassle and effort of developing a voting scheme will outweigh its benefits.
But it's subjective and each SP holder will have his or her own tipping point.
So it seems to me that there should be a market in curation. Let each SP hodler determine how much curation value his or her vote is worth. Ideally there would be some sort of feedback loop where the more generous you are with your curation the more it is worth - possibly in visibility terms.
Part of the difficulty with steem is how multi-dimensional it is. Optimizing on one parameter will necessarily de-optimize another.