Yes, for sure, upvotes on a comment doesn't mean comment quality. Further brainstorming about what things will more reliably indicate comment quality... maybe an upvote from the post author. But what if the post author upvotes comments out of politeness? That might happen less with authors having more education about the algorithm. This might work, although it certainly seems like a weak spot.
But what if both an upvote from the post author and upvotes from others are considered in a combined fashion? So that only one of the two wouldn't hold much weight, but if both are present, then it holds larger weight? Maybe this will work more reliably. And also upvote weight can be considered - is it a 100% upvote or a 5% upvote? Getting upvotes with a high weight from both the post author and others seems more reliable to me. What do you think?