This calls for a discussion of what constitutes harm and what is only hurt. In many things, we have so blurred the distinction between harm and hurt that many natural, honorable thoughts and feelings are suspicious as coming from a deep well of innate badness. While this is true for all of us - all of us have been conditioned to find fault with ourselves from the earliest age - men have been especially battered with the ram of self hate. And womens' hate of men has been fueled, I believe undeservedly and counter-productively.
Harm to me is bodily injury, or injury to property. Hurt is a feeling of un-comfortability.
The men I should have been afraid of never openly expressed their appreciation of me. It was the quiet ones who could do harm. There is an assumption that open appreciation of beauty leads to abuse, but my experience has not born that out.
When I was younger, if a man (or woman because that happened often too) expressed appreciation I would feel complimented, not afraid. At some point this changed for me, but not because I had been harmed by a cat caller. More because I was told that cat callers were potentially dangerous, the gateway to abuse, as you pretty much said.
So this is just one instance of how we have been made to hate ourselves so much that someone appreciating our beauty is hated by us.
There's so much more to say here, but I gots to go. Will come back later. Thanks for your comment!! I really mean that!