I certainly agree with what you said.
Someone else mentioned "common law" in the sense that it was actually the law of nature and God himself, and the law was "Do not harm other humans or cause loss to them." And right away, despite the fact that it sounded like a nice law on the surface, I felt like it was riddled with severe weaknesses.
It's the sort of concept where if I was to seriously question it, I'd have to dismiss it because it doesn't make sense. For one, it's like you said, where I'm probably having a better life at the expense of someone else in some form, somewhere down the chain, and for two, why are only humans given protection of this law and not animals, plants, and the rest of nature?
It's the sort of law that isn't actually a law, but a mere plea. A wish. But it doesn't correlate to reality, and because of this, I cannot consider it to be valid. So I ignore it and continue living my life using functional, practical ethics and ideas.
I have to live each day, and I won't do it with guilt. None whatsoever. But I also do not serve evil or malice. I just do as I do and I do my best.