Then why do we have such a problem when it comes to not harming others?
Because even if a principle is simple to understand, it doesn't mean that people understand it, or want to, or have tried. It takes wanting to learn. This isn't taught in schools...