The intention does matter. If you do something where you want to hurt someone, vs. you do something but accidentally hurt someone, there is a difference ;) But hurting someone is what matters. Hypothetical: I drove and had an accident and harmed another, vs. I drove and purposefully hit someone with my car.
Most actions aren't dealing with immoral wrongs, but with incorrectness or optimized behavior to achieve a goal. Some are preferences as well. I tend to speak in generalities ;)