Yes men matter and their opinions, feelings, and personal experiences (negative or positive) definitely deserve to be a part of the discussion on paternity, maintenance law, and the kind.
The issue is that if there was legal recourse for women who choose to abort a baby, when it is not the potential (or proven) father's wish, stopping her from doing so would be a forced pregnancy. Isn't that more serious? Nothing can make a woman have a healthy pregnancy, or stop her from a 'back alley' abortion if it was somehow possible to bar her from obtaining a professional one. Or drinking or using drugs while pregnant, anything, really.
I don't know what you have in mind by legal recourse for men in those situations though. What should be done? It is unenforceable, anyway; a woman doesn't have to even tell anyone she is pregnant and can go get an abortion.
A lot of the subjugation of women in the past was based on the desire to control pregnancy (and by extension soldiers and populations) It's part of why women (and male supporters) had to fight for the rights for women.
Some things can never equate between the genders and they just end up with false equivalences being made. Men can't have children without a woman's complete consent...that is good.