To me at least, my individual liberty stops when something I do infringes upon yours. There will always be rules against harm to others and their property, and I should also keep my word (contracts).
Differences in opinion should never be the deciding factor. The deciding factor should be whether someone or their property has been harmed or a contract they agreed to has been broken.
",,,,do whatever you would like as long as ALL parties involved and impacted are consenting" Seems we are fairly close to same ideas. The non-consent is the breaking of the contract or "word".
Yes, this gets into voluntaryism too. Are all parties consenting and voluntary or not? That's the key to something being ethical or not.