Self-government is all about contracts and rules. If there is no contract or at least way of ruling, there is minimum to expect in terms of being able to dissent.
If others want to "discriminate" and not hang out with you or anyone else, they should have the option not to and to bare the consequenses of personal responsibility. The same applies to you. That's freedom of association 101 and you can't have free speech without it.