It is about biological fact.
The rate of trans and non-binary people is higher in people with various other conditions which affect hormones or brain development.
People with Polycystic ovary syndrome are more likely to be trans than people without out and polycystic ovary syndrome involves a biological female getting higher than normal levels of androgens / male hormones.
Neurodiversity tends to correlate with higher proportions of trans, non-binary and gender non-conforming individuals. Ableist people tend to assume it is because autistic people are poor autistic people who don't understand gender, but they are wrong (and very ableist). It is because neurodiverse brains are different structurally to a neurotypical brain and sometimes this difference also translates to being trans, non-binary or gender non-conforming.
Even without any conditions like the ones I mentioned above, a trans persons brain is typically closer to the brain of their gender than their biological sex ie a transman's brain is closer to a typical male brain (their gender) than a female brain (their sex).
Sex and gender used to be synonymous, not because gender identity (what people mean when they say gender in terms of Male / Female / Non-binary) didn't exist or because trans people didn't exist (people didn't get sex changes for the fun of it prior to the definition change), but because scientists weren't really studying gender identity. Once they started studying it, they used the relatively redundant (since sex had the same definition) word, gender.
People often think being trans is about gender expression (ie what you wear and like etc) or gender stereotypes (men do this and women do that) but it is about gender identity which is the internal sense of gender. They may inform each other in that someone's gender identity might influence how they express that gender (ie a woman wearing a dress) but they also might not (a cis "tomboy" would be someone whose gender expression does not reflect their gender identity). Gender expression is where gender non-conforming fits in, unlike trans and non binary which relate to gender identity.
Gender and sex are different things but they are both biological fact. The things that make people cis or trans or non-binary are biologically driven.
Also with your point one there, no ability to get hormones, puberty blockers or surgery would not make gender irrelevant. People would still have gender dysphoria even if they can't do anything about it and the consequences of that are far worse than the consequences of our current situation. People can have dysphoria without knowing about treatment or even the concept of trans or non-binary people.
It's biological and it's science.