Because when something happens to me, I use I
Using you instead of I implies not taking responsability for his feelings, as if it was a generic person.
And why does he get to use the "it's not natural" argument and when he does it it's fine? Since you are so clever @desmoind you can probably explain that to me