It identifies not as you as the person, but as the phone number (MDN) along with a keypair. There isn't a concept of personages, just MDNs. The MDN is tied to a publickey, and the private key is on the device. Thus if the SMS is redirected, then the initial setup handshake would just fail. But you still need to communicate ownership of the MDN out of band with your contacts.
This is why signal checks your contact list and automatically begins to share public keys. IMHO this is the biggest vulnerability with signal. It should ask you for each contact if you do or do not want to go encrypted with that particular contact and it should allow you to reject. Otherwise anyone who has you in their contact list will know you are on signal. This is more of an OPSEC vulnerability than a real issue though.
One other place where signal looks to be failing is it looks like it is in fact public key per MDN instead of a unique key per contact pair. Dang it, you got me deep diving their code base now :D
Guess I'll make a post and maybe compare signal vs silence.