It seems to me that in this example, the pastor was actually the adulterer as he should not have remarried (or dated) until he had proof that his ex-wife had cheated.
That is correct.
There would be a valid position that she had forsaken God and abandoned her husband. These would be considered the acts of an unbeliever, permitting divorce.
She wasn't an unbeliever as far as I can tell. She was saved and once you are saved you cannot lose your salvation. I hope I haven't opened up a salvation can of worms with that one?
This link will take you an explanation of Matt 19, 1 Cor 7 and 1 Cor 5 - these verses are all there and the pastor gives what I think is a very sound explanation.
I'd encourage anyone that is interested to take a look at it.