Great read!
One day, we would actually get to a point where self-driven cars are quite common and perhaps are the normal thing. Technology moves fast nowadays.
But I think what will forestall the issues you raised is to develop the transport system and network as a whole to improve the safety of self-driven cars. It's not just about working on technology in the cars to make them well-suited enough. We would have to bring technological advancements to the roads too. Perhaps we could create networks on the road and recognition points for cars such that they can safely run on them.
For instance, a train would always keep to its rails because the rails were built for it.
In a similar manner, a transport network with features built for the smooth running of self-driven cars would be the best option for its success.
Your breakdown actually makes sense. I guess the complication would be that the switch between regular cars and self-driven cars won't be instant. But your point about trains stands, maybe the way cars move should also be rethought.