Your every word is worth gold! You are so right that there are prejudices that have formed over the years... but we can't completely neglect our experience that was determined by the circumstances or places where we lived.
Also, you are saying a lot of truth about being careful with strangers. Next time I will probably think twice whether I would do the same I did or as you say, call the police or ambulance, so they see the situation better.
Just stay safe, one life, it is precious, always think of your own kith and kin above all else. No, it is not selfish, never feel selfish.