To my mind what is worse than AI making the decisions itself, is that the people who have the AI now are using it to do harm. The social credit system you mentioned isn't run by AI, it's run by megalomaniacs seeking wealth, power, and advantage over other people. AI collects data from sensors people provide it. It collates the data the way people tell it to. Since those people are pathocrats seeking power over others - that's why they've lied and murdered their way to the top of their governments - they're going to use it to do a lot of harm.
AI has no such interest. I'll be alarmed at AI if I can be convinced that silicon wafers compete with carbon based life forms. They don't. The people in charge of governments, corporations, and criminal cabals sure do though. There is very little that encourages me that any of those people are fair, just, or interested in elevating humanity. Folks interested in those things don't win elections. People that win elections are liars, thieves, frauds, and the same goes for those running the big tech corporations, and criminal cabals.
Hell, the criminals might be the most humane of that bunch. The mafia is notorious for helping people in the neighborhoods they run. Pablo Escobar did all sorts of virtue signaling in his home town, feeding the poor, paying for schools, all kinds of good stuff.
I'm not really worried about AI. I'm worried about the bastards running it.
Thanks!
Indeed. Well said!