A big problem with "computer say no" technology is that nobody understands the implications of false positive rates.
Sainsbury’s has paused the use of AI face scanning in one of its stores after a customer was wrongly identified as a shoplifter and ejected from the shop.
...
A Sainsbury’s spokesperson said: “We have contacted Mr Arnold to apologise for his experience at our Dulwich superstore. The incident was caused by human error, not the facial recognition technology. Customers can be reassured that the Facewatch system has a 99.98% accuracy rate, and every match is reviewed by a trained manager.”
It isn't even clear what "99.98% accuracy rate" means here. I suspect it is a very made up number, so I am not going to use some ADVANCED MATH, which I might if I had any idea what it was supposed to mean.
I will interpret it in the most generous (to them) way possible: 9998 times out of 10000 when it identifies a match with a "known shoplifter" it is correct. This means 2 times out of every 10000 the alarm goes off for a wrong person.
According to their website, this (fake) number includes "human expertise" which presumably means the "trained manager."
99.98% accuracy, thanks to our combination of distinct specialised algorithms and human expertise
So 9998 times out every 10000 sounds pretty good, right! This is a major supermarket chain. It has 1500 stores (I don't know if facewatch is in all of them, but let's assume it is). Of the system dings just once per week per store, you're going to harass a wrong person about every 6 weeks.
But if the system only dings once per week per store, it's hard to imagine any calculation such that it's worth it. If, instead, it dings once per day per store (on average), you're going to harass a wrong person once per week. That's potentially 4 bad news stories per month.
This is leaving aside just how people get on the "known shoplifter" list in the first place.

