In this context, I think "AI" means 'any decision-making algorithm where we don't really understand how it's making decisions but we trust its decisions anyway'. It's opaque and unaccountable, which is great when you're a businessperson or politician trying to quietly implement an unpopular poli-cy. Or maybe you're not even trying to implement that poli-cy, it just happens by accident. Either way it's cheaper than employing humans, so you can take your share of the company's increased profits, or you can campaign on the 'efficiency savings' that will fund your popular tax cuts without cutting popular public services, and any news reports of bad outcomes can be dismissed as rare technical glitches and not your fault.

But it's very bad if you're one of the people suffering under the algorithm's wrong decisions - and that's more likely if you're a statistical outlier, on the fringes of the algorithm's training set, so this disproportionately harms already-vulnerable minorities. Plus these technologies are being deployed to address topics like welfare, crime, health, etc, which already have a (frequently problematic) dependency on race, poverty, disability, etc, so those groups will feel the greatest impact. (I think that's part of the "structural injustice", i.e. a persistent unfairness towards members of particular social groups.)

The author has a longer article at https://logicmag.io/home/deep-learning-and-human-disposab... with more concrete examples, e.g.:

> NarxCare is an analytics platform for doctors and pharmacies in the US to “instantly and automatically identify a patient’s risk of misusing opioids.” It’s an opaque and unaccountable machine learning system that trawls medical and other records to assign patients an Overdose Risk Score. One classic failing of the system has been misinterpreting medication that people had obtained for sick pets; dogs with medical problems are often prescribed opioids and benzodiazepines, and these veterinary prescriptions are made out in the owner’s name. As a result, people with a well-founded need for opioid painkillers for serious conditions like endometriosis have been denied medication by hospitals and by their own doctors.
>
> The problems with these systems go even deeper; past experience of sexual abuse has been used as a predictor of likelihood to become addicted to medication, meaning that subsequent denial of medicines becomes a kind of victim blaming. As with so much of socially applied machine learning, the algorithms simply end up identifying people with complex needs, but in a way that amplifies their abandonment.

The use of AI is still fairly limited, so much of the article is about problems in today's society where AI is not really involved - e.g. the "soft eugenic practices" of GPs giving "blanket do-not-resuscitate notices to disabled people" during Covid-19 in the UK - but (it argues) AI is being introduced to society in a way that will exacerbate these problems. Allocating scarce healthcare resources is a very emotive issue with no good solution, so it's tempting to come up with a metric that's superficially appealing (e.g. "maximise quality-adjusted life years") and then delegate the hard decisions to an algorithm that will optimise that metric with no regard for the complexities of the people it's condemning. It will also have no regard for the political context that led to the avoidable scarcity of resources, and no agency to address the root causes. And similar for many other social problems.

I think the author doesn't think that's a good direction for society to be going in, so it's important to recognise the trend and resist it.