Everyone searches for “AI”
Today, when people look for information about automated decisions, they type “AI”. Public debate, research funding and new laws all follow this word. This is a problem for anyone who documents what automated systems have actually done to people. Several of the most harmful cases of the past decade did not use artificial intelligence in any technical sense.
The clearest example is Robodebt in Australia. The system took each welfare recipient’s yearly income from the tax office, divided it by 26, and compared the result with the income the person had reported for each two-week period. Every difference became a debt. Between 2016 and 2019 about 433,000 people received such debts, and A$751 million was wrongly taken from 381,000 of them. A Royal Commission later found that the scheme had been unlawful from the start (Royal Commission into the Robodebt Scheme 2023). The technical core of the system was one division and one comparison.
What the EU AI Act covers
The EU Artificial Intelligence Act defines an AI system as a machine-based system that “infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions” (Regulation (EU) 2024/1689, Article 3(1)). Recital 12 of the Act adds that this definition does not cover “systems that are based on the rules defined solely by natural persons to automatically execute operations”. In other words, if people wrote all the rules, it is not AI under this law.
The European Commission’s guidelines from February 2025 go further. They list types of systems that are outside the definition even if they make some kind of inference: basic data processing, systems based on classical heuristics, simple prediction systems, and some optimisation methods (European Commission 2025).
There are good reasons for this line. A law that covered every spreadsheet would be impossible to enforce. Learning systems also bring their own risks, such as opacity and unpredictable behaviour, and these need their own rules. But the result is clear. If a system like Robodebt were built in an EU country today, it would most likely fall outside the AI Act. The Act’s obligations for high-risk public benefit systems, such as risk management, human oversight and the right to an explanation, would not apply.
Harm does not depend on the technology
The first series of my work documents six cases. Two of them are clearly not machine learning. One is Robodebt. The other is the risk score of the French family benefits fund, a statistical model (logistic regression) with about forty variables, whose code the fund has now published. One case is clearly machine learning: the commercial healthcare algorithm studied by Obermeyer and colleagues (2019). It used past healthcare costs to estimate medical need, and so it ranked Black patients as healthier than white patients who were equally sick. The remaining cases are harder to classify.
I score the harm in each case on seven dimensions, and I never add the scores together. The results do not follow the technology. The machine-learning case is serious for health and dignity. Robodebt shows harm on all seven dimensions, and its health record includes deaths that families linked to the scheme in front of the Royal Commission.
The severe cases have something else in common. In each of them, the output of the system was treated as a finding, not as a lead to check. Robodebt reversed the burden of proof: the state did not prove a debt; people had to prove they did not owe it. In the Dutch childcare benefits scandal, families flagged as a risk had to pay back years of benefits at once. In both cases, the damage came from the administrative process around the system, and from the decision to act on its output without independent checks. Philip Alston (2019), the UN Special Rapporteur on extreme poverty, described the same pattern in many countries in his report on the “digital welfare state”: automation was introduced to save money, inside systems that already treated poor people with suspicion.
When nobody knows what the system is
For two of the six cases, the public record cannot say for sure whether the system was “AI”.
SyRI was a Dutch risk-scoring system that the District Court of The Hague stopped in 2020. It combined data from six public bodies and flagged addresses based on risk indicators that were never published. The court ruled that because the model and its indicators were secret, nobody could check the system: not the court, and not the people it scored (Rechtbank Den Haag 2020). Since then, some experts have described SyRI as a rule-based matching system and others as a machine-learning model. Amnesty International (2021) wrote that the risk model used in the Dutch childcare benefits scandal included a self-learning element. Official investigations focused on another point: that it used nationality as a risk factor. In my records of these two cases, I note the classification that the evidence supports, and how uncertain it is.
This uncertainty is itself an important finding. If a law depends on the technique, then the operator of a secret system decides first whether the law applies. The people being scored cannot see the model. Courts and regulators often learned what kind of system it was only years later, or never.
The older rule worked better
European data protection law has had a rule that does not depend on technology since 1995. Today it is Article 22 of the GDPR. It gives people the right not to be subject to a decision “based solely on automated processing” that has legal or similarly significant effects on them. It does not ask how the system works.
In December 2023 the Court of Justice of the EU applied this rule to SCHUFA, the German credit agency that scores about 68 million people. The Court decided that when a lender relies heavily on a score to make a decision, creating the score is itself an automated decision under Article 22 (Court of Justice 2023, C-634/21). The ruling did not depend on whether SCHUFA’s model is statistical or machine learning. It would also apply to a simple fixed formula. It is the clearest legal success in my series, and it came from the older law.
This comparison has limits. Article 22 protects individuals, one decision at a time. It helps less against a system that harms a whole population through many small decisions. The SyRI case, where people did not even know they were being scored, was decided on the right to private life, not on Article 22. The AI Act, on the other hand, regulates systems before they are used. The two laws are meant to work together. My point is narrower: the law that asks what a decision does has reached cases that a law asking how a system works would not reach.
Which word to use
None of this means we should stop saying “AI”. It is the word people use, and a public record that avoids it will not be found. But inside the record, the difference should stay visible. I label every case with the type of system, as far as the evidence allows: machine learning, statistical model, or fixed rules.
This matters for two reasons. First, it shows that the harms we now discuss under the name “AI” are older than the technology. Second, it shows who is responsible. A rule written by a civil servant has an author in a way that a learned model does not. And the Robodebt record shows that even when the author is known, it can take years and a Royal Commission to hold anyone to account.
References
- Alston, P. (2019). Report of the Special Rapporteur on extreme poverty and human rights (digital welfare state). UN General Assembly, A/74/493, 11 October 2019.
- Amnesty International (2021). Xenophobic Machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal. EUR 35/4686/2021.
- Court of Justice of the European Union (2023). Case C-634/21, OQ v Land Hessen (SCHUFA Holding), judgment of 7 December 2023.
- European Commission (2025). Guidelines on the definition of an artificial intelligence system established by Regulation (EU) 2024/1689. February 2025.
- Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
- Rechtbank Den Haag (2020). Judgment of 5 February 2020, ECLI:NL:RBDHA:2020:865 (English translation ECLI:NL:RBDHA:2020:1878).
- Regulation (EU) 2016/679 (General Data Protection Regulation), Article 22.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 3(1) and Recital 12.
- Royal Commission into the Robodebt Scheme (2023). Report. Canberra, 7 July 2023.
Text: CC BY-NC-ND 4.0. Corrections to contact@andriizupko.com.