Andrii ZupkoDocumenting AI and automated decisions
← Essays

Recognition is not repair: what happens after an automated system is stopped

A court ruling or an inquiry usually ends the news story about a harmful system. For the people affected, it is often only the middle.

The white line on the chart

In each of my spectrograms, the moment when someone with authority steps in is drawn as a white vertical line. It can be a court ruling, a parliamentary inquiry or a regulator’s decision. In public memory, this line is usually where the story ends. The ruling is in the news, the system is stopped, and people move on.

My method treats this line differently. Everything after it is scored again, on the same seven dimensions of harm, and drawn with a dotted texture. This makes it impossible to read the period after the ruling as “back to normal”. The rule exists to prevent one specific mistake: confusing the recognition of harm with its repair. When you follow six cases past their white lines, you see how often this mistake is made.

A checklist from human rights law

International law has a clear vocabulary for repair. In 2005 the UN General Assembly adopted the Basic Principles and Guidelines on the Right to a Remedy and Reparation. They list five forms of repair:

  • restitution: putting things back as they were before the harm;
  • compensation: money for damage that can be measured;
  • rehabilitation: medical, psychological, legal and social support;
  • satisfaction: stopping the harm, telling the truth publicly, apologising, and sanctions for those responsible;
  • guarantees of non-repetition: changes that make sure it does not happen again.

These principles were written for gross violations of human rights. None of the cases below is in that category, so I use the list as a comparison, not as a legal standard. It is useful because it breaks the single word “remedy” into five separate questions, and each case can be checked against each one.

Six cases after the line

SyRI, the Dutch fraud-scoring system, was stopped across the country by the District Court of The Hague on 5 February 2020. This is the first part of satisfaction: the harm stopped. But the people who were scored were never told, so none of them could ask for a personal remedy. The data the system produced stayed in government databases after the ruling. A related fraud list kept by the tax authority was later found unlawful by the Dutch Data Protection Authority. The recognition was complete in law, but there was almost no repair.

The Dutch childcare benefits scandal received the most recognition of any case in the series. A parliamentary inquiry called it Ongekend Onrecht (“Unprecedented Injustice”) in December 2020. The government resigned in January 2021. In May 2022 it admitted institutional racism inside the tax administration. Compensation followed, starting with €30,000 for each parent recognised as harmed. By 30 June 2026 about 69,700 parents had registered with the repair programme, and 44,045 had been confirmed as harmed. Parts of the programme continue into 2027.

But restitution was impossible where it mattered most. Statistics Netherlands counted 2,090 children from affected families who were placed outside their homes between 2015 and mid-2022. At the last count, 645 of them were still outside the home. The statistics office notes that not every placement was caused by the scandal. Still, no payment can give back those years. On non-repetition: a second parliamentary inquiry in January 2025 found similar failures in fraud policy across the Dutch government (Parlementaire enquêtecommissie Fraudebeleid en Dienstverlening 2025).

Robodebt in Australia was paused in November 2019. In 2020 the government agreed to refund about A$721 million, and in 2021 the Federal Court approved a class-action settlement worth about A$1.8 billion. The Royal Commission in 2023 told the truth about the system in rare detail. Sanctions came last and only in part. In March 2026 the National Anti-Corruption Commission found that two former senior public servants had engaged in serious corrupt conduct, but neither was sent for prosecution. Some families told the Royal Commission that they linked a relative’s death to the scheme. For them, no form of repair is possible.

Optum Impact Pro, the healthcare algorithm studied in Science in 2019, was corrected together with the researchers in 2020. The new version reduced the measured bias by about 84% on a test dataset. But nobody has publicly confirmed that the correction was used in the actual product. No patient was ever told. Without knowing they were affected, patients could not ask for anything. Here, the harm itself made it impossible to have victims in a legal sense.

The French family benefits risk score has not reached its white line yet. In October 2024, fifteen organisations asked France’s highest administrative court, the Conseil d’État, to cancel the system for indirect discrimination. In January 2026, ten more organisations joined. The system is still in use. This record shows what the time before recognition looks like when nobody knows when it will end.

SCHUFA, the German credit agency, was the subject of a 2023 ruling by the Court of Justice of the EU. The ruling placed credit scores under the GDPR rules for automated decisions. In March 2026 SCHUFA introduced a new score based on twelve published criteria. Consumer groups welcomed it, but noted that only about a quarter of SCHUFA’s business partners used it at launch. The ruling changed the law in 27 countries. How much it changed the scores people actually receive is still unknown.

The pattern

When you compare the six cases with the five forms of repair, the same picture appears.

Stopping the harm is the most common form. But it usually comes late. In Robodebt, a tribunal ruled the method unlawful from March 2017, two and a half years before the scheme was paused. In the Dutch case, the ombudsman warned in 2017.

Compensation comes when there is a clear group of victims and political pressure to pay them. Where people were never told, as in SyRI and Optum, there is no group and no compensation.

Restitution is impossible for the most serious harms in the series: children not returned, health lost, deaths. In the spectrograms, these are the points marked in red, as harm that cannot be undone.

Rehabilitation is the least documented form in every case. Some repair programmes include social and psychological support, but the public record says little about what it achieved.

Guarantees of non-repetition fail most clearly. The second Dutch inquiry found the same kind of failure in other parts of the government four years after the first. The ruling against SyRI did not stop the data it had produced from staying in the system.

Why the second texture matters

These findings explain a decision in the artworks. After each white line, the spectrogram does not go back to black. It continues, with a dotted texture, at the level of harm that the record still shows. Sometimes it is a little lower. Sometimes it does not change. In the SyRI spectrogram, dignity falls from 8 to 4 after the ruling. Reputation only falls from 7 to 6, because the records that labelled people stayed after the system was stopped.

A record like this can also stay open. Each case shows when it was scored and when it was last checked. It is scored again when a court, a commission or a statistics office adds a new finding. Robodebt is the reason for this rule. Its most important finding about responsibility came almost seven years after the scheme was paused, and almost three years after the Royal Commission reported.

References

  • Court of Justice of the European Union (2023). Case C-634/21, OQ v Land Hessen (SCHUFA Holding), judgment of 7 December 2023.
  • Federal Court of Australia (2021). Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, 11 June 2021.
  • Ministry of Finance, Netherlands (2026). 23rd progress report on the benefits repair programme, reference date 30 June 2026, 18 September 2026.
  • National Anti-Corruption Commission (2026). Operation Myrtleford, investigation report. Canberra.
  • Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
  • Parlementaire ondervragingscommissie Kinderopvangtoeslag (2020). Ongekend Onrecht. The Hague, 17 December 2020.
  • Parlementaire enquêtecommissie Fraudebeleid en Dienstverlening (2025). Blind voor mens en recht. The Hague, 23 January 2025.
  • Rechtbank Den Haag (2020). Judgment of 5 February 2020, ECLI:NL:RBDHA:2020:865.
  • Royal Commission into the Robodebt Scheme (2023). Report. Canberra, 7 July 2023.
  • Statistics Netherlands (CBS) (2022). Children of parents affected by the childcare-benefits affair placed outside the home.
  • UN General Assembly (2005). Basic Principles and Guidelines on the Right to a Remedy and Reparation for Victims of Gross Violations of International Human Rights Law and Serious Violations of International Humanitarian Law. A/RES/60/147, 16 December 2005.

Text: CC BY-NC-ND 4.0. Corrections to contact@andriizupko.com.