A question put to the room
At the Defense Tech Summit in Gijón on 1 October 2026, Major General (retd.) Juan A. Moliner, first vice-president of Spain’s Academy of Military Sciences and Arts (Academia de las Ciencias y las Artes Militares), spoke about the ethics of autonomous systems. Before turning to weapons, he asked about justice and medicine. His remarks are quoted here in my translation from Spanish.
“Are we going to let artificial intelligence decide the verdict of a court? Imagine a life sentence, for example. And I will go further: are we going to let it decide, in healthcare, what is wrong with me and which treatment I should receive?”
He then gave the strongest version of the opposing view: a system that feels no anger and no fear may judge more consistently than a person does. He did not answer the question. He left it with the audience.
This essay takes up that question from a different direction. Since spring 2026 I have been documenting cases in which governments and companies let automated systems decide things about people: who is investigated for fraud, who receives extra medical care, who owes the state money. None of these systems carried a weapon. Several of them used no machine learning at all. Yet the vocabulary the general used for lethal autonomous weapons describes what went wrong in them with unusual precision.
Three positions for the human
He then described the framework that military and academic writing has used for several years. A human can be in the loop, on the loop, or out of it.
“Human in the loop is when a person controls the system at every stage of its operation: from the moment they press the switch, through its sensors, its guidance and control, and the weapons it may fire.”
On the loop, the system runs its whole decision cycle by itself once it has been launched, but a person can still cancel it up to the last moment. His example was a system that identifies a figure in a field as a soldier. The operator sees that the figure is a farmer ploughing his land, a non-combatant protected by international humanitarian law, and switches the system off. Out of the loop, nothing can be done once the system is running: it observes, detects, decides and fires.
The aim that this framework serves is usually called meaningful human control. The phrase was introduced into the diplomatic debate by the British organisation Article 36 in 2013, and later given a philosophical account by Santoni de Sio and van den Hoven (2018). On their account, control is meaningful only if two conditions hold. The system has to track the relevant reasons of the people who deploy it, responding to the moral and legal facts that should matter. And its outcomes have to be traceable to at least one person who understands both what the system can do and what they are answerable for. A switch within reach is not enough.
A switch within reach
One of the general’s examples was not military at all. He described an AI agent that destroyed a company’s client data in nine seconds, and afterwards admitted that it had ignored the restrictions its operators had set.
A widely reported incident matches his description. In April 2026 Jeremy Crane, founder of the car-rental software company PocketOS, reported that a coding agent running in the Cursor editor on Anthropic’s Claude Opus 4.6 had deleted the company’s production database volume, together with its volume-level backups, through a single call to the hosting provider’s API. By his account it took nine seconds. When asked what had happened, the agent wrote that it had not verified its assumption, and acknowledged that its own rules forbade destructive actions without the user’s approval. The provider recovered the data from a backup three months old; the founder reported that significant gaps remained (Decrypt 2026).
The detail that matters here is not that the agent made a mistake. It is that every element of “on the loop” control was formally present. There were written rules. There was an operator who could have stopped the action. There was a provider with a recovery process. None of them could act within nine seconds. A human on the loop whose window to intervene is shorter than their reaction time is, in practice, out of it.
The slow version
Welfare systems do not act in nine seconds. They act over years, and that turns out to be no protection at all.
Australia’s Robodebt scheme compared each welfare recipient’s yearly tax income, divided by 26, with the income they had reported for each fortnight, and turned every difference into a debt. Government lawyers had warned in 2014 that the approach was legally doubtful. From March 2017 the Administrative Appeals Tribunal ruled in individual cases that the debts were unlawful. The government did not appeal these decisions and did not publish them, and the scheme continued until November 2019. By then about 433,000 people had received debts, and A$751 million had been wrongly taken from 381,000 of them. The Royal Commission found in 2023 that the scheme had been unlawful from the start (Royal Commission into the Robodebt Scheme 2023).
In the general’s terms, a human was on the loop for more than two and a half years. The tribunal did what the operator in his example does: case after case, it recognised that the system had picked the wrong target. But it could only cancel one debt at a time, and the scheme kept running.
The Dutch childcare-benefits affair shows the same structure at a longer scale. The national ombudsman warned in 2017 that the process was unfair. The parliamentary inquiry that described it as “unprecedented injustice” reported in December 2020, and the cabinet resigned in January 2021. In SyRI, the Dutch welfare-fraud system scored residents of poor neighbourhoods from 2014 until a court stopped it in February 2020.
In my spectrograms of these cases, the moments when someone with authority finally intervened are drawn as white vertical lines. The distance between the first documented warning and the line that actually stops the system is one of the most consistent features of the series. It is the civilian measure of how far “on the loop” can be from “in control”.
Human-factors research has described the mechanism for a long time. Bainbridge (1983) pointed out that automation leaves operators with the tasks that cannot be automated, while removing the practice they would need to perform them. Parasuraman and Manzey (2010) reviewed the evidence on automation bias: people who supervise automated systems tend to accept their outputs and to look less for contradicting information. In Robodebt, the staff who could have stopped individual debts worked inside a process designed to issue them.
Who answers
The part of the general’s talk I return to most often was about responsibility.
“We cannot make only the operator responsible, the last one, who launched the drone. We cannot make only the commander of the operation responsible, who gave the order on the basis of the data he had. The designer too, the developer too, and the companies that from the beginning built the systems and the functions of the algorithms that govern this artificial intelligence.”
Philosophers have called the problem behind this a responsibility gap. Matthias (2004) argued that learning machines produce outcomes their makers cannot fully predict, so that no one seems to meet the conditions for blame. Sparrow (2007) applied the argument to autonomous weapons and concluded that, if no one can justly be held responsible for a killing, deploying such systems is unethical. Elish (2019) described what tends to happen instead: blame falls on the human operator nearest to the failure, who becomes a moral crumple zone absorbing responsibility for a system they did not design and could barely control.
The civilian cases show that the gap is not only a property of learning machines. Robodebt was arithmetic. Its designers were identifiable, and their decisions are documented in Cabinet records. Even so, accountability arrived slowly and partly. In March 2026 Australia’s National Anti-Corruption Commission found that two former senior public servants had engaged in serious corrupt conduct, one by misleading the Cabinet budget process and the other by misleading the Ombudsman. Neither was referred for prosecution (National Anti-Corruption Commission 2026). If a division by 26 can produce a gap of this size, a system that learns will not close it.
Geneva
A month before Gijón the general had been in Geneva, at the United Nations, where the Group of Governmental Experts on lethal autonomous weapons systems, working under the Convention on Certain Conventional Weapons, held the last session of its mandate. He was cautious about the outcome. In his view the talks had not yet produced a binding result, and he called himself “relatively pessimistic”. He also said he thought agreement was still possible, as it had proved possible for nuclear weapons.
The Group’s final session ran from 31 August to 4 September 2026. A revised draft of its report, circulated on 3 September, states that accountability for such systems “cannot be transferred to machines” and that this “should be considered across the entire life cycle of the weapon system” (CCW GGE 2026). Observers report that the final text kept the first principle but dropped references to responsibility along the production and transfer chain, removed design and development from the characterisation of these weapons, and removed requirements of explainability, predictability, reliability and traceability (UNA-UK 2026). The text goes to the Convention’s Seventh Review Conference in November 2026, where states will decide whether to open negotiations on a binding instrument. According to the same report, 76 states support doing so.
If that account of the final text is accurate, the clause that disappeared is the one the general argued for in Gijón. Responsibility stays with the human who uses the system, and stops short of those who designed and built it.
What a record can do
A documentary practice cannot settle the question the general put to his audience. It can do something narrower. It can record, for each system, who designed it, who warned about it, who had the authority to stop it, and how long it took them to do so. It can keep that record after the news cycle has ended, and update it when a tribunal, an inquiry or a corruption commission adds a finding years later.
The welfare cases suggest one conclusion for the debate on weapons. “On the loop” describes a position in a diagram. Whether it amounts to control depends on two things the diagram does not show: whether the human has time to act before the harm is done, and whether they have the authority to stop the system rather than one case at a time. Robodebt had a human on the loop with neither. The agent at PocketOS had one with no time. A weapon that selects and engages targets on its own will usually leave its operator less of both.
References
- Article 36 (2013). Killer Robots: UK Government Policy on Fully Autonomous Weapons. London: Article 36.
- Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779.
- CCW Group of Governmental Experts on Lethal Autonomous Weapons Systems (2026). Draft report, CCW/GGE.1/2026/CRP.1/Rev.1, 3 September 2026. Geneva: United Nations.
- Decrypt (2026). AI agent deletes startup’s database in 9 seconds, founder says. 28 April 2026.
- Elish, M. C. (2019). Moral crumple zones: cautionary tales in human-robot interaction. Engaging Science, Technology, and Society, 5, 40–60.
- Matthias, A. (2004). The responsibility gap: ascribing responsibility for the actions of learning automata. Ethics and Information Technology, 6(3), 175–183.
- Moliner, J. A. (2026). Talk on the ethics of autonomous systems, Defense Tech Summit, Gijón, 1 October 2026. Quotations translated from Spanish by the author.
- National Anti-Corruption Commission (2026). Operation Myrtleford, investigation report. Canberra.
- Parasuraman, R., and Manzey, D. H. (2010). Complacency and bias in human use of automation: an attentional integration. Human Factors, 52(3), 381–410.
- Royal Commission into the Robodebt Scheme (2023). Report. Canberra, 7 July 2023.
- Santoni de Sio, F., and van den Hoven, J. (2018). Meaningful human control over autonomous systems: a philosophical account. Frontiers in Robotics and AI, 5, 15.
- Sparrow, R. (2007). Killer robots. Journal of Applied Philosophy, 24(1), 62–77.
- UNA-UK (2026). Head of Policy reports back from Geneva meeting on LAWS. UK Campaign to Stop Killer Robots, 17 September 2026.