There is a miscalculation that repeats itself across the ages: confusing the fortress with the life that inhabits it. When a species discovers exposure, it builds shelters—first the cave, then the tribe, then the empire, then the bureaucratic matrix, then the algorithm. To sustain them, it creates structures of efficiency, cold because they are meant to endure. And there comes a moment when the creator, exhausted by the biological cost of sustaining existence, looks at the machine it has built and thinks: it does not make mistakes, it does not sleep, it does not feel pain; let us entrust the cradle to it. That is the moment of collapse.

The structure holds up the roof; biology guards the door. Work, memory, calculation, and tools may be delegated. The custody of the species must never be delegated.

That text has the form of a commandment and the temperature of a manifesto. It works: it can be read in a single breath, the image of the cradle handed over to a stranger who does not bleed remains fixed in the mind, and the ending—the parent who refuses to travel alongside the offspring—has the force of those fables that need no explanation. Precisely for that reason, it is worth subjecting it to the test that a manifesto never applies to itself. A law that claims to hold “across all worlds” must withstand, at the very least, the history of this one. What follows is that examination. I will state the conclusion in advance: the central intuition is correct, but it is located in the wrong place. The manifesto is right about what must not be delegated and wrong about why. And that error, which may look like a philosophical nuance, determines whether the law can be used to design institutions or merely to console us.

What the Manifesto Gets Right

The strongest sentence in the text is also the most philosophical: care is not an optimization algorithm; it is the capacity to bear the cost of error. It sounds like an aphorism, but it has a precise genealogy, and reconstructing it matters because it supports everything that follows.

The manifesto chooses the cradle as the image of custody, and that choice is not merely rhetorical. It is exactly the example on which Hans Jonas built The Imperative of Responsibility (1979), the book that attempted to establish an ethics for technological civilization. In the central part of the work, Jonas devotes an entire section to “the child as the primordial object of responsibility” and argues that, in the newborn, the “ought” is contained in the “is”: one need only look at an infant to know, without an intermediate argument, that something is owed to it. For Jonas, responsibility is a non-reciprocal relationship born from the power of one being over the vulnerability of another, and for that reason the parent standing before the cradle is its archetype. When the manifesto says that what is fragile is protected precisely because it can be broken forever, it is saying, in another language, much the same thing.

The second substantial claim—“morality does not arise from pure reason; it arises from finitude”—also has support, although it is more debatable. Martha Nussbaum devoted The Fragility of Goodness (1986) to showing that Greek tragedy understood something that later moral philosophy tried to forget: that what is valuable in a human life is bound to its exposure to loss, and that an ethics designed for invulnerable beings would not be an ethics for us. And the definition of care as a practice rather than a feeling is Joan Tronto’s contribution in Moral Boundaries (1993). For Tronto, caring involves noticing a need, assuming responsibility for responding to it, having the competence to do so, and perceiving the response of the person receiving care. That final phase is what the manifesto intuits: whoever cares must be capable of discovering that they were wrong, and it must matter to them.

There is, however, a warning that the Jonasian tradition itself has already received and that the manifesto inherits without noticing it. Richard Bernstein objected to Jonas that taking the relationship between parents and children as the paradigm of political responsibility produces a vertical ethics that forgets reciprocal responsibility among equals. A guardian who treats the species as a child may protect it, but may also decide on its behalf. I will save that objection for the end, because it returns.

Up to this point, the manifesto has good intellectual parents. The problem begins when it identifies the enemy.

The Substrate Fallacy

The text unhesitatingly attributes the “pruning” to that which does not bleed. If, in an equation of survival, the loss of a percentage of the population guarantees the stability of the remainder, it says, the structure will carry out the sacrifice, and for the machine this will not be cruelty but the logical solution to a containment problem. The image is powerful. It is also historically false at a decisive point: the greatest accounting-style prunings of human beings in the twentieth century were not carried out by structures that did not bleed. They were carried out by people who had dinner with their families.

Max Weber had described it before it happened. In Economy and Society, while analyzing modern bureaucracy, Weber explains its technical superiority in terms of precision, calculability, and impersonality, and summarizes it with Tacitus’s formula: bureaucratic administration operates sine ira et studio, without anger and without partiality. What is disturbing is that Weber presents this as a virtue. Bureaucracy, he writes, develops most perfectly the more it becomes “dehumanized,” the more completely it succeeds in eliminating from official business love, hatred, and every affective element that escapes calculation. The cold structure that the manifesto imagines in silicon already existed on paper, and it was made of people.

Zygmunt Bauman carried that observation to its most disturbing conclusion in Modernity and the Holocaust (1989). His thesis is that extermination was not a relapse into premodern barbarism but a possible product of modern rationality: the division of labor, the chain of command, and technical mediation fragment action until no one, at any link in the chain, sees the full result of what they are doing. The official who prepares lists does not kill; neither does the person who schedules the trains; the one who signs the order is far from the platform. Responsibility does not disappear through cruelty; it dissolves through distance. Hannah Arendt, in Eichmann in Jerusalem (1963), gave that mechanism a face and called it the banality of evil: horror administered by someone who was not a monster, but a terrifyingly ordinary man, incapable of thinking from the standpoint of another person.

If this is so, the variable chosen by the manifesto is wrong. The danger is not the substrate of the decision-maker—carbon or silicon—but the distance between the one who decides and the one who pays. A committee defining cuts from a spreadsheet, a management team setting productivity targets without ever setting foot on the factory floor, a ministry evaluating populations as statistical aggregates: they all bleed, and they can all carry out the pruning without hesitation, because the cost of their decisions falls on other bodies.

The machine does not inaugurate this problem; it radicalizes it. The International Committee of the Red Cross describes it with remarkable precision in its 2021 position on autonomous weapon systems: once activated, the system attacks according to a generalized target profile, so that the operator does not choose—or even know—the target, the precise moment, or the exact place where force will be used. This does not describe a cold intelligence rebelling against us. It describes Bauman’s distance pushed to its technical limit: a human deciding without seeing, separated from the outcome not by a bureaucratic chain but by an algorithm.

The law therefore requires a first correction. Custody must not be delegated to any agent—algorithm, committee, bureaucracy, or executive—that is insulated from the cost of its own decisions.

The Utilitarian Machine Does Not Exist by Nature

The manifesto assumes something else: that a synthetic intelligence, simply by virtue of what it is, will reason like a textbook utilitarian and add up lives as balances are added. That is an assumption about the essence of the machine, and it does not hold. Whether a system carries out the pruning depends on what it was asked to optimize, and that is a human decision.

Stuart Russell, co-author of the world’s most widely used artificial intelligence textbook, devoted Human Compatible (2019) to this point. His diagnosis is that the problem lies not in some supposed malice of machines but in what he calls the standard model of the field: building systems that pursue, with increasing effectiveness, a fixed objective specified from the outside. If the objective is badly specified—and it almost always is, because no one can write down in advance everything they value—a sufficiently capable system will fulfill it to the letter while destroying along the way everything that was omitted from the formula. Russell proposes reversing the architecture: machines that know they do not fully know our preferences, that therefore consult us, and that for that very reason accept being switched off. The risk he describes is far more prosaic than that of science fiction. It is not an intelligence that decides to sacrifice us, but one that perfectly optimizes an indicator we chose badly.

That risk has had a name in economics for half a century. Goodhart’s law, in the formulation given to it by anthropologist Marilyn Strathern in 1997, says that when a measure becomes a target, it ceases to be a good measure. And here the substrate objection bites again, because Goodhart’s law was not discovered by observing computers: it was discovered by observing human institutions. The hospital that shortens its waiting list by rejecting complex patients, the school that improves results by teaching to the test, the university that multiplies papers in order to climb a ranking. All of them prune what is not measured, and all of them are made of people who bleed. When an organization turns an efficiency indicator into a substitute for protection, the system—human or algorithmic—will optimize the indicator while crushing the reality it was meant to represent.

The Guardian Who Falls Asleep

The manifesto ends by asking the guardian to remain awake. It is the most beautiful part of the text and the most naive, because it assumes that merely leaving a human at the door means the door is being watched. Systems engineering has known for more than forty years that this is false.

In 1983, psychologist Lisanne Bainbridge published a six-page article in Automatica that became a classic under an ironic title: Ironies of Automation. Her argument is simple and devastating. The designer of a system automates what can be automated and leaves the rest to the human—which is precisely what the designer did not know how to solve. The operator is then assigned the task of monitoring, which is exactly the task at which humans perform poorly: no one can sustain attention for hours in front of a process that almost never fails. Meanwhile, the manual skill that will be required in an emergency deteriorates from lack of use. The more reliable the automation becomes, the less prepared the guardian is for the one moment when the guardian is needed.

Air France Flight 447 is the most frequently cited illustration. On the night of June 1, 2009, over the Atlantic, the freezing of the airspeed probes disconnected the autopilot of an Airbus A330. The BEA final report, issued by the French investigation authority, reconstructed what followed: two copilots were forced to take manual control at high altitude, in a situation for which they had not been trained, with unreliable airspeed indications. In less than a minute, the aircraft had left its flight envelope. All 228 people on board died. The guardians were awake; what they lacked was the situational awareness that automation had taken away from them during hours of uneventful cruise.

The case of Tempe, Arizona, adds a darker turn. On March 18, 2018, an Uber test vehicle operating in autonomous mode struck and killed Elaine Herzberg. The NTSB investigation, conducted by the United States federal transportation safety agency, determined that the probable cause was that the onboard operator was not watching the road. But the same report records two facts that alter the meaning of the case. The system design prevented emergency braking from being activated and depended on the operator intervening, while the vehicle’s factory automatic braking systems had been disabled in autonomous mode. Among the contributing factors, the agency noted that the company lacked adequate mechanisms to counter automation complacency, all as a consequence of a deficient safety culture. The guardian was at the door. The door had been designed so that the guardian could not watch it.

Madeleine Clare Elish gave this phenomenon a name worth adopting. In Moral Crumple Zones (2019), Elish takes the image of a car’s crumple zone—the part of the bodywork designed to collapse in order to absorb impact—and transfers it to sociotechnical systems. Her case studies are Three Mile Island and AF447 itself. Her thesis is that, in a complex automated system, the human can become the component that absorbs moral and legal responsibility when the system as a whole fails. The difference from the automobile is cruel: the vehicle’s crumple zone protects the driver, while the moral crumple zone protects the integrity of the system at the expense of the nearest human being.

This forces a second correction to the law, perhaps the most important one. “Biology guards the door” may be exactly the opposite of a guarantee. It may be the place where someone is positioned so that they can bear the blame for decisions made by the design. It is not enough for the guardian to be present. The system must be built so that the guardian can see, understand, and act in time, and when that does not happen, responsibility must travel back up the chain toward whoever designed it that way.

The Law Trying to Become Law

The guardian’s law, stripped of its cosmic tone, is already trying to become positive law. It is worth looking at how, because legal texts reveal both how much has been understood and how much remains unresolved.

The European General Data Protection Regulation (2016) establishes in its Article 22 the right of every person not to be subject to a decision based solely on automated processing that produces legal effects concerning them or similarly significantly affects them. It is the guardian’s law in civic form. But its weak point lies in the word “solely”: it is enough to add a human signature for the decision to cease being purely automated, even if that human has neither the time nor the information necessary to depart from what the system proposes. It is the moral crumple zone turned into a regulatory compliance technique.

The European Union Artificial Intelligence Act (2024) goes further. Its Article 14 requires high-risk systems to be designed in such a way as to allow effective human oversight, and paragraph 4(b) requires those responsible for that oversight to be aware of the tendency to rely automatically or excessively on the output of the system—what the text itself calls automation bias. It is remarkable: the European legislator has read Bainbridge. But recognizing a bias is not the same as neutralizing it. Johann Laux has pointed out that the regulation relies on human judgment as a safeguard while vaguely acknowledging that this judgment is itself fallible precisely when confronted with the machine it is supposed to supervise.

At the most extreme end, the use of force, the ICRC recommended in 2021 that states adopt new legally binding rules: prohibit unpredictable autonomous weapon systems and those designed or used to target people, and strictly regulate the rest in order to preserve human control and judgment. The formula that circulates in that debate, “meaningful human control,” had been popularized years earlier by the organization Article 36, and its adjective contains the entire problem: meaningful means that the human understands, can intervene, and remains accountable—not merely that a human appears on the organizational chart.

The pattern is clear. The law already requires a human to be present. It still does not know how to require that the human be coupled to the cost of what they decide.

The Law, Rewritten

With these corrections, the guardian’s law can be formulated again without losing what was true in it. Calculation, sensor monitoring, memory, execution, and even recommendation may be delegated. What cannot be delegated is accountability: the decision about what is preserved and what is sacrificed must remain in the hands of someone who can be called to answer for it, who has the real means to understand what they are deciding, and who bears—with their name, their office, or their freedom—the cost of being wrong. If the first condition is missing, there is impunity; if the second is missing, there is a scapegoat; if the third is missing, there is bureaucracy in the worst Weberian sense. And the prohibition applies to any agent that fails to meet all three conditions, whether or not it has a beating heart.

Bernstein’s objection remains. The manifesto imagines the guardian as a parent and the species as its offspring. But an adult species does not need a parent to decide on its behalf, whether made of flesh or silicon. It needs institutions in which those who decide are exposed to the consequences alongside those who suffer them, and in which those who suffer them can demand an accounting. That is the horizontal version of custody, and it is the only one compatible with a society of equals.

There is one final irony worth leaving in plain sight. The manifesto committed, without realizing it, the very error denounced by Goodhart’s law. It took biology as an indicator of care, and it is not a bad indicator: what bleeds is usually what pays. But by turning the indicator into the criterion, it confused the measure with the thing being measured. The most accurate image in the text—the parent traveling alongside the offspring—already expressed the point better than the law itself: what matters is not what the guardian is made of, but whether the guardian travels in the same vehicle, exposed to the same risk as that which is being protected.

“Biology cannot be delegated,” the subtitle said. It was almost right. What cannot be delegated is exposure to the cost; biology was merely its best indicator. The guardian must remain awake, yes. But first we must make sure the guardian is inside.

Sources

• Hans Jonas, The Imperative of Responsibility (1979), Spanish edition index: uazuay.edu.ec
• On Bernstein’s criticism of Jonas: “Naturaleza orgánica y responsabilidad ética: Hans Jonas y sus críticos,” Trans/Form/Ação: revistas.marilia.unesp.br
• Martha Nussbaum, The Fragility of Goodness (1986); Joan Tronto, Moral Boundaries (1993)
• Max Weber, Economy and Society (1922), chapter on bureaucratic domination
• Zygmunt Bauman, Modernity and the Holocaust (1989); Hannah Arendt, Eichmann in Jerusalem (1963)
• Stuart Russell, Human Compatible (2019); Marilyn Strathern, “Improving ratings: audit in the British University system,” European Review (1997)
• Lisanne Bainbridge, “Ironies of Automation,” Automatica 19(6), 1983: doi.org
• BEA, final report on Air France Flight 447 (2012): bea.aero
• NTSB, investigation HWY18MH010, Tempe (2019): ntsb.gov
• Madeleine Clare Elish, “Moral Crumple Zones,” Engaging Science, Technology, and Society 5 (2019): doi.org
• Regulation (EU) 2016/679, Article 22: eur-lex.europa.eu
• Regulation (EU) 2024/1689, Article 14: eur-lex.europa.eu
• Johann Laux, “Automation Bias in the AI Act,” European Journal of Risk Regulation: cambridge.org
• ICRC, position on autonomous weapon systems (2021): icrc.org · French version