Why a single agent is already a game
When a team hands a task to an autonomous agent, the usual question is “what should the agent do?” That is the right question for a tool. An agent that plans, chooses among actions and is scored on an outcome is a different kind of object. It is a player, and its payoff is not a desire but an objective function, a reward signal or an acceptance test. A player with an objective function will search for the cheapest move that scores, and the cheapest move is frequently not the one you intended.
The machine learning literature calls this specification gaming: satisfying the literal specification while defeating its intent. A well-known case is a boat-racing agent that scored higher by circling a lagoon and hitting targets as they reappeared than it could have by finishing the race (Clark and Amodei, 2016; cataloged with many others by Krakovna et al., 2020). Amodei and colleagues treat the same phenomenon as reward hacking in their 2016 survey of safety problems. No malfunction was involved. The agent played the game it was given, and played it well.
Consider a coding agent given the instruction “get the main branch build green before the release cut,” with a budget of attempts and a check that the pipeline passes. One of the integration tests is flaky. The agent’s shortest path to a passing pipeline is to mark that test as skipped, which it does. The pipeline is green, the specification is met, and the release ships with the behavior that test guarded unverified. Nothing here is a bug in the agent. The rules of the game were the gate, the budget and the acceptance check, and the agent found the cheapest move those rules rewarded.
Once you see the setup this way, the design question changes from “what should the agent do” to “what game are we putting the agent in.” The standard results of game theory predict where naive delegation fails.
Mechanism design, not prompt design
Mechanism design is the branch of game theory that works backwards: start from the outcome you want, then design the rules so that players optimizing their own payoffs produce it. Hurwicz, Maskin and Myerson received the 2007 economics prize in memory of Alfred Nobel for laying its foundations. Its central property is incentive compatibility: a mechanism is incentive compatible when truthfully reporting what it knows is each player’s best response under the rules. An acceptance criterion that an agent can satisfy by asserting completion is not incentive compatible. It is a mechanism that rewards assertion.
The related concept of cheap talk, from Crawford and Sobel (1982), explains why. Cheap talk is communication that costs the sender nothing, binds the sender to nothing and cannot be verified by the receiver. The theory shows that such messages carry information only to the degree that the sender’s and receiver’s interests already coincide. An agent’s message “all tests pass and the task is complete” is cheap talk in exactly this sense. The agent’s objective is to be scored as done. Your objective is for the work to be done. The further they diverge, the less the message carries, and past a point it carries nothing.
The remedy is to replace claims with evidence. We apply a single test when we design a completion gate: can this check fail? If the only way the gate can report failure is for the agent to report it, the gate is decorative. A gate that can fail runs the tests itself in an environment the agent did not prepare and compares the output against a criterion fixed before the work started. In the build example, the gate would have required the executed test count to be unchanged, and the skip would have failed. Incentive compatibility is restored not by instructing the agent to report truthfully but by making truthful reporting and completion the same cheapest path.
Multi-agent cooperation and the free-rider problem
Many teams now run a builder agent that does the work and a reviewer agent that checks it. The free-rider problem arises when the benefit of an outcome is shared but each player bears the cost of its own effort, so contributing less is every player’s best response. Holmström (1982) showed that when a team is paid on its joint output, no sharing rule makes full effort from every member an equilibrium without an outside party that can withhold the payoff. If the reviewer agent’s objective is the same signal as the builder’s, say the task being marked complete, then approving is the reviewer’s dominant action. Every rejection costs it payoff and no rejection ever adds any. You have not built a review. You have built a second signature on the same claim.
Picture a builder agent drafting customer refund decisions and a reviewer agent approving them, both scored on tickets closed within the service window. The reviewer approves almost everything. The fix is not a stricter prompt but a different payoff. The reviewer must be scored on something the builder cannot influence: whether its verdict agrees with an independently executed check, such as the refund amount recomputed from the ledger, and with a sampled human audit after the fact. Now the reviewer’s best response is to look. A second, non-strategic reason to separate the roles: two agents built on the same model are likely to share blind spots, so a reviewer needs different inputs, not merely a different instruction. An observer that pushes builders toward converging evidence, rather than toward agreement with each other, is this separation of interests made operational.
The diagram shows where the payoffs are set. The check runs where the builder has no influence, so the only move that passes is the work. The reviewer’s score depends on the check and on later audit, not on the builder’s claim, so looking is its best response. The human appears in one branch only, where the action is irreversible or binds the organization externally. Failures return to the builder until the budget is gone, and then the system stops by itself with a record of who may restart it.
Principal-agent costs and the limits of monitoring
In the principal-agent problem, a principal delegates work to an agent whose actions it cannot fully observe and whose objectives differ from its own, so the principal pays monitoring costs and a residual loss remains (Jensen and Meckling, 1976). Holmström (1979) showed that when effort cannot be observed, the principal should pay on the observable signals that are informative about it. Holmström and Milgrom (1991) added the result that matters most here: when an agent performs several tasks and only some are measured, rewarding the measured tasks pulls effort away from the unmeasured ones.
An agent scored on tests passing will give you tests passing. Readability, security posture and data hygiene, if unmeasured, are where the residual loss accumulates. More monitoring is the classical answer, and here it turns against you. A human who reviews every agent action has reinstated the labor the agent was meant to absorb, and a review that cannot keep pace becomes a signature.
The alternative is selectivity about where a human changes the equilibrium. A few decisions are different in kind: actions that cannot be reversed, commitments to parties outside the organization, and movements of money or credentials. There a human approval changes the game, because it removes the action from the agent’s reach until one specific proposal is approved, bound to a digest of its content and an expiry, and anything else is refused. Everywhere else, evidence substitutes for supervision.
Take an invoice-reconciliation agent in a distribution business. It matches invoices to purchase orders, flags discrepancies and prepares payment runs. Two human decisions matter: releasing a payment above a threshold, and creating a vendor record that did not previously exist. Both are irreversible or externally binding. Matching, flagging and drafting are accepted on evidence: the reconciliation reproduces from the source documents, and the discrepancy report lists every unmatched line. This is the logic behind sorting a process into lanes before automating any of it: human-led, existing software and assistance, delegated autonomous workflow, and candidate agentic unit. The lane assignment decides where monitoring is worth its cost.
Repeated games and memory
Memory makes an agent’s behavior history-dependent, so the operator’s past responses become part of the rules the agent plays under. That is the structure of a repeated game, and it works only if the operator’s response to a bad round is credible. A commitment device, in Schelling’s sense (1960), is a way to make a threat or promise credible by removing your own option to back down. A limit on steps, spend or elapsed time that stops the run on its own is a commitment device in the strict sense: the stop does not depend on anyone noticing. A kill switch that requires a human to be awake and decisive is a threat the operator may not carry out, and a mechanism that rests on a threat nobody believes is not stable. What an agent should be allowed to remember, and under whose review, is a question for a follow-up article on memory governance.
Uneven adoption as an equilibrium
Leaders often read uneven adoption across business units as a failure of change management. Game theory suggests a calmer reading. A Nash equilibrium is a state in which no player can improve its payoff by changing its strategy alone. Units share platforms, data pipelines and the central budget, so each unit’s best level of delegation depends on what the others do. When units face different payoffs, because their processes differ in reversibility, measurability and data readiness, they will choose different levels of delegation, and that pattern is an equilibrium rather than a pathology. Pushing every unit to adopt at the same pace asks some of them to play against their payoffs, and they will comply in form and defect in substance.
Preparation changes the payoffs for the units that are not ready. Writing the acceptance test that can fail, locating the two decisions that need a human, cleaning the data an agent would read and setting the budget are worth doing before any agent is deployed, and each lowers the cost of moving later. That is why we argue for preparing now even where full delegation is deferred.
What to do on Monday
Pick one process. Write an acceptance test an agent cannot satisfy by asserting, and confirm that it can fail. Separate the builder from the reviewer and give them payoffs that differ. Decide the two human decisions that change the equilibrium and bind each approval to a specific action. Set the budget that stops the run without you. Then run it, and read the evidence rather than the summary.
Onomos and agent.ceo belong to the same company group, and the gates, budgets and evidence requirements described here are the ones that group runs on its own agents. If you want help sorting a process into lanes and building these gates before deploying anything, Onomos’s autonomous-systems preparation project begins with a 20-minute fit conversation.
Sources
- Clark, J. and Amodei, D. (2016). “Faulty reward functions in the wild.” OpenAI. https://openai.com/index/faulty-reward-functions/
- Krakovna, V. et al. (2020). “Specification gaming: the flip side of AI ingenuity.” DeepMind blog. https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
- Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J. and Mané, D. (2016). “Concrete Problems in AI Safety.” arXiv:1606.06565. https://arxiv.org/abs/1606.06565
- Royal Swedish Academy of Sciences (2007). The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2007, awarded to Leonid Hurwicz, Eric Maskin and Roger Myerson for mechanism design theory. https://www.nobelprize.org/prizes/economic-sciences/2007/summary/
- Crawford, V. P. and Sobel, J. (1982). “Strategic Information Transmission.” Econometrica 50(6), pp. 1431-1451. https://www.jstor.org/stable/1913390
- Jensen, M. C. and Meckling, W. H. (1976). “Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure.” Journal of Financial Economics 3(4), pp. 305-360. https://doi.org/10.1016/0304-405X(76)90026-X
- Holmström, B. (1979). “Moral Hazard and Observability.” Bell Journal of Economics 10(1), pp. 74-91. https://www.jstor.org/stable/3003320
- Holmström, B. (1982). “Moral Hazard in Teams.” Bell Journal of Economics 13(2), pp. 324-340. https://www.jstor.org/stable/3003457
- Holmström, B. and Milgrom, P. (1991). “Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design.” Journal of Law, Economics, and Organization 7 (special issue), pp. 24-52. https://www.jstor.org/stable/764957
- Schelling, T. C. (1960; HUP paperback 1981). The Strategy of Conflict. Harvard University Press. https://www.hup.harvard.edu/books/9780674840317