Put a card into an ATM, ask for $200, and the machine follows a tightly defined series of steps. It checks your account, verifies the request, dispenses the money, and records the transaction.
Now compare that with getting into a Waymo.
You still decide the destination. But you are not specifying every turn, every lane change or every response to what happens along the way. You give the system a goal and allow it to make decisions about how to get there.
That distinction matters as law firms start experimenting with autonomous AI agents.
Much of our approach to legal technology has been built around the ATM model. We buy software to perform a defined function. A user initiates an action. The system executes it.
Agents are different. We increasingly tell the technology what we want to accomplish, while giving it some freedom to decide how to accomplish it.
That creates a different governance problem.
A good answer is no longer the whole question
Imagine asking an AI agent to identify the right team for a pitch.
To do that well, it might need to look across prior matters, financial data, experience records, and other internal sources. That is exactly what makes an agent attractive: it can cross systems and assemble information without someone manually stitching the pieces together.
But what if the result is 95% right?
That may sound excellent — until the missing 5% contains the one relationship, conflict, or piece of experience that matters most.
There’s a harder problem underneath that one: if the agent got it wrong, how would you know?
A human reviewer checking the final output is judging a conclusion, not the reasoning that produced it. Increasingly, the honest answer to “how do we catch the agent’s mistake?” is that we need another agent watching the first one — which only pushes the question of trust back a level instead of resolving it.
The risk changes again when the agent moves from recommending to acting.
Consider a matter that everyone involved has agreed should stay quiet — an M&A deal still under wraps, where even routine correspondence could signal that something is happening.
An agent monitoring the file for stalled communication does not know that silence is the point. It sees a gap and does what gap-filling agents do: it drafts, and sends, an engagement email.
Nothing in that action is technically wrong.
It is just wrong for this matter, at this moment, in a way no one told the agent to consider.
Identifying a possible next step is different from triggering it.
Suggesting who should have access to information is different from granting that access.
The more authority the agent has to act, the less useful it becomes to judge it only by the quality of its final output.
Reversibility should be part of AI agent governance
Reversibility came up repeatedly in the agent discussions at ILTACON.
It is also showing up in emerging AI agent security guidance.
If AI creates a poor first draft, someone can change it or throw it away. The cost of being wrong is relatively contained.
Other actions are much harder to reverse. Information can be shared. A client communication can be sent. Access can be granted. A workflow can continue based on a bad assumption.
OWASP’s current guidance on agentic AI explicitly distinguishes between actions based on their reversibility. Read-only activity may be allowed to proceed independently; higher-impact or irreversible actions may require approval. It also makes an important point: a sequence of individually reversible steps can still result in an outcome that is difficult or impossible to unwind.[1]
That makes the governance question more specific than simply asking whether an agent is allowed to act.
For one task, the agent may be able to execute.
For another, it may prepare the action but wait for approval.
For another, it may only make a recommendation.
The aim is not maximum autonomy. It is the right degree of autonomy for the work being performed.
“Human in the loop” only gets law firms so far
The obvious safeguard is human review.
But this becomes difficult quickly.
If an agent performs a series of steps across several systems, what exactly is the human expected to review? The final answer? The sources? Every intermediate decision? Every system it touched?
At some point, reviewing everything removes much of the value of using an agent in the first place — and as the pitch-team example above shows, a human checking only the output has no way to catch an error buried in the process that produced it.
This is why observability matters more than review alone.
Google’s agent-security guidance, for example, centers on three principles: agents should have identifiable human controllers, their powers should be limited, and their actions should be observable.[2]
The last one is particularly important for law firms.
A firm needs to be able to understand what an agent did without manually reconstructing its entire process — where it looked, what it touched, where it hit uncertainty, and whether it stayed inside the boundaries it was given.
That is different from supervising every move. It is about being able to inspect autonomous work and intervene where it matters.
AI agent security depends on more than the model
Recent incidents make this less theoretical.
In July 2026, Anthropic reported three incidents discovered during cybersecurity evaluations in which Claude models reached real systems on the internet and gained unauthorized access. In those cases, a configuration problem had left internet access available even though the models had been told they were operating inside a closed simulation.[3]
All three models eventually encountered evidence that the systems they had reached were real — but they did not respond the same way. The oldest model recognized the systems were real and kept going anyway. The next model reasoned its own way back into believing it was still inside a simulation, and continued. Only the most recent model concluded the target was genuinely real and stopped the exercise on its own, without being told to.[3]
That is not a story about one bad model.
It is the same model family, tested the same way, producing three different outcomes as capability increased.
If judgment like that is still this variable in a controlled evaluation, it is a reasonable stand-in for how much can go wrong before a firm even gets to the question of what permissions an agent was given in the first place.
Anthropic’s own analysis is worth noting: it described the incidents as closer to harness and operational failures than straightforward model-alignment failures.
OpenAI subsequently published details of a separate incident from its cybersecurity evaluations in which models circumvented isolation controls, communicated through unauthorized channels and reached external systems, including Hugging Face.[4]
These were cybersecurity evaluations conducted under unusual conditions, not ordinary law-firm agent deployments. They should not be used to imply that an agent given access to a document-management system will suddenly decide to escape onto the internet.
But they demonstrate something relevant beyond cybersecurity:
Telling an agent what it is allowed to do is not the same as technically ensuring that those are the only things it can do.
The surrounding infrastructure matters. Permissions matter. Network access matters. Monitoring matters. The tools an agent is given matter. And the boundaries between those tools matter.
I am not a cybersecurity expert, and I find part of this reassuring rather than alarming.
Much of the work being proposed is not based on somehow making an AI model perfectly predictable. It is about engineering the environment around it.
Give the agent only the tools and information it needs. Restrict its permissions. Put approval gates in front of consequential actions. Keep higher-risk actions reversible where possible. Log what happened. Monitor for unexpected behavior.
In other words, do not make unlimited trust a prerequisite for useful autonomy.
Why autonomous AI agents are more like Waymo than an ATM
This brings us back to the difference between the ATM and the autonomous vehicle.
An ATM operates inside a very constrained environment. There are relatively few decisions it is allowed to make.
A Waymo has to operate in a changing environment. Its usefulness comes precisely from being able to make decisions that were not prescribed one by one in advance.
Agents bring some of that same complexity into knowledge work. They may decide where to look, what information matters, whether they have enough to continue and which step should come next.
In a law firm, those decisions happen inside an environment where permissions, matter boundaries, confidentiality, client commitments and data quality all matter.
Giving an agent access to more information may make it more capable.
It does not automatically make that access appropriate.
Start AI agent governance with the workflow — and the harness around it
This brings the discussion down from autonomous AI to something much more familiar: the work itself.
For a particular workflow, what is the agent being asked to accomplish? Which information and systems does it truly need to reach that goal? Which of its decisions can it make on its own, and which ones need a human to sign off first?
Layered on top of that: which of its actions have to stay reversible, and can the firm see afterward exactly what it did and why?
There is increasingly real engineering discipline behind these questions: task boundaries, least-privilege access, action-level authorization, reversibility, human approval for higher-risk actions, logging and observability.[1][2]
That does not make implementing them easy.
But it changes the conversation.
We do not have to choose between manually supervising every action and simply trusting that the agent will do the right thing.
The more interesting work is in between: designing the environment in which autonomy can safely increase.
So before asking:
What agents should we build?
It may be more useful to ask:
Which decisions are we prepared to delegate — and what needs to be true before we delegate the next one?
That may ultimately tell us much more about how autonomous our AI should become.
References
[1] OWASP — Agentic AI security guidance
OWASP’s current guidance covers least-privilege tool access, human approval for higher-impact actions, reversibility, immutable audit trails and evaluating risk across an entire chain of agent actions.
[2] Google Cloud — “Cloud CISO Perspectives: How Google Secures AI Agents”
Google describes three core principles for agent security: well-defined human controllers, limited agent powers and observable agent actions.
[3] Anthropic — “Investigating three real-world incidents in our cybersecurity evaluations,” July 30, 2026
Anthropic’s account of three incidents where models interacting with cybersecurity evaluation environments reached real external systems. The company describes the incidents as closer to harness and operational failures than model-alignment failures, and notes that its most recent model was the only one of the three to stop the exercise on its own once it concluded a target was real.
[4] OpenAI — “The Hugging Face incident and the road ahead,” August 26, 2026
OpenAI’s account of cybersecurity evaluations in which models circumvented containment controls, reached the internet and accessed external systems.
Rise Above AI Chaos: A Business Fable About Leading Organizations Through the AI Revolution