A few months ago I was shown a procurement agent. It read supplier emails, compared quotes, and could raise a purchase order up to a set limit without asking anyone. The team was proud of it, and honestly it was good work. Then I asked who else could send it an email.
The answer was anyone. That is the whole problem with agents in one sentence.
A chatbot answers. An agent acts.
For years the worst thing a language model could do was say something embarrassing. That is a reputational problem. The moment you connect the model to tools, an inbox, a database, a payment API, a shell, it becomes an operational one. The model is now a user of your systems, and it is a user who believes whatever it reads.
I spent most of my career breaking into organisations for a living. The way in was rarely a clever exploit. It was a person, a process, and a bit of trust in the wrong place. An agent with tools is all three, wrapped in an API.
The four questions
When I threat‑model an agent I do not start with the model. I start with four questions about the system around it.
What can it read? Every input is an attack surface. Emails, web pages, documents, tickets, tool outputs. If the agent reads a supplier's PDF, the supplier can put instructions in the PDF. If it reads the web, so can anyone. This is prompt injection, and no amount of prompt engineering fixes it, because the model cannot reliably tell data from instructions. Treat everything it reads as hostile.
What can it do? List the tools. For each one, ask what the worst call looks like. Send an email to whom? Read which tables? Spend how much? Then take away everything the job does not need. The procurement agent did not need to email arbitrary addresses. It needed to email the four suppliers on the approved list.
Who can talk to it? Direct users are the easy case. The hard case is indirect: anyone who can get content in front of it. That includes the people it emails, the sites it browses and the other agents it calls. Draw that map. It is usually larger than the team expects.
What happens when it is wrong? Not if. Models are wrong at a steady rate and attackers will raise that rate on purpose. Which actions are reversible? Which need a human? What does the log show afterwards?
Controls that actually work
Once you have the map, the controls are familiar. Security people have been doing this for service accounts for twenty years.
- Least privilege for tools, scoped per task. A separate credential for the agent, with the narrowest permissions the job allows, not the developer's own token.
- Hard limits outside the model. A spending cap enforced by the payment API, not by a line in the system prompt asking nicely. The model can be talked out of a prompt. It cannot be talked out of a policy the API enforces.
- Human approval for anything irreversible. Money, deletion, external communication, access changes. The agent drafts, a person clicks.
- Full logging of inputs, tool calls and outputs, kept somewhere the agent cannot reach. When something goes wrong, and it will, you need to replay the conversation.
- Adversarial testing before launch and after every meaningful change. Try to make it spend money, leak data and email the wrong person. If your red team has not attacked your agents yet, they are behind.
If you want a checklist to argue with, OWASP's Top 10 for LLM applications maps cleanly onto the four questions above, and NIST's AI Risk Management Framework gives you language your risk committee already understands.
Back to the credit card
The procurement agent is still running. It now emails four suppliers instead of the world, its spending cap lives in the payment system, and any order over the limit waits for a human. It took about a week to change and it lost none of the value the team built.
That is the usual outcome. Nobody has to stop building agents. They have to build them the way they would build any other system that can move money, with the assumption that someone, eventually, is going to try to talk it into something.
