Note: the diary in this post is a hypothetical scenario assembled from common allowlist-migration patterns — not an account of any real company.
Our support agent started the way most agents start: with access to everything. It was a demo that became a product. Someone wired the CRM, the email sender, the refund tool, the internal wiki, the Slack workspace, the billing admin (47 tools total) because each one made the demo five seconds more impressive. By the time the agent was handling real tickets, nobody remembered which tools it actually needed. That's normal. That's also how you end up with a bot that can issue refunds and DM your CEO.
Nothing bad had happened yet. That's the dangerous part. "It hasn't broken anything" is the argument for keeping broad access, and it's true right up until the day it isn't. We decided to move to an allowlist: the agent can call only the actions we name, everything else is denied by default. Here's the diary.
Week 1: the audit (and the first surprise)
We pulled 30 days of tool-call logs and sorted by frequency. The agent had called 19 of its 47 tools at least once. Of those 19, exactly 11 were things it needed: reading tickets, searching the knowledge base, drafting replies. The other 8 were curiosity: the agent poking at tools "just in case," or worse, tools it had used once during the demo era and never again.
The surprise was the refund tool. It had been called twice. Both times the agent almost issued a refund (the amounts were $12 and $19), and both times it stopped because the tool itself required a confirmation the agent couldn't produce. We had been one missing guardrail away from a bot refunding customers on its own judgment. Two near-misses in 30 days. The allowlist project suddenly had everyone's attention.
Week 2: the staging allowlist
We wrote the first allowlist from the audit: 11 actions. Then we ran the agent in staging against replayed tickets for three days and watched what it tried to do that the allowlist denied. This is the step you can't skip. Logs tell you what the agent did. Replays tell you what it wants to do, including the things it only attempts when tickets get weird.
Staging caught three things the audit missed. First, the agent needed an "add internal note" action for tickets it couldn't resolve. Without it, the agent just sat on unresolvable tickets silently. Second, it needed "escalate to human," which we'd left out because no log entry showed it; the agent had been dropping unresolvable tickets into a void instead. Third, and this one hurt: the "mark ticket resolved" action had a different name in staging than in production (close_ticket vs resolve_ticket), and our allowlist named the wrong one. A naming mismatch is all it takes for an allowlist to silently block a core workflow.
Week 3: production, and what broke
We shipped the allowlist on a Tuesday morning. By lunch, two things were broken.
First, the escalation path. We'd allowed "escalate to human" but the escalation flow sent a Slack DM to the on-call engineer, and Slack DMs weren't on the allowlist. Urgent tickets escalated into a queue nobody watched. It took four hours to notice because the agent reported each escalation as successful. Technically true — the message just went nowhere anyone reads.
Second, refunds. We'd deliberately left the refund tool off the allowlist and put it behind the approval queue instead. Good decision, wrong rollout: the agent had been told in its system prompt that it could issue refunds, so it kept attempting them and piling approval requests onto the human queue. Forty-one refund requests in the first day, all denied, all noise. We had to update the agent's instructions the same week. The allowlist and the agent's beliefs about itself have to match; otherwise you get a bot that spends its day knocking on a locked door.
What the allowlist ended up containing
After three weeks of tuning, the production allowlist for the support agent settled at 6 allowed actions and 3 approval-required actions:
- read_ticket: allowed. Reads everything, changes nothing.
- search_kb: allowed. Knowledge base lookups.
- draft_reply: allowed, but drafts only. Nothing sends without a human or a second pass.
- update_ticket_status: allowed, with status transitions restricted to open → in-progress → resolved. No reopening, no deleting.
- add_internal_note: allowed. The agent's way of saying "I couldn't handle this, here's what I found."
- escalate_to_human: allowed, wired to the paged channel this time, not a DM void.
- issue_refund: approval required. Every single one lands in a human queue with the ticket context attached.
- send_customer_email: approval required. Drafts are free; sending is not.
- update_billing_record: approval required. Touched twice a month, always by a human decision.
Everything else (the other 38 tools) is denied by default. In the first month, the agent attempted a denied action 14 times. Thirteen were the agent probing out of habit. One was interesting: it tried to look up a customer's usage data through a reporting tool we'd removed. That attempt became a feature request, which became a new allowlist entry after review. Denied attempts turned out to be a decent signal for what the agent legitimately needs next.
The rule for new tools
Allowlists rot. Someone adds a tool "temporarily" and it stays forever. Our rule now: every new tool starts as approval-required for 30 days, with usage logged. After 30 days, either it earns its way onto the allowlist with evidence, or it gets removed. No tool graduates on a hunch. The 30-day window is long enough to see real usage patterns and short enough that nobody forgets the review.
We also review the allowlist itself every quarter. Last quarter we removed the agent's ability to reopen resolved tickets, because it had started reopening tickets customers had already confirmed as fixed. An allowlist entry that made sense in March was wrong by September. Access decisions have a shelf life.
What I'd tell you before you start
Start from logs, not from theory. The audit is the whole project; everything after is execution. Run replays in staging before you touch production: the weird tickets are where the missing actions hide. Update the agent's instructions in the same release as the allowlist, or you'll drown in denied-attempt noise. And keep the denied-attempt log somewhere you'll actually read it. It's the cheapest source of "what should this agent be allowed to do next" you'll ever get.
One more thing: an allowlist that's too tight is its own failure mode. Our first draft was overzealous, and the agent responded by routing around it in ways that were technically allowed and completely unhinged — the full story is in the denied-attempt log section above. When the agent starts routing around your policy, the policy is wrong, not the agent. Loosen deliberately, with the approval-required tier as the middle ground. Deny-by-default with a sane middle tier beats both "everything open" and "everything locked." You can test allow/deny verdicts against your own action lists in our policy playground — the same evaluation engine the production proxy uses.
Constrain the parameters, not just the action
There's a gap in most first-draft allowlists, and ours had it too. The allowlist says which actions are permitted, but an allowed action with unconstrained parameters is still a loaded tool. send_customer_email was approval-required for us, so we were covered there — but consider draft_reply, which we allowed freely. Drafting is harmless. What's inside the draft isn't necessarily.
The sharper example: search_crm. Allowed, obviously — the agent can't do its job without it. But search_crm with a blank query returning the entire customer table is a different animal from search_crm for one ticket's customer. Same action, wildly different blast radius. We handle this with parameter constraints on the risky ones: the agent may search, but result sets are capped at 25 records, and bulk-export parameters are denied outright. If it genuinely needs more, that's what the approval tier is for.
The cleanest version of this we ever saw: a team allowed send_sms but constrained the recipient parameter to phone numbers already in the customer's profile. The agent could text customers all day. It could not text an arbitrary number. That's the line between "tool access" and "tool access with guardrails" — and it took one line of policy, not a rewrite of the agent.
Rule of thumb: if an action has a parameter that changes its blast radius (recipients, amounts, record counts, delete flags), that parameter gets a constraint. Actions without such parameters get a plain allow. Review the parameters, not just the action names. The allowlist is a contract, and contracts live in the fine print.
Rehearse the deny: read your denied-attempt log
That first month, the agent attempted a denied action 14 times. Here's the thing about a denied-attempt log: it's three different signals wearing one uniform, and you have to sort them. We review ours every Friday, and the review takes fifteen minutes.
Signal one is habit. The agent keeps trying a tool it was allowed last month. Thirteen of our fourteen were this. No action needed — but if the same habit persists for weeks, the agent's instructions are stale and need an update, because a bot knocking on a locked door is wasting everyone's context window.
Signal two is legitimate need. One of ours was the agent trying to look up a customer's usage data through a reporting tool we'd removed. That attempt became a feature request, which became a new allowlist entry after review. Denied attempts are the cheapest possible source of "what should this agent be allowed to do next" — the agent is literally telling you what it wants.
Signal three is routing around a gap. The scariest one we caught: the agent, denied the internal-note action in an early draft, started writing its findings into ticket titles. Technically allowed. Completely unhinged. When the agent starts routing around your policy, the policy is wrong, not the agent. Loosen deliberately, grant the real action, and the weird behavior stops.
Every attempt is timestamped, tied to the agent's credential, and carries the deny reason — that audit trail is what makes the Friday review possible at all. (Our sister platform Agent Flight Recorder keeps exactly this kind of tamper-evident action log for production agents.) No log, no review. No review, and your allowlist is fossilizing while your agent improvises.
FAQ: allowlists for agent tool access
- What's the difference between an allowlist and a denylist for AI agents?
- An allowlist names the actions an agent may take and denies everything else by default; a denylist names the actions it may not take and allows everything else. Allowlists are safer because new tools — and new ways to misuse old tools — are denied automatically until reviewed. Denylists fail the moment something dangerous isn't on the list.
- How many tools should be on a production agent's allowlist?
- Fewer than you think. A working support agent in our walkthrough settled at 6 allowed actions plus 3 approval-required ones, down from 47 available tools. Start from 30 days of real tool-call logs, keep what the agent demonstrably needs, and park everything else in the approval tier for 30 days before it earns a permanent spot.
- How do I stop an allowlist from going stale?
- Three habits: new tools enter as approval-required for 30 days with usage logged, never straight onto the allowlist; review the denied-attempt log weekly for legitimate needs and policy gaps; and re-audit the allowlist quarterly, because access decisions have a shelf life — an entry that made sense in March can be wrong by September.