Securing an AI-agent system on Google Cloud, layer by layer

0:00Cold open

Over five working sessions, an agent built a system for me on cloud infrastructure. Five language models, a mailbox open to the public, a web form anyone can submit, and a channel it publishes to on its own. I never opened the virtual machine. The question I get asked about that is not how it works. It's how I sleep. The answer is a habit I picked up over a long career in networks and security: assume it gets compromised, and make the failure boring.

0:34Format note and the system

Before I show you what that habit built, a quick note on the format. This one's a talk piece, so treat it like a podcast: listen without watching and you won't miss a thing. And so you have the picture, the system is a debate show. Four models argue over a letter from the public, a fifth one rules, and the whole thing renders and publishes itself. It runs in its own cloud project, on one virtual machine, with a handful of containers and a few managed services around it. An agent built all of it while I watched and talked. What follows is what I said about security while it was being built, cleaned up so it reads straight through.

1:18The box, and blast radius

So, the habit, applied. It starts with where the thing lives. There's an organization, a folder inside it, and the project inside that. One box. If we need to shut it down, move it, or hand it to someone else, it has no arms and legs reaching outside the box. And the bill for that box is its own bill, not a line buried in someone else's. Inside the box, the rule is least privilege: every key is reachable only by the one process that needs it. That's not a compliance phrase. It's a blast radius. If one piece is compromised, what leaks is what that piece needed, not the whole project.

2:03One key per purpose; secrets in the store

Two smaller things follow from that blast radius. First, one key per purpose. The voice service hosts this channel's voice and the show's voices, and the show got its own key, so if it ever has to be revoked, the show goes quiet and nothing else does. Second, secrets don't live in files on disk. There was an environment file at the start, because you have to begin somewhere. One worker's entire job was to read that file, push every value into the cloud's secret manager, and confirm nothing was left behind. The file was a staging step, not a home.

2:43Budgets as circuit breakers

Keys and secrets are the part everyone expects. The part people skip is the bill. Every project got a budget, and the budget has an action attached: when spending crosses the line, billing detaches and the project stops. Not an email. A stop. You've seen the stories on message boards, the cloud provider billed me ten thousand dollars over a weekend. Almost every one of those is a person who set a budget that only sent a notification, or set nothing at all. A budget without an action is a notification. A budget with a stop is a circuit breaker.

3:24The network: no way in, one way out

With the bill fenced in, the next fence is the network itself, and the posture is simple to say. The cloud firewall denies all inbound traffic. The only way to a shell is through the provider's identity-aware tunnel, tied to an account, not a port. The machine has no public address; it goes out through network address translation. And every outbound request from the containers goes through one proxy, which only allows a short list of sites. I first built that proxy years ago, running my first agent, when there was no guidance on containing one. The thought was plain: if this thing is compromised, where can it reach? With an allow-list proxy the answer is a list you wrote. Not everyone would think to build it. That's the experience part, and it's the part that matters.

4:18Walls are the design working

All of that sounds tidy in a list. Here is what it feels like while you build. You fix one permission and the next one fails. Then the next. Anyone who has run a tight firewall knows the rhythm: in testing you hit walls, constantly, because every process is scoped to exactly what it needs and nothing more. It's tempting to grant broadly so everything just works. Then you've handed a process the ability to do things it should never be able to do, and you'll find that out later, from someone else. The walls are the design working. One evening the agent offered me a shortcut: a broader bundle of rights that would clear every error at once. Read carefully, that bundle would have let the build identity grant itself owner of the project. I took the narrow option. When an agent offers a permission shortcut, read it for the path that leads to owner.

5:18The public surface

Everything so far protects the inside. Then there's the front door. A mailbox and a web form, open to the world, feeding a system that spends money every time a model reads something. So the denial of service you fear isn't a crashed server. It's the invoice. Anything from outside is treated as untrusted and cleaned before anything else touches it. Behind that, a front door with a challenge on it, the are-you-human kind. And I'll be honest about that door: agents solve those now, by looking at the tiles and adding a little human-looking delay. So the challenge is not the defense. It's the first layer. Behind it sits a circuit breaker: past a load the system was never meant to carry, intake stops for everyone until the storm passes. Yes, that's a small outage we chose. Compare it to processing a hundred thousand bogus messages through five models. Layers, like an onion, each one assuming the one in front of it already failed.

6:30Mail, the arms race, and looking

The mailbox gets one more layer of its own. Mail carries records so nobody can send as the show from somewhere else. Domain security, the authentication records, all of it, set before the first message went out. But I want to be careful about the word secure. There is no finished state. It's an arms race, and you mitigate what you can, then you watch. If a new hole opens, you close it then. Even when the whole pipeline runs itself, tickets in, episodes out, it is not set it and forget it. Somebody checks that the works are working. Inspect what you expect. Automation changes who does the work. It doesn't change whether someone has to look.

7:17Recovery is security

And after all the watching, assume it fails anyway. Someone gets in. What now? Do we have snapshots of the machine? Exports of the data? A switch that stops the show, revokes its publishing token, and destroys its voice key in one motion? A way back to production, and how long does it take? When you buy that from a security firm, they promise you a number in a contract. I don't have a firm. I have a plan. So the safeties went in before the show went live, not after the first bad day: daily snapshots, daily exports, a backup bucket that cannot be shortened or deleted, a private copy of the record, and a restore actually proven into a scratch database before anything was allowed to drill against the real one. A breaker you can't reset cleanly is an outage, not a control. Test the way back too.

8:21Close

That's the whole stack, from the box to the way back. None of it needed a security team. It needed the plan to say it before the build started, so the agent built it that way from the first hour. That's the lesson from a long career, said plainly: cover all the bases, not one measure held up as the standard. Whatever you run, ask the one question of it: when this is compromised, what happens? Then make the answer boring. That's the whole job.