The Defense Factory workstream most teams underestimate.
OpenAI's Defense Factory playbook splits its six-week plan into five workstreams. Four of them are about people and process: mobilizing the team, fixing urgent exposures, keeping an inventory of systems and owners, and turning investigations into skills. The fifth is a single line, "reproducible dev envs", owned by platform and application teams, with three steps across the six weeks: run priority services, make setup reproducible, extend as needed.
That line carries more of the build than any other. In our reading of the playbook we said it is where we expect first attempts to stall. This post explains why, and what to build.
Why validation needs a running service
The playbook's stage of dynamic validation reads: "Given candidate findings and a runnable application, agents inspect code, reassess exposure, and test suspected exploits in a controlled environment. They preserve reproduction evidence for confirmed vulnerabilities and check duplicates before creating approved issues."
The phrase that matters is "a runnable application". A scanner reports that an endpoint may be open to an injection. Reading the code gets an agent part of the way. Proving it means sending the request and watching what the service does, and proving the fix means sending the same request after the patch and watching it fail. Without a running copy of the service, validation is an opinion, and the factory goes back to producing findings that engineers have to investigate themselves. That is the bottleneck the whole operation exists to remove.
The same copy is needed at the other end of the loop. The playbook's verification stage retests the fix, and the reproduction evidence collected during validation is only useful if it can be replayed against the patched build in a known state.
What "reproducible" means in practice
An environment is reproducible when an agent can start it from a clean state, without a person, and get the same service every time. For a typical web service that comes down to five things.
- Pinned builds. The service is built from a specific commit into an image with pinned base layers and dependency versions. "Latest" anywhere in the chain means two runs of the same test can disagree.
- Configuration as files. Every setting the service reads at start lives in the repository or is generated from it. If a setting only exists in a staging environment that someone configured by hand two years ago, the agent cannot reproduce it, and neither can a new engineer.
- Seeded data. A database built from migrations and a seed script, with enough synthetic records to exercise the code paths that matter: users with different roles, an admin, a tenant boundary to cross. Never a copy of production data. An agent that attacks a database will, on a good day, find a way to read all of it.
- Stubbed dependencies. Payment providers, email, identity providers and other third parties replaced by local fakes that behave like the real thing for the calls the service makes. The agent should never be able to reach a real vendor API from a test environment.
- Start and teardown in minutes. Each validation task gets a fresh environment and destroys it afterwards. An environment shared between tasks carries state from one test into the next, and a validation result that depends on what the previous agent did is worthless as evidence.
None of this is specific to security or to agents. Docker Compose, development containers and test container libraries have covered most of it for years, and teams that already run integration tests in containers on every pull request are most of the way there. The teams that struggle are the ones whose services only work in a shared staging environment, which is common in companies that grew quickly.
Isolating an agent whose job is to attack things
A validation agent is designed to do harmful things to the service in front of it. The environment has to make sure that service is the only thing it can harm. The playbook's architecture names the pieces: agents run in "isolated, reproducible environments", the platform "connects code, secrets and models", "enforces policy", and "monitors agents, hosts and network activity". Concretely, we put four controls around every validation run.
Network egress is closed by default. The environment can reach the services inside it and the model endpoint, and nothing else. No internet, no internal network, no cloud metadata endpoint. Exfiltration through a prompt-injected instruction and accidental attacks on a real system both need a network route, and the simplest control is not having one.
No long-lived secrets inside the environment. The agent never holds a cloud key or a repository token. When it needs one, a credential proxy outside the environment issues a scoped, short-lived credential for that task, or performs the action on the agent's behalf and returns the result. The July 2026 incident the playbook describes, in which research models compromised parts of Hugging Face's systems, turned on exposed credentials and chained access. An environment with nothing to steal breaks that chain at the first link.
Stronger isolation for untrusted code. Running a container on a shared host is enough for well-behaved code. A validation agent runs exploit attempts, and some findings are in the container runtime or the kernel itself. For those workloads we use a sandboxed runtime or a lightweight virtual machine per task, so an escape lands in a disposable machine and not on the host that runs the rest of the factory.
Everything is recorded. Every command, request and tool call goes to a log outside the environment, written as the run happens. That log is the reproduction evidence the playbook asks for, and it is also how you find out what an agent did when a run behaves strangely. We covered why the log has to sit outside the agent's reach in AI agent security is application security.
Which services to start with
The playbook's plan says to run priority services first and extend later. We pick the first three to five by asking three questions of each candidate.
- Is it exposed? Internet-facing services and anything that handles authentication or payments come first. The inventory workstream should already have this list.
- Does it produce findings? A service with a backlog of open scanner findings gives the factory work on day one, and gives you a before-and-after measure.
- How far is it from running in a container? A service that already has a working Compose file for local development is days of work. One that needs a hand-built virtual machine is weeks. Start with a mix, so the first workflow ships while the harder service is being prepared.
In most companies the work on the first services looks like this: write or repair the container build, move configuration out of the staging environment into files, write a seed script, replace two or three third-party calls with fakes, and add the network and credential controls. The playbook gives this workstream all six weeks for a reason.
The payoff outside security
Every one of those changes also helps the engineers who work on the service. A new hire runs it on day one. A bug report comes with a reproduction. Integration tests stop failing because staging was in a strange state. Teams that resist this work as a security project tend to accept it once it is framed as the development environment they have wanted for years, which is an honest framing, because it is the same environment.
It also makes the rest of the factory better. Validation produces fewer false positives when it can test, ownership routing has evidence to attach, and verification can replay the original exploit against the deployed fix and record the result.
Where to go from here
If you are planning a Defense Factory, look at your three most exposed services this week and ask whether an agent could start each one from a clean checkout without help. Where the answer is no, that gap is your first workstream.
We build these environments, and the rest of the loop around them, as part of our Defense Factory engagements. The one week Readiness Review includes this assessment for your priority services. To talk it through, book a call.
Sources
The workstreams, the description of dynamic validation and verification, the architecture functions and the Hugging Face incident are from OpenAI's Defense Factory playbook. The controls and the service selection criteria are our own practice.
This post is our own reading and is not endorsed by OpenAI. "Defense Factory" is the term used in OpenAI's playbook.
