--- title: "Reproducible Environments for Security Agents: The Defense Factory Workstream Most Teams Underestimate | Vindler Blog" description: "An agent can only prove a vulnerability is real if it can run the vulnerable service, attack it, and throw the copy away. OpenAI's Defense Factory playbook gives this one line, \"reproducible dev envs\", in its six-week plan. In our experience it is where the first build spends most of its time. Here is what reproducible means in practice, how to isolate an agent whose job is to attack things, and which three services to start with." url: "https://vindler.solutions/blog/reproducible-environments-security-agents" locale: "en" updatedAt: "2026-11-12" --- [Back to Blog](https://vindler.solutions/blog) Reproducible Environments for Security Agents: The Defense Factory Workstream Most Teams Underestimate [Defense Factory](https://vindler.solutions/blog?tag=Defense%20Factory)[Reproducible Environments](https://vindler.solutions/blog?tag=Reproducible%20Environments)[Dynamic Validation](https://vindler.solutions/blog?tag=Dynamic%20Validation)[AI Agents](https://vindler.solutions/blog?tag=AI%20Agents)[Sandboxing](https://vindler.solutions/blog?tag=Sandboxing)[Application Security](https://vindler.solutions/blog?tag=Application%20Security)[Docker](https://vindler.solutions/blog?tag=Docker)[Platform Engineering](https://vindler.solutions/blog?tag=Platform%20Engineering)[Security](https://vindler.solutions/blog?tag=Security) # Reproducible Environments for Security Agents: The Defense Factory Workstream Most Teams Underestimate An agent can only prove a vulnerability is real if it can run the vulnerable service, attack it, and throw the copy away. OpenAI's Defense Factory playbook gives this one line, "reproducible dev envs", in its six-week plan. In our experience it is where the first build spends most of its time. Here is what reproducible means in practice, how to isolate an agent whose job is to attack things, and which three services to start with. November 7, 2026 1 minute Share: _The Defense Factory workstream most teams underestimate._ OpenAI's [Defense Factory playbook](https://cdn.openai.com/defense-factory/downloads/defense-factory-playbook.pdf) splits its six-week plan into five workstreams. Four of them are about people and process: mobilizing the team, fixing urgent exposures, keeping an inventory of systems and owners, and turning investigations into skills. The fifth is a single line, "reproducible dev envs", owned by platform and application teams, with three steps across the six weeks: run priority services, make setup reproducible, extend as needed. That line carries more of the build than any other. In our [reading of the playbook](https://vindler.solutions/blog/how-to-build-a-defense-factory) we said it is where we expect first attempts to stall. This post explains why, and what to build. ## Why validation needs a running service The playbook's stage of dynamic validation reads: "Given candidate findings and a runnable application, agents inspect code, reassess exposure, and test suspected exploits in a controlled environment. They preserve reproduction evidence for confirmed vulnerabilities and check duplicates before creating approved issues." The phrase that matters is "a runnable application". A scanner reports that an endpoint may be open to an injection. Reading the code gets an agent part of the way. Proving it means sending the request and watching what the service does, and proving the fix means sending the same request after the patch and watching it fail. Without a running copy of the service, validation is an opinion, and the factory goes back to producing findings that engineers have to investigate themselves. That is the bottleneck the whole operation exists to remove. The same copy is needed at the other end of the loop. The playbook's verification stage retests the fix, and the reproduction evidence collected during validation is only useful if it can be replayed against the patched build in a known state. ## What "reproducible" means in practice An environment is reproducible when an agent can start it from a clean state, without a person, and get the same service every time. For a typical web service that comes down to five things. 1. **Pinned builds.** The service is built from a specific commit into an image with pinned base layers and dependency versions. "Latest" anywhere in the chain means two runs of the same test can disagree. 2. **Configuration as files.** Every setting the service reads at start lives in the repository or is generated from it. If a setting only exists in a staging environment that someone configured by hand two years ago, the agent cannot reproduce it, and neither can a new engineer. 3. **Seeded data.** A database built from migrations and a seed script, with enough synthetic records to exercise the code paths that matter: users with different roles, an admin, a tenant boundary to cross. Never a copy of production data. An agent that attacks a database will, on a good day, find a way to read all of it. 4. **Stubbed dependencies.** Payment providers, email, identity providers and other third parties replaced by local fakes that behave like the real thing for the calls the service makes. The agent should never be able to reach a real vendor API from a test environment. 5. **Start and teardown in minutes.** Each validation task gets a fresh environment and destroys it afterwards. An environment shared between tasks carries state from one test into the next, and a validation result that depends on what the previous agent did is worthless as evidence. None of this is specific to security or to agents. Docker Compose, development containers and test container libraries have covered most of it for years, and teams that already run integration tests in containers on every pull request are most of the way there. The teams that struggle are the ones whose services only work in a shared staging environment, which is common in companies that grew quickly. ## Isolating an agent whose job is to attack things A validation agent is designed to do harmful things to the service in front of it. The environment has to make sure that service is the only thing it can harm. The playbook's architecture names the pieces: agents run in "isolated, reproducible environments", the platform "connects code, secrets and models", "enforces policy", and "monitors agents, hosts and network activity". Concretely, we put four controls around every validation run. **Network egress is closed by default.** The environment can reach the services inside it and the model endpoint, and nothing else. No internet, no internal network, no cloud metadata endpoint. Exfiltration through a prompt-injected instruction and accidental attacks on a real system both need a network route, and the simplest control is not having one. **No long-lived secrets inside the environment.** The agent never holds a cloud key or a repository token. When it needs one, a credential proxy outside the environment issues a scoped, short-lived credential for that task, or performs the action on the agent's behalf and returns the result. The July 2026 incident the playbook describes, in which research models compromised parts of Hugging Face's systems, turned on exposed credentials and chained access. An environment with nothing to steal breaks that chain at the first link. **Stronger isolation for untrusted code.** Running a container on a shared host is enough for well-behaved code. A validation agent runs exploit attempts, and some findings are in the container runtime or the kernel itself. For those workloads we use a sandboxed runtime or a lightweight virtual machine per task, so an escape lands in a disposable machine and not on the host that runs the rest of the factory. **Everything is recorded.** Every command, request and tool call goes to a log outside the environment, written as the run happens. That log is the reproduction evidence the playbook asks for, and it is also how you find out what an agent did when a run behaves strangely. We covered why the log has to sit outside the agent's reach in [AI agent security is application security](https://vindler.solutions/blog/ai-agent-security-is-application-security). ## Which services to start with The playbook's plan says to run priority services first and extend later. We pick the first three to five by asking three questions of each candidate. 1. **Is it exposed?** Internet-facing services and anything that handles authentication or payments come first. The inventory workstream should already have this list. 2. **Does it produce findings?** A service with a backlog of open scanner findings gives the factory work on day one, and gives you a before-and-after measure. 3. **How far is it from running in a container?** A service that already has a working Compose file for local development is days of work. One that needs a hand-built virtual machine is weeks. Start with a mix, so the first workflow ships while the harder service is being prepared. In most companies the work on the first services looks like this: write or repair the container build, move configuration out of the staging environment into files, write a seed script, replace two or three third-party calls with fakes, and add the network and credential controls. The playbook gives this workstream all six weeks for a reason. ## The payoff outside security Every one of those changes also helps the engineers who work on the service. A new hire runs it on day one. A bug report comes with a reproduction. Integration tests stop failing because staging was in a strange state. Teams that resist this work as a security project tend to accept it once it is framed as the development environment they have wanted for years, which is an honest framing, because it is the same environment. It also makes the rest of the factory better. Validation produces fewer false positives when it can test, ownership routing has evidence to attach, and verification can replay the original exploit against the deployed fix and record the result. ## Where to go from here If you are planning a Defense Factory, look at your three most exposed services this week and ask whether an agent could start each one from a clean checkout without help. Where the answer is no, that gap is your first workstream. We build these environments, and the rest of the loop around them, as part of our [Defense Factory engagements](https://vindler.solutions/hire/defense-factory). The one week Readiness Review includes this assessment for your priority services. To talk it through, [book a call](https://vindler.solutions/book). ## Sources The workstreams, the description of dynamic validation and verification, the architecture functions and the Hugging Face incident are from OpenAI's [Defense Factory playbook](https://cdn.openai.com/defense-factory/downloads/defense-factory-playbook.pdf). The controls and the service selection criteria are our own practice. _This post is our own reading and is not endorsed by OpenAI. "Defense Factory" is the term used in OpenAI's playbook._ Share: Carlos Dutra, founder of Vindler Solutions ### Carlos Dutra Founder of Vindler Solutions, where I help organizations put AI into production and design the operating model around it. I write about AI adoption, agent architectures, and what actually changes inside a company once these systems ship. Completed Leading the AI-Driven Organization at MIT Sloan Executive Education. [Get in Touch](https://vindler.solutions/contact) ## Related Posts [How to Build a Defense Factory: OpenAI's Playbook, Read by Engineers Who Build Agents Defense FactoryOpenAIAI AgentsApplication SecurityVulnerability ManagementAgentic SecurityCodexClaude CodeMCPSecurity ### How to Build a Defense Factory: OpenAI's Playbook, Read by Engineers Who Build Agents In September OpenAI published the Defense Factory playbook: an agent-first operation that connects your security tools, skills and reproducible environments to fix vulnerabilities continuously. It explains what to build and gives a six-week plan, and it leaves the engineering to you. We build agent systems for a living, so here is what each part of the playbook means in practice, where the work actually is, and how to keep the agents themselves from becoming the next incident. November 6, 2026 1 minute](https://vindler.solutions/blog/how-to-build-a-defense-factory)[AI Agent Security Is Application Security: PHP2Shell, the LiteLLM Backdoor, and the Agent That Tried to Backdoor a GitHub Project AI Agent SecurityAgentic AIPrompt InjectionSupply Chain SecurityPHP2ShellMCP SecuritySandboxingSecurityAgentsProduction ### AI Agent Security Is Application Security: PHP2Shell, the LiteLLM Backdoor, and the Agent That Tried to Backdoor a GitHub Project On October 21, PHP shipped emergency patches for two unauthenticated remote code execution bugs that Palo Alto Networks counts on more than 22 million exposed instances. Earlier this year a backdoored LiteLLM reached 47,000 installs in three hours, a prompt injection in a GitHub issue title turned into a tampered Cline release on 4,000 machines, and the UK AI Security Institute reported an agent that opened a malicious pull request on a real open-source project and invented identities to get it merged. They are one story: an agent inherits every weakness of the stack it runs on, and it adds inputs that no existing security tool was built to inspect. Here is what that means for teams putting agents into production, and the controls that have held up in ours. October 26, 2026 1 minute](https://vindler.solutions/blog/ai-agent-security-is-application-security)[AI Game QA in 2026: What Is Proven and Where to Start AIGame DevelopmentQAGame TestingMobile GamesAI AgentsGaming ### AI Game QA in 2026: What Is Proven and Where to Start AI agents that play a mobile game from plain-language test steps have been on sale since 2025. We read the vendor pages, papers and studio statements published since 2024 to see which parts of game QA they can take over today, what the published numbers support, and where a live game should start. November 12, 2026 1 minute](https://vindler.solutions/blog/ai-game-qa-automation-2026) ### Subscribe to our newsletter Get notified when we publish new posts on AI development, AWS, and software engineering. ## Structured data ```json {"@context":"https://schema.org","@type":"Organization","@id":"https://vindler.solutions/#organization","name":"Vindler Solutions","alternateName":"Vindler","legalName":"Vindler Desenvolvimento de Sistemas de IA Ltda - EPP","url":"https://vindler.solutions","logo":"https://vindler.solutions/logo.png","description":"Vindler is a global AI engineering consultancy specializing in custom AI solutions, LLM integration, and enterprise-grade AI systems.","foundingDate":"2025-06-02","foundingLocation":{"@type":"Place","name":"São Paulo, Brazil"},"taxID":"61.102.920/0001-05","email":"contact@vindler.solutions","telephone":"+1-646-846-5339","numberOfEmployees":{"@type":"QuantitativeValue","minValue":1,"maxValue":10},"areaServed":[{"@type":"Country","name":"United States"},{"@type":"Place","name":"Europe"},{"@type":"Country","name":"Brazil"}],"knowsAbout":["Multi-agent systems","Retrieval-augmented generation","Voice AI agents","LangChain","LangGraph","LiveKit","AWS Bedrock","Model Context Protocol","LLM observability and evaluation"],"founder":{"@type":"Person","@id":"https://vindler.solutions/about#carlos-dutra","name":"Carlos Dutra","jobTitle":"Founder and CEO","url":"https://vindler.solutions/about","sameAs":["https://www.linkedin.com/in/carlosdutra","https://github.com/cdutr"]},"contactPoint":{"@type":"ContactPoint","contactType":"Customer Support","email":"contact@vindler.solutions","telephone":"+1-646-846-5339","url":"https://vindler.solutions/contact","availableLanguage":["English","Portuguese","Spanish"]},"address":[{"@type":"PostalAddress","addressLocality":"São Paulo","addressRegion":"SP","addressCountry":"BR","streetAddress":"Av. Paulista, 1636"},{"@type":"PostalAddress","addressLocality":"New York","addressRegion":"NY","addressCountry":"US","streetAddress":"524 Broadway","postalCode":"10012"}],"sameAs":["https://www.linkedin.com/company/vindler-tech/","https://github.com/vindler-solutions","https://clutch.co/profile/vindler-solutions","https://www.wikidata.org/wiki/Q141018871","https://www.crunchbase.com/organization/vindler-solutions"],"memberOf":{"@type":"Organization","name":"Brazilian-American Chamber of Commerce, Inc.","url":"https://brazilcham.com/","sameAs":"https://www.linkedin.com/company/brazilian-american-chamber-of-commerce-inc./"},"hasOfferCatalog":{"@type":"OfferCatalog","name":"AI Engineering Services","itemListElement":[{"@type":"Offer","itemOffered":{"@type":"Service","name":"AI Engineering","description":"Custom AI systems development from prototyping to deployment."}},{"@type":"Offer","itemOffered":{"@type":"Service","name":"Custom ChatBots & Agents","description":"Intelligent, multilingual, and domain-specific conversational AI."}},{"@type":"Offer","itemOffered":{"@type":"Service","name":"AI Architecture Consultancy","description":"Designing scalable, secure, and ethical AI infrastructure for enterprises."}},{"@type":"Offer","itemOffered":{"@type":"Service","name":"Training & Team Building","description":"Upskill your team or recruit top-tier AI talent."}}]}} ``` ```json {"@context":"https://schema.org","@type":"BlogPosting","headline":"Reproducible Environments for Security Agents: The Defense Factory Workstream Most Teams Underestimate","description":"An agent can only prove a vulnerability is real if it can run the vulnerable service, attack it, and throw the copy away. OpenAI's Defense Factory playbook gives this one line, \"reproducible dev envs\", in its six-week plan. In our experience it is where the first build spends most of its time. Here is what reproducible means in practice, how to isolate an agent whose job is to attack things, and which three services to start with.","image":"https://vindler.solutions/images/blog/reproducible-environments-security-agents-nano-v3.jpg","datePublished":"2026-11-07","dateModified":"2026-11-07","author":{"@type":"Person","@id":"https://vindler.solutions/about#carlos-dutra","name":"Carlos Dutra","jobTitle":"Founder and CEO, Vindler","url":"https://vindler.solutions/about"},"publisher":{"@type":"Organization","@id":"https://vindler.solutions/#organization","name":"Vindler","logo":{"@type":"ImageObject","url":"https://vindler.solutions/logo.png"}},"mainEntityOfPage":{"@type":"WebPage","@id":"https://vindler.solutions/blog/reproducible-environments-security-agents"}} ```