[{"_1":2,"_32":-5,"_33":-5},"loaderData",{"_3":4},"routes/home",{"_5":6},"posts",[7],{"_8":9,"_30":31},"post",{"_10":11,"_12":13,"_14":-7,"_15":16,"_17":18,"_24":25,"_26":27,"_28":29},"title","12 Factor Agents","description","Principles for building reliable LLM applications that's stable enough for production","author","date","11/2/2025","tags",[19,20,21,22,23],"agents","llm","ai","best-practices","architecture","image","/12-factor-agents/040-4-components.png","slug","12-factor-agents","body","\n# 12 Factor Agents\n\nBased on the original 12 Factor Agents post [on GitHub](https://github.com/humanlayer/12-factor-agents). *In the spirit of [12 Factor Apps](https://12factor.net/).* \n\n## Introduction\n\nI've tried every agent framework out there, from the plug-and-play crew/langchains to the \"minimalist\" smolagents of the world to the \"production grade\" langraph, griptape, etc.\n\nI've talked to a lot of really strong founders, in and out of YC, who are all building really impressive things with AI. Most of them are rolling the stack themselves. I don't see a lot of frameworks in production customer-facing agents.\n\nI've been surprised to find that most of the products out there billing themselves as \"AI Agents\" are not all that agentic. A lot of them are mostly deterministic code, with LLM steps sprinkled in at just the right points to make the experience truly magical.\n\nAgents, at least the good ones, don't follow the \"here's your prompt, here's a bag of tools, loop until you hit the goal\" pattern. Rather, they are comprised of mostly just software.\n\nSo, I set out to answer:\n\n> **What are the principles we can use to build LLM-powered software that is actually good enough to put in the hands of production customers?**\n\n## Table of Contents\n\n- [How We Got Here: A Brief History of Software](#how-we-got-here-a-brief-history-of-software)\n- [Factor 1: Natural Language to Tool Calls](#factor-1-natural-language-to-tool-calls)\n- [Factor 2: Own Your Prompts](#factor-2-own-your-prompts)\n- [Factor 3: Own Your Context Window](#factor-3-own-your-context-window)\n- [Factor 4: Tools Are Just Structured Outputs](#factor-4-tools-are-just-structured-outputs)\n- [Factor 5: Unify Execution State and Business State](#factor-5-unify-execution-state-and-business-state)\n- [Factor 6: Launch/Pause/Resume with Simple APIs](#factor-6-launchpauseresume-with-simple-apis)\n- [Factor 7: Contact Humans with Tool Calls](#factor-7-contact-humans-with-tool-calls)\n- [Factor 8: Own Your Control Flow](#factor-8-own-your-control-flow)\n- [Factor 9: Compact Errors into Context Window](#factor-9-compact-errors-into-context-window)\n- [Factor 10: Small, Focused Agents](#factor-10-small-focused-agents)\n- [Factor 11: Trigger from Anywhere](#factor-11-trigger-from-anywhere)\n- [Factor 12: Make Your Agent a Stateless Reducer](#factor-12-make-your-agent-a-stateless-reducer)\n\n---\n\n## How We Got Here: A Brief History of Software\n\n### You don't have to listen to me\n\nWhether you're new to agents or an ornery old veteran like me, I'm going to try to convince you to throw out most of what you think about AI Agents, take a step back, and rethink them from first principles. (spoiler alert if you didn't catch the OpenAI responses launch a few weeks back, but pushing MORE agent logic behind an API ain't it)\n\n\n## Agents are software, and a brief history thereof\n\nlet's talk about how we got here\n\n### 60 years ago\n\nWe're gonna talk a lot about Directed Graphs (DGs) and their Acyclic friends, DAGs. I'll start by pointing out that...well...software is a directed graph. There's a reason we used to represent programs as flow charts.\n\n![010-software-dag](/12-factor-agents/010-software-dag.png)\n\n### 20 years ago\n\nAround 20 years ago, we started to see DAG orchestrators become popular. We're talking classics like [Airflow](https://airflow.apache.org/), [Prefect](https://www.prefect.io/), some predecessors, and some newer ones like ([dagster](https://dagster.io/), [inggest](https://www.inngest.com/), [windmill](https://www.windmill.dev/)). These followed the same graph pattern, with the added benefit of observability, modularity, retries, administration, etc.\n\n![015-dag-orchestrators](/12-factor-agents/015-dag-orchestrators.png)\n\n### 10-15 years ago\n\nWhen ML models started to get good enough to be useful, we started to see DAGs with ML models sprinkled in. You might imagine steps like \"summarize the text in this column into a new column\" or \"classify the support issues by severity or sentiment\".\n\n![020-dags-with-ml](/12-factor-agents/020-dags-with-ml.png)\n\nBut at the end of the day, it's still mostly the same good old deterministic software.\n\n### The promise of agents\n\nI'm not the first [person to say this](https://youtu.be/Dc99-zTMyMg?si=bcT0hIwWij2mR-40&t=73), but my biggest takeaway when I started learning about agents, was that you get to throw the DAG away. Instead of software engineers coding each step and edge case, you can give the agent a goal and a set of transitions:\n\n![025-agent-dag](/12-factor-agents/025-agent-dag.png)\n\nAnd let the LLM make decisions in real time to figure out the path\n\n![026-agent-dag-lines](/12-factor-agents/026-agent-dag-lines.png)\n\nThe promise here is that you write less software, you just give the LLM the \"edges\" of the graph and let it figure out the nodes. You can recover from errors, you can write less code, and you may find that LLMs find novel solutions to problems.\n\n### Agents as loops\n\nPut another way, you've got this loop consisting of 3 steps:\n\n1. LLM determines the next step in the workflow, outputting structured json (\"tool calling\")\n2. Deterministic code executes the tool call\n3. The result is appended to the context window\n4. repeat until the next step is determined to be \"done\"\n\n```python\ninitial_event = {\"message\": \"...\"}\ncontext = [initial_event]\nwhile True:\n next_step = await llm.determine_next_step(context)\n context.append(next_step)\n\n if (next_step.intent === \"done\"):\n return next_step.final_answer\n\n result = await execute_step(next_step)\n context.append(result)\n```\n\nOur initial context is just the starting event (maybe a user message, maybe a cron fired, maybe a webhook, etc),\nand we ask the llm to choose the next step (tool) or to determine that we're done.\n\nHere's a multi-step example:\n\n![027-agent-loop-animation](/12-factor-agents/027-agent-loop-animation.gif)\n\nAnd the \"materialized\" DAG that was generated would look something like:\n\n![027-agent-loop-dag](/12-factor-agents/027-agent-loop-dag.png)\n\n### The problem with this \"loop until you solve it\" pattern\n\nThe biggest problems with this pattern:\n\n- Agents get lost when the context window gets too long - they spin out trying the same broken approach over and over again\n- literally thats it, but that's enough to kneecap the approach\n\nEven if you haven't hand-rolled an agent, you've probably seen this long-context problem in working with agentic coding tools. They just get lost after a while and you need to start a new chat.\n\nI'll even perhaps posit something I've heard in passing quite a bit, and that YOU probably have developed your own intuition around:\n\n> ### **Even as models support longer and longer context windows, you'll ALWAYS get better results with a small, focused prompt and context**\n\nMost builders I've talked to **pushed the \"tool calling loop\" idea to the side** when they realized that anything more than 10-20 turns becomes a big mess that the LLM can't recover from. Even if the agent gets it right 90% of the time, that's miles away from \"good enough to put in customer hands\". Can you imagine a web app that crashed on 10% of page loads?\n\n### What actually works - micro agents\n\nOne thing that I **have** seen in the wild quite a bit is taking the agent pattern and sprinkling it into a broader more deterministic DAG.\n\n![028-micro-agent-dag](/12-factor-agents/028-micro-agent-dag.png)\n\nYou might be asking - \"why use agents at all in this case?\" - we'll get into that shortly, but basically, having language models managing well-scoped sets of tasks makes it easy to incorporate live human feedback, translating it into workflow steps without spinning out into context error loops.\n\n> #### having language models managing well-scoped sets of tasks makes it easy to incorporate live human feedback...without spinning out into context error loops\n\n### A real life micro agent\n\nHere's an example of how deterministic code might run one micro agent responsible for handling the human-in-the-loop steps for deployment.\n\n![029-deploybot-high-level](/12-factor-agents/029-deploybot-high-level.png)\n\n* **Human** Merges PR to GitHub main branch\n* **Deterministic Code** Deploys to staging env\n* **Deterministic Code** Runs end-to-end (e2e) tests against staging\n* **Deterministic Code** Hands to agent for prod deployment, with initial context: \"deploy SHA 4af9ec0 to production\"\n* **Agent** calls `deploy_frontend_to_prod(4af9ec0)`\n* **Deterministic code** requests human approval on this action\n* **Human** Rejects the action with feedback \"can you deploy the backend first?\"\n* **Agent** calls `deploy_backend_to_prod(4af9ec0)`\n* **Deterministic code** requests human approval on this action\n* **Human** approves the action\n* **Deterministic code** executed the backend deployment\n* **Agent** calls `deploy_frontend_to_prod(4af9ec0)`\n* **Deterministic code** requests human approval on this action\n* **Human** approves the action\n* **Deterministic code** executed the frontend deployment\n* **Agent** determines that the task was completed successfully, we're done!\n* **Deterministic code** run the end-to-end tests against production\n* **Deterministic code** task completed, OR pass to rollback agent to review failures and potentially roll back\n\n![033-deploybot-animation](/12-factor-agents/033-deploybot.gif)\n\nThis example is based on a real life [OSS agent we've shipped to manage our deployments at Humanlayer](https://github.com/got-agents/agents/tree/main/deploybot-ts) - here is a real conversation I had with it last week:\n\n![035-deploybot-conversation](/12-factor-agents/035-deploybot-conversation.png)\n\n\nWe haven't given this agent a huge pile of tools or tasks. The primary value in the LLM is parsing the human's plaintext feedback and proposing an updated course of action. We isolate tasks and contexts as much as possible to keep the LLM focused on a small, 5-10 step workflow.\n\n### So what's an agent really?\n\n- **prompt** - tell an LLM how to behave, and what \"tools\" it has available. The output of the prompt is a JSON object that describe the next step in the workflow (the \"tool call\" or \"function call\").\n- **switch statement** - based on the JSON that the LLM returns, decide what to do with it.\n- **accumulated context** - store the list of steps that have happened and their results\n- **for loop** - until the LLM emits some sort of \"Terminal\" tool call (or plaintext response), add the result of the switch statement to the context window and ask the LLM to choose the next step.\n\n![040-4-components](/12-factor-agents/040-4-components.png)\n\nIn the \"deploybot\" example, we gain a couple benefits from owning the control flow and context accumulation:\n\n- In our **switch statement** and **for loop**, we can hijack control flow to pause for human input or to wait for completion of long-running tasks\n- We can trivially serialize the **context** window for pause+resume\n- In our **prompt**, we can optimize the heck out of how we pass instructions and \"what happened so far\" to the LLM\n\n---\n\n## Factor 1: Natural Language to Tool Calls\n\nOne of the most common patterns in agent building is to convert natural language to structured tool calls. This is a powerful pattern that allows you to build agents that can reason about tasks and execute them.\n\n![110-natural-language-tool-calls](/12-factor-agents/110-natural-language-tool-calls.png)\n\nThis pattern, when applied atomically, is the simple translation of a phrase like\n\n> can you create a payment link for $750 to Terri for sponsoring the february AI tinkerers meetup?\n\nto a structured object that describes a Stripe API call like\n\n```json\n{\n \"function\": {\n \"name\": \"create_payment_link\",\n \"parameters\": {\n \"amount\": 750,\n \"customer\": \"cust_128934ddasf9\",\n \"product\": \"prod_8675309\",\n \"price\": \"prc_09874329fds\",\n \"quantity\": 1,\n \"memo\": \"Hey Jeff - see below for the payment link for the february ai tinkerers meetup\"\n }\n }\n}\n```\n\n**Note**: in reality the stripe API is a bit more complex, a [real agent that does this](https://github.com/dexhorthy/mailcrew) ([video](https://www.youtube.com/watch?v=f_cKnoPC_Oo)) would list customers, list products, list prices, etc to build this payload with the proper ids, or include those ids in the prompt/context window (we'll see below how those are kinda the same thing though!)\n\nFrom there, deterministic code can pick up the payload and do something with it.\n\n```python\n# The LLM takes natural language and returns a structured object\nnextStep = await llm.determineNextStep(\n \"\"\"\n create a payment link for $750 to Jeff\n for sponsoring the february AI tinkerers meetup\n \"\"\"\n )\n\n# Handle the structured output based on its function\nif nextStep.function == 'create_payment_link':\n stripe.paymentlinks.create(nextStep.parameters)\n return # or whatever you want, see below\nelif nextStep.function == 'something_else':\n # ... more cases\n pass\nelse: # the model didn't call a tool we know about\n # do something else\n pass\n```\n\n**NOTE**: While a full agent would then receive the API call result and loop with it, eventually returning something like\n\n> I've successfully created a payment link for $750 to Terri for sponsoring the february AI tinkerers meetup. Here's the link: https://buy.stripe.com/test_1234567890\n\n**Instead**, We're actually going to skip that step here, and save it for another factor, which you may or may not want to also incorporate (up to you!)\n\n---\n\n## Factor 2: Own Your Prompts\n\nDon't outsource your prompt engineering to a framework.\n\n![120-own-your-prompts](/12-factor-agents/120-own-your-prompts.png)\n\nBy the way, [this is far from novel advice:](https://hamel.dev/blog/posts/prompt/)\n\nSome frameworks provide a \"black box\" approach like this:\n\n```python\nagent = Agent(\n role=\"...\",\n goal=\"...\",\n personality=\"...\",\n tools=[tool1, tool2, tool3]\n)\n\ntask = Task(\n instructions=\"...\",\n expected_output=OutputModel\n)\n\nresult = agent.run(task)\n```\n\nThis is great for pulling in some TOP NOTCH prompt engineering to get you started, but it is often difficult to tune and/or reverse engineer to get exactly the right tokens into your model.\n\nInstead, own your prompts and treat them as first-class code:\n\n```rust\nfunction DetermineNextStep(thread: string) -> DoneForNow | ListGitTags | DeployBackend | DeployFrontend | RequestMoreInformation {\n prompt #\"\n {{ _.role(\"system\") }}\n\n You are a helpful assistant that manages deployments for frontend and backend systems.\n You work diligently to ensure safe and successful deployments by following best practices\n and proper deployment procedures.\n\n Before deploying any system, you should check:\n - The deployment environment (staging vs production)\n - The correct tag/version to deploy\n - The current system status\n\n You can use tools like deploy_backend, deploy_frontend, and check_deployment_status\n to manage deployments. For sensitive deployments, use request_approval to get\n human verification.\n\n Always think about what to do first, like:\n - Check current deployment status\n - Verify the deployment tag exists\n - Request approval if needed\n - Deploy to staging before production\n - Monitor deployment progress\n\n {{ _.role(\"user\") }}\n\n {{ thread }}\n\n What should the next step be?\n \"#\n}\n```\n\n(the above example uses [BAML](https://github.com/boundaryml/baml) to generate the prompt, but you can do this with any prompt engineering tool you want, or even just template it manually)\n\nIf the signature looks a little funny, we'll get to that in [factor 4 - tools are just structured outputs](#factor-4-tools-are-just-structured-outputs)\n\n```typescript\nfunction DetermineNextStep(thread: string) -> DoneForNow | ListGitTags | DeployBackend | DeployFrontend | RequestMoreInformation {\n```\n\nKey benefits of owning your prompts:\n\n1. **Full Control**: Write exactly the instructions your agent needs, no black box abstractions\n2. **Testing and Evals**: Build tests and evals for your prompts just like you would for any other code\n3. **Iteration**: Quickly modify prompts based on real-world performance\n4. **Transparency**: Know exactly what instructions your agent is working with\n5. **Role Hacking**: take advantage of APIs that support nonstandard usage of user/assistant roles - for example, the now-deprecated non-chat flavor of OpenAI \"completions\" API. This includes some so-called \"model gaslighting\" techniques\n\nRemember: Your prompts are the primary interface between your application logic and the LLM.\n\nHaving full control over your prompts gives you the flexibility and prompt control you need for production-grade agents.\n\nI don't know what's the best prompt, but I know you want the flexibility to be able to try EVERYTHING.\n\n---\n\n## Factor 3: Own Your Context Window\n\nYou don't necessarily need to use standard message-based formats for conveying context to an LLM.\n\n> #### At any given point, your input to an LLM in an agent is \"here's what's happened so far, what's the next step\"\n\nEverything is context engineering. [LLMs are stateless functions](https://thedataexchange.media/baml-revolution-in-ai-engineering/) that turn inputs into outputs. To get the best outputs, you need to give them the best inputs.\n\nCreating great context means:\n\n- The prompt and instructions you give to the model\n- Any documents or external data you retrieve (e.g. RAG)\n- Any past state, tool calls, results, or other history\n- Any past messages or events from related but separate histories/conversations (Memory)\n- Instructions about what sorts of structured data to output\n\n### on context engineering\n\nThis guide is all about getting as much as possible out of today's models. Notably not mentioned are:\n\n- Changes to models parameters like temperature, top_p, frequency_penalty, presence_penalty, etc.\n- Training your own completion or embedding models\n- Fine-tuning existing models\n\nAgain, I don't know what's the best way to hand context to an LLM, but I know you want the flexibility to be able to try EVERYTHING.\n\n#### Standard vs Custom Context Formats\n\nMost LLM clients use a standard message-based format like this:\n\n```yaml\n[\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant...\"\n },\n {\n \"role\": \"user\",\n \"content\": \"Can you deploy the backend?\"\n },\n {\n \"role\": \"assistant\",\n \"content\": null,\n \"tool_calls\": [\n {\n \"id\": \"1\",\n \"name\": \"list_git_tags\",\n \"arguments\": \"{}\"\n }\n ]\n },\n {\n \"role\": \"tool\",\n \"name\": \"list_git_tags\",\n \"content\": \"{\\\"tags\\\": [{\\\"name\\\": \\\"v1.2.3\\\", \\\"commit\\\": \\\"abc123\\\", \\\"date\\\": \\\"2024-03-15T10:00:00Z\\\"}, {\\\"name\\\": \\\"v1.2.2\\\", \\\"commit\\\": \\\"def456\\\", \\\"date\\\": \\\"2024-03-14T15:30:00Z\\\"}, {\\\"name\\\": \\\"v1.2.1\\\", \\\"commit\\\": \\\"abe033d\\\", \\\"date\\\": \\\"2024-03-13T09:15:00Z\\\"}]}\",\n \"tool_call_id\": \"1\"\n }\n]\n```\n\nWhile this works great for most use cases, if you want to really get THE MOST out of today's LLMs, you need to get your context into the LLM in the most token- and attention-efficient way you can.\n\nAs an alternative to the standard message-based format, you can build your own context format that's optimized for your use case. For example, you can use custom objects and pack/spread them into one or more user, system, assistant, or tool messages as makes sense.\n\nHere's an example of putting the whole context window into a single user message:\n```yaml\n\n[\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant...\"\n },\n {\n \"role\": \"user\",\n \"content\": |\n Here's everything that happened so far:\n\n \n From: @alex\n Channel: #deployments\n Text: Can you deploy the backend?\n \n\n \n intent: \"list_git_tags\"\n \n\n \n tags:\n - name: \"v1.2.3\"\n commit: \"abc123\"\n date: \"2024-03-15T10:00:00Z\"\n - name: \"v1.2.2\"\n commit: \"def456\"\n date: \"2024-03-14T15:30:00Z\"\n - name: \"v1.2.1\"\n commit: \"ghi789\"\n date: \"2024-03-13T09:15:00Z\"\n \n\n what's the next step?\n }\n]\n```\n\nThe model may infer that you're asking it `what's the next step` by the tool schemas you supply, but it never hurts to roll it into your prompt template.\n\n### code example\n\nWe can build this with something like:\n\n```python\n\nclass Thread:\n events: List[Event]\n\nclass Event:\n # could just use string, or could be explicit - up to you\n type: Literal[\"list_git_tags\", \"deploy_backend\", \"deploy_frontend\", \"request_more_information\", \"done_for_now\", \"list_git_tags_result\", \"deploy_backend_result\", \"deploy_frontend_result\", \"request_more_information_result\", \"done_for_now_result\", \"error\"]\n data: ListGitTags | DeployBackend | DeployFrontend | RequestMoreInformation |\n ListGitTagsResult | DeployBackendResult | DeployFrontendResult | RequestMoreInformationResult | string\n\ndef event_to_prompt(event: Event) -> str:\n data = event.data if isinstance(event.data, str) \\\n else stringifyToYaml(event.data)\n\n return f\"<{event.type}>\\n{data}\\n\"\n\n\ndef thread_to_prompt(thread: Thread) -> str:\n return '\\n\\n'.join(event_to_prompt(event) for event in thread.events)\n```\n\n#### Example Context Windows\n\nHere's how context windows might look with this approach:\n\n**Initial Slack Request:**\n```xml\n\n From: @alex\n Channel: #deployments\n Text: Can you deploy the latest backend to production?\n\n```\n\n**After Listing Git Tags:**\n```xml\n\n From: @alex\n Channel: #deployments\n Text: Can you deploy the latest backend to production?\n Thread: []\n\n\n\n intent: \"list_git_tags\"\n\n\n\n tags:\n - name: \"v1.2.3\"\n commit: \"abc123\"\n date: \"2024-03-15T10:00:00Z\"\n - name: \"v1.2.2\"\n commit: \"def456\"\n date: \"2024-03-14T15:30:00Z\"\n - name: \"v1.2.1\"\n commit: \"ghi789\"\n date: \"2024-03-13T09:15:00Z\"\n\n```\n\n**After Error and Recovery:**\n```xml\n\n From: @alex\n Channel: #deployments\n Text: Can you deploy the latest backend to production?\n Thread: []\n\n\n\n intent: \"deploy_backend\"\n tag: \"v1.2.3\"\n environment: \"production\"\n\n\n\n error running deploy_backend: Failed to connect to deployment service\n\n\n\n intent: \"request_more_information_from_human\"\n question: \"I had trouble connecting to the deployment service, can you provide more details and/or check on the status of the service?\"\n\n\n\n data:\n response: \"I'm not sure what's going on, can you check on the status of the latest workflow?\"\n\n```\n\nFrom here your next step might be:\n\n```python\nnextStep = await determine_next_step(thread_to_prompt(thread))\n```\n\n```python\n{\n \"intent\": \"get_workflow_status\",\n \"workflow_name\": \"tag_push_prod.yaml\",\n}\n```\n\nThe XML-style format is just one example - the point is you can build your own format that makes sense for your application. You'll get better quality if you have the flexibility to experiment with different context structures and what you store vs. what you pass to the LLM.\n\nKey benefits of owning your context window:\n\n1. **Information Density**: Structure information in ways that maximize the LLM's understanding\n2. **Error Handling**: Include error information in a format that helps the LLM recover. Consider hiding errors and failed calls from context window once they are resolved.\n3. **Safety**: Control what information gets passed to the LLM, filtering out sensitive data\n4. **Flexibility**: Adapt the format as you learn what works best for your use case\n5. **Token Efficiency**: Optimize context format for token efficiency and LLM understanding\n\nContext includes: prompts, instructions, RAG documents, history, tool calls, memory\n\nRemember: The context window is your primary interface with the LLM. Taking control of how you structure and present information can dramatically improve your agent's performance.\n\n### Don't take it from me\n\nAbout 2 months after 12-factor agents was published, context engineering started to become a pretty popular term.\n\nThere's also a quite good [Context Engineering Cheat Sheet](https://x.com/lenadroid/status/1943685060785524824) from [@lenadroid](https://x.com/lenadroid) from July 2025.\n\nRecurring theme here: I don't know what's the best approach, but I know you want the flexibility to be able to try EVERYTHING.\n\n---\n\n## Factor 4: Tools Are Just Structured Outputs\n\nTools don't need to be complex. At their core, they're just structured output from your LLM that triggers deterministic code.\n\n![140-tools-are-just-structured-outputs](/12-factor-agents/140-tools-are-just-structured-outputs.png)\n\nFor example, lets say you have two tools `CreateIssue` and `SearchIssues`. To ask an LLM to \"use one of several tools\" is just to ask it to output JSON we can parse into an object representing those tools.\n\n```python\n\nclass Issue:\n title: str\n description: str\n team_id: str\n assignee_id: str\n\nclass CreateIssue:\n intent: \"create_issue\"\n issue: Issue\n\nclass SearchIssues:\n intent: \"search_issues\"\n query: str\n what_youre_looking_for: str\n```\n\nThe pattern is simple:\n1. LLM outputs structured JSON\n3. Deterministic code executes the appropriate action (like calling an external API)\n4. Results are captured and fed back into the context\n\nThis creates a clean separation between the LLM's decision-making and your application's actions. The LLM decides what to do, but your code controls how it's done. Just because an LLM \"called a tool\" doesn't mean you have to go execute a specific corresponding function in the same way every time.\n\nIf you recall our switch statement from above\n\n```python\nif nextStep.intent == 'create_payment_link':\n stripe.paymentlinks.create(nextStep.parameters)\n return # or whatever you want, see below\nelif nextStep.intent == 'wait_for_a_while':\n # do something monadic idk\nelse: #... the model didn't call a tool we know about\n # do something else\n```\n\n**Note**: there has been a lot said about the benefits of \"plain prompting\" vs. \"tool calling\" vs. \"JSON mode\" and the performance tradeoffs of each. We'll link some resources to that stuff soon, but not gonna get into it here. See [Prompting vs JSON Mode vs Function Calling vs Constrained Generation vs SAP](https://www.boundaryml.com/blog/schema-aligned-parsing), [When should I use function calling, structured outputs, or JSON mode?](https://www.vellum.ai/blog/when-should-i-use-function-calling-structured-outputs-or-json-mode#:~:text=We%20don%27t%20recommend%20using%20JSON,always%20use%20Structured%20Outputs%20instead) and [OpenAI JSON vs Function Calling](https://docs.llamaindex.ai/en/stable/examples/llm/openai_json_vs_function_calling/).\n\nThe \"next step\" might not be as atomic as just \"run a pure function and return the result\". You unlock a lot of flexibility when you think of \"tool calls\" as just a model outputting JSON describing what deterministic code should do. Put this together with [factor 8 own your control flow](#factor-8-own-your-control-flow).\n\n---\n\n## Factor 5: Unify Execution State and Business State\n\nEven outside the AI world, many infrastructure systems try to separate \"execution state\" from \"business state\". For AI apps, this might involve complex abstractions to track things like current step, next step, waiting status, retry counts, etc. This separation creates complexity that may be worthwhile, but may be overkill for your use case.\n\nAs always, it's up to you to decide what's right for your application. But don't think you *have* to manage them separately.\n\nMore clearly:\n\n- **Execution state**: current step, next step, waiting status, retry counts, etc.\n- **Business state**: What's happened in the agent workflow so far (e.g. list of OpenAI messages, list of tool calls and results, etc.)\n\nIf possible, SIMPLIFY - unify these as much as possible.\n\n![155-unify-state](/12-factor-agents/155-unify-state-animation.gif)\n\nIn reality, you can engineer your application so that you can infer all execution state from the context window. In many cases, execution state (current step, waiting status, etc.) is just metadata about what has happened so far.\n\nYou may have things that can't go in the context window, like session ids, password contexts, etc, but your goal should be to minimize those things. By embracing [factor 3](#factor-3-own-your-context-window) you can control what actually goes into the LLM\n\nThis approach has several benefits:\n\n1. **Simplicity**: One source of truth for all state\n2. **Serialization**: The thread is trivially serializable/deserializable\n3. **Debugging**: The entire history is visible in one place\n4. **Flexibility**: Easy to add new state by just adding new event types\n5. **Recovery**: Can resume from any point by just loading the thread\n6. **Forking**: Can fork the thread at any point by copying some subset of the thread into a new context / state ID\n7. **Human Interfaces and Observability**: Trivial to convert a thread into a human-readable markdown or a rich Web app UI\n\n---\n\n## Factor 6: Launch/Pause/Resume with Simple APIs\n\nAgents are just programs, and we have things we expect from how to launch, query, resume, and stop them.\n\n![165-pause-resume](/12-factor-agents/165-pause-resume-animation.gif)\n\nIt should be easy for users, apps, pipelines, and other agents to launch an agent with a simple API.\n\nAgents and their orchestrating deterministic code should be able to pause an agent when a long-running operation is needed.\n\nExternal triggers like webhooks should enable agents to resume from where they left off without deep integration with the agent orchestrator.\n\nClosely related to [factor 5 - unify execution state and business state](#factor-5-unify-execution-state-and-business-state) and [factor 8 - own your control flow](#factor-8-own-your-control-flow), but can be implemented independently.\n\n**Note** - often AI orchestrators will allow for pause and resume, but not between the moment of tool selection and tool execution. See also [factor 7 - contact humans with tool calls](#factor-7-contact-humans-with-tool-calls) and [factor 11 - trigger from anywhere, meet users where they are](#factor-11-trigger-from-anywhere).\n\n---\n\n## Factor 7: Contact Humans with Tool Calls\n\nBy default, LLM APIs rely on a fundamental HIGH-STAKES token choice: Are we returning plaintext content, or are we returning structured data?\n\n![170-contact-humans-with-tools](/12-factor-agents/170-contact-humans-with-tools.png)\n\nYou're putting a lot of weight on that choice of first token, which, in the `the weather in tokyo` case, is\n\n> \"the\"\n\nbut in the `fetch_weather` case, it's some special token to denote the start of a JSON object.\n\n> |JSON>\n\nYou might get better results by having the LLM *always* output json, and then declare it's intent with some natural language tokens like `request_human_input` or `done_for_now` (as opposed to a \"proper\" tool like `check_weather_in_city`).\n\nAgain, you might not get any performance boost from this, but you should experiment, and ensure you're free to try weird stuff to get the best results.\n\n```python\n\nclass Options:\n urgency: Literal[\"low\", \"medium\", \"high\"]\n format: Literal[\"free_text\", \"yes_no\", \"multiple_choice\"]\n choices: List[str]\n\n# Tool definition for human interaction\nclass RequestHumanInput:\n intent: \"request_human_input\"\n question: str\n context: str\n options: Options\n\n# Example usage in the agent loop\nif nextStep.intent == 'request_human_input':\n thread.events.append({\n type: 'human_input_requested',\n data: nextStep\n })\n thread_id = await save_state(thread)\n await notify_human(nextStep, thread_id)\n return # Break loop and wait for response to come back with thread ID\nelse:\n # ... other cases\n```\n\nLater, you might receive a webhook from a system that handles slack, email, sms, or other events.\n\n```python\n\n@app.post('/webhook')\ndef webhook(req: Request):\n thread_id = req.body.threadId\n thread = await load_state(thread_id)\n thread.events.push({\n type: 'response_from_human',\n data: req.body\n })\n # ... simplified for brevity, you likely don't want to block the web worker here\n next_step = await determine_next_step(thread_to_prompt(thread))\n thread.events.append(next_step)\n result = await handle_next_step(thread, next_step)\n # todo - loop or break or whatever you want\n\n return {\"status\": \"ok\"}\n```\n\nThe above includes patterns from [factor 5 - unify execution state and business state](#factor-5-unify-execution-state-and-business-state), [factor 8 - own your control flow](#factor-8-own-your-control-flow), [factor 3 - own your context window](#factor-3-own-your-context-window), and [factor 4 - tools are just structured outputs](#factor-4-tools-are-just-structured-outputs), and several others.\n\nIf we were using the XML-y formatted from [factor 3 - own your context window](#factor-3-own-your-context-window), our context window after a few turns might look like this:\n\n```xml\n\n(snipped for brevity)\n\n\n From: @alex\n Channel: #deployments\n Text: Can you deploy backend v1.2.3 to production?\n Thread: []\n\n\n\n intent: \"request_human_input\"\n question: \"Would you like to proceed with deploying v1.2.3 to production?\"\n context: \"This is a production deployment that will affect live users.\"\n options: {\n urgency: \"high\"\n format: \"yes_no\"\n }\n\n\n\n response: \"yes please proceed\"\n approved: true\n timestamp: \"2024-03-15T10:30:00Z\"\n user: \"alex@company.com\"\n\n\n\n intent: \"deploy_backend\"\n tag: \"v1.2.3\"\n environment: \"production\"\n\n\n\n status: \"success\"\n message: \"Deployment v1.2.3 to production completed successfully.\"\n timestamp: \"2024-03-15T10:30:00Z\"\n\n```\n\nBenefits:\n\n1. **Clear Instructions**: Tools for different types of human contact allow for more specificity from the LLM\n2. **Inner vs Outer Loop**: Enables agents workflows **outside** of the traditional chatGPT-style interface, where the control flow and context initialization may be `Agent->Human` rather than `Human->Agent` (think, agents kicked off by a cron or an event)\n3. **Multiple Human Access**: Can easily track and coordinate input from different humans through structured events\n4. **Multi-Agent**: Simple abstraction can be easily extended to support `Agent->Agent` requests and responses\n5. **Durable**: Combined with [factor 6 - launch/pause/resume with simple APIs](#factor-6-launchpauseresume-with-simple-apis), this makes for durable, reliable, and introspectable multiplayer workflows\n\n[More on Outer Loop Agents over here](https://theouterloop.substack.com/p/openais-realtime-api-is-a-step-towards)\n\n![175-outer-loop-agents](/12-factor-agents/175-outer-loop-agents.png)\n\nWorks great with [factor 11 - trigger from anywhere, meet users where they are](#factor-11-trigger-from-anywhere)\n\n---\n\n## Factor 8: Own Your Control Flow\n\nIf you own your control flow, you can do lots of fun things.\n\n![180-control-flow](/12-factor-agents/180-control-flow.png)\n\nBuild your own control structures that make sense for your specific use case. Specifically, certain types of tool calls may be reason to break out of the loop and wait for a response from a human or another long-running task like a training pipeline. You may also want to incorporate custom implementation of:\n\n- summarization or caching of tool call results\n- LLM-as-judge on structured output\n- context window compaction or other [memory management](#factor-3-own-your-context-window)\n- logging, tracing, and metrics\n- client-side rate limiting\n- durable sleep / pause / \"wait for event\"\n\nThe below example shows three possible control flow patterns:\n\n- request_clarification: model asked for more info, break the loop and wait for a response from a human\n- fetch_git_tags: model asked for a list of git tags, fetch the tags, append to context window, and pass straight back to the model\n- deploy_backend: model asked to deploy a backend, this is a high-stakes thing, so break the loop and wait for human approval\n\n```python\ndef handle_next_step(thread: Thread):\n\n while True:\n next_step = await determine_next_step(thread_to_prompt(thread))\n\n # inlined for clarity - in reality you could put\n # this in a method, use exceptions for control flow, or whatever you want\n if next_step.intent == 'request_clarification':\n thread.events.append({\n type: 'request_clarification',\n data: nextStep,\n })\n\n await send_message_to_human(next_step)\n await db.save_thread(thread)\n # async step - break the loop, we'll get a webhook later\n break\n elif next_step.intent == 'fetch_open_issues':\n thread.events.append({\n type: 'fetch_open_issues',\n data: next_step,\n })\n\n issues = await linear_client.issues()\n\n thread.events.append({\n type: 'fetch_open_issues_result',\n data: issues,\n })\n # sync step - pass the new context to the LLM to determine the NEXT next step\n continue\n elif next_step.intent == 'create_issue':\n thread.events.append({\n type: 'create_issue',\n data: next_step,\n })\n\n await request_human_approval(next_step)\n await db.save_thread(thread)\n # async step - break the loop, we'll get a webhook later\n break\n```\n\nThis pattern allows you to interrupt and resume your agent's flow as needed, creating more natural conversations and workflows.\n\n**Example** - the number one feature request I have for every AI framework out there is we need to be able to interrupt\na working agent and resume later, ESPECIALLY between the moment of tool **selection** and the moment of tool **invocation**.\n\nWithout this level of resumability/granularity, there's no way to review/approve the tool call before it runs, which means\nyou're forced to either:\n\n1. Pause the task in memory while waiting for the long-running thing to complete (think `while...sleep`) and restart it from the beginning if the process is interrupted\n2. Restrict the agent to only low-stakes, low-risk calls like research and summarization\n3. Give the agent access to do bigger, more useful things, and just yolo hope it doesn't screw up\n\nYou may notice this is closely related to [factor 5 - unify execution state and business state](#factor-5-unify-execution-state-and-business-state) and [factor 6 - launch/pause/resume with simple APIs](#factor-6-launchpauseresume-with-simple-apis), but can be implemented independently.\n\n---\n\n## Factor 9: Compact Errors into Context Window\n\nThis one is a little short but is worth mentioning. One of these benefits of agents is \"self-healing\" - for short tasks, an LLM might call a tool that fails. Good LLMs have a fairly good chance of reading an error message or stack trace and figuring out what to change in a subsequent tool call.\n\nMost frameworks implement this, but you can do JUST THIS without doing any of the other 11 factors. Here's an example:\n\n```python\nthread = {\"events\": [initial_message]}\n\nwhile True:\n next_step = await determine_next_step(thread_to_prompt(thread))\n thread[\"events\"].append({\n \"type\": next_step.intent,\n \"data\": next_step,\n })\n try:\n result = await handle_next_step(thread, next_step) # our switch statement\n except Exception as e:\n # if we get an error, we can add it to the context window and try again\n thread[\"events\"].append({\n \"type\": 'error',\n \"data\": format_error(e),\n })\n # loop, or do whatever else here to try to recover\n```\n\nYou may want to implement an errorCounter for a specific tool call, to limit to ~3 attempts of a single tool, or whatever other logic makes sense for your use case.\n\n```python\nconsecutive_errors = 0\n\nwhile True:\n\n # ... existing code ...\n\n try:\n result = await handle_next_step(thread, next_step)\n thread[\"events\"].append({\n \"type\": next_step.intent + '_result',\n data: result,\n })\n # success! reset the error counter\n consecutive_errors = 0\n except Exception as e:\n consecutive_errors += 1\n if consecutive_errors < 3:\n # do the loop and try again\n thread[\"events\"].append({\n \"type\": 'error',\n \"data\": format_error(e),\n })\n else:\n # break the loop, reset parts of the context window, escalate to a human, or whatever else you want to do\n break\n }\n}\n```\n\nHitting some consecutive-error-threshold might be a great place to [escalate to a human](#factor-7-contact-humans-with-tool-calls), whether by model decision or via deterministic takeover of the control flow.\n\n![195-factor-09-errors](/12-factor-agents/195-factor-09-errors.gif)\n\nBenefits:\n\n1. **Self-Healing**: The LLM can read the error message and figure out what to change in a subsequent tool call\n2. **Durable**: The agent can continue to run even if one tool call fails\n\nI'm sure you will find that if you do this TOO much, your agent will start to spin out and might repeat the same error over and over again.\n\nThat's where [factor 8 - own your control flow](#factor-8-own-your-control-flow) and [factor 3 - own your context building](#factor-3-own-your-context-window) come in - you don't need to just put the raw error back on, you can completely restructure how it's represented, remove previous events from the context window, or whatever deterministic thing you find works to get an agent back on track.\n\nBut the number one way to prevent error spin-outs is to embrace [factor 10 - small, focused agents](#factor-10-small-focused-agents).\n\n---\n\n## Factor 10: Small, Focused Agents\n\nRather than building monolithic agents that try to do everything, build small, focused agents that do one thing well. Agents are just one building block in a larger, mostly deterministic system.\n\n![1a0-small-focused-agents](/12-factor-agents/1a0-small-focused-agents.png)\n\nThe key insight here is about LLM limitations: the bigger and more complex a task is, the more steps it will take, which means a longer context window. As context grows, LLMs are more likely to get lost or lose focus. By keeping agents focused on specific domains with 3-10, maybe 20 steps max, we keep context windows manageable and LLM performance high.\n\n> #### As context grows, LLMs are more likely to get lost or lose focus\n\nBenefits of small, focused agents:\n\n1. **Manageable Context**: Smaller context windows mean better LLM performance\n2. **Clear Responsibilities**: Each agent has a well-defined scope and purpose\n3. **Better Reliability**: Less chance of getting lost in complex workflows\n4. **Easier Testing**: Simpler to test and validate specific functionality\n5. **Improved Debugging**: Easier to identify and fix issues when they occur\n\n### What if LLMs get smarter?\n\nDo we still need this if LLMs get smart enough to handle 100-step+ workflows?\n\ntl;dr yes. As agents and LLMs improve, they **might** naturally expand to be able to handle longer context windows. This means handling MORE of a larger DAG. This small, focused approach ensures you can get results TODAY, while preparing you to slowly expand agent scope as LLM context windows become more reliable. (If you've refactored large deterministic code bases before, you may be nodding your head right now).\n\n![1a5-agent-scope-grow](/12-factor-agents/1a5-agent-scope-grow.gif)\n\nBeing intentional about size/scope of agents, and only growing in ways that allow you to maintain quality, is key here. As the [team that built NotebookLM put it](https://open.substack.com/pub/swyx/p/notebooklm?selection=08e1187c-cfee-4c63-93c9-71216640a5f8&utm_campaign=post-share-selection&utm_medium=web):\n\n> I feel like consistently, the most magical moments out of AI building come about for me when I'm really, really, really just close to the edge of the model capability\n\nRegardless of where that boundary is, if you can find that boundary and get it right consistently, you'll be building magical experiences. There are many moats to be built here, but as usual, they take some engineering rigor.\n\n---\n\n## Factor 11: Trigger from Anywhere\n\nIf you're waiting for the [humanlayer](https://humanlayer.dev) pitch, you made it. If you're doing [factor 6 - launch/pause/resume with simple APIs](#factor-6-launchpauseresume-with-simple-apis) and [factor 7 - contact humans with tool calls](#factor-7-contact-humans-with-tool-calls), you're ready to incorporate this factor.\n\n![1b0-trigger-from-anywhere](/12-factor-agents/1b0-trigger-from-anywhere.png)\n\nEnable users to trigger agents from slack, email, sms, or whatever other channel they want. Enable agents to respond via the same channels.\n\nBenefits:\n\n- **Meet users where they are**: This helps you build AI applications that feel like real humans, or at the very least, digital coworkers\n- **Outer Loop Agents**: Enable agents to be triggered by non-humans, e.g. events, crons, outages, whatever else. They may work for 5, 20, 90 minutes, but when they get to a critical point, they can contact a human for help, feedback, or approval\n- **High Stakes Tools**: If you're able to quickly loop in a variety of humans, you can give agents access to higher stakes operations like sending external emails, updating production data and more. Maintaining clear standards gets you auditability and confidence in agents that [perform bigger better things](#what-if-llms-get-smarter)\n\n---\n\n## Factor 12: Make Your Agent a Stateless Reducer\n\nOkay so we're over 1000 lines of markdown at this point. This one is mostly just for fun.\n\n![1c0-stateless-reducer](/12-factor-agents/1c0-stateless-reducer.png)\n\n![1c5-agent-foldl](/12-factor-agents/1c5-agent-foldl.png)\n","url","/blog/12-factor-agents","actionData","errors"]