Designing recoverable agent workflows

Most agent demos have a very convenient property: nothing goes wrong.

The model gets a task, calls a few tools, writes some code, runs the tests, and finishes.

Real software engineering workflows do not behave like that.

A model times out halfway through an implementation. A test command fails. The process crashes after files were written but before state was persisted. A developer closes the terminal. An agent spends fifteen minutes researching a bug and then loses its context. A verification step discovers that the implementation was wrong and sends the workflow back to repair.

Once an agent is allowed to perform work that takes more than a few seconds, failure stops being an edge case.

It becomes part of the execution model.

This was one of the problems that pushed us toward building our own agent runner.

We were not trying to build another chat interface around an LLM. We wanted something that could run software engineering workflows for feature implementation, bug hunting, documentation, research, testing, and review.

More importantly, we wanted those workflows to survive failure.

That requirement changed the architecture quite a bit.

The problem with the autonomous loop

The obvious way to build a coding agent is something like this:

while (!done) {
  const response = await model.generate({
    messages,
    tools,
  })
 
  messages.push(response)
 
  await executeToolCalls(response.toolCalls)
}

This architecture is surprisingly capable.

Give the model access to the file system, shell, tests, and Git, and it can accomplish a lot.

The problem is that the conversation becomes the workflow.

The model decides what happens next. The transcript contains most of the useful state. Progress is represented implicitly through messages such as:

I inspected the authentication implementation.
 
I found three possible causes.
 
I am going to modify session.ts next.

This works until the process disappears.

At that point you have to answer an uncomfortable question:

What exactly was the workflow doing?

Not what the model was talking about.

What was actually completed?

Which files were authoritative?

Which decisions had already been approved?

Which tests had run?

Which tests had failed?

Was the last tool call completed before the process died?

Is it safe to execute it again?

A transcript is useful evidence, but it is a bad database.

That distinction became one of the main design principles of our agent runner.

The model can reason about the workflow.

It does not own the workflow.

Making the workflow explicit

We use XState as the orchestration layer.

Instead of asking the model to continuously decide what phase it is in, the runner has explicit states.

A simplified feature workflow looks roughly like this:

Each state has a limited responsibility.

Research can inspect the repository.

Specification can produce a proposed implementation.

Implementation can modify product code.

Verification runs deterministic checks.

Repair receives concrete failures and attempts to resolve them.

The model still makes many decisions inside those states, but it cannot invent a new state transition just because it feels appropriate.

If the machine says implementation cannot begin before specification approval, then no amount of prompting changes that.

This becomes especially useful during recovery because the runner has a concrete answer to the question:

Where are we?

We are not somewhere inside a 40,000 token conversation.

We are in verification, with these persisted artifacts, after these events, with this exact machine snapshot.

That is a much better starting point.

Durable state instead of durable conversations

Every run gets its own directory.

Conceptually, it looks like this:

.agent-work/
  JOB-142/
    run.json
    events.ndjson
    state.snapshot.json
 
    prompt.md
    spec.md
    approved-spec.sha256
 
    research/
      repository-analysis.md
      architecture-notes.md
 
    implementation/
      summary.md
 
    verification/
      typecheck.log
      lint.log
      tests.log
      result.json
 
    subagents/
      database-analysis.md
      frontend-analysis.md

We deliberately keep important information outside the model transcript.

run.json contains metadata about the run.

state.snapshot.json contains the persisted XState snapshot.

events.ndjson is an append only record of important workflow events.

Artifacts contain the actual outputs produced by different phases.

Verification logs contain evidence of what was executed.

The model session can disappear entirely and we should still be able to understand the run.

That is the standard we are aiming for.

For example, the event log might contain:

{"type":"RUN_CREATED","at":"2026-09-05T13:02:11Z"}
{"type":"RESEARCH_STARTED","at":"2026-09-05T13:02:14Z"}
{"type":"RESEARCH_COMPLETED","at":"2026-09-05T13:07:42Z"}
{"type":"SPEC_CREATED","at":"2026-09-05T13:10:08Z"}
{"type":"SPEC_APPROVED","at":"2026-09-05T13:14:33Z"}
{"type":"IMPLEMENTATION_STARTED","at":"2026-09-05T13:14:34Z"}
{"type":"IMPLEMENTATION_COMPLETED","at":"2026-09-05T13:28:17Z"}
{"type":"STATIC_CHECK_FAILED","at":"2026-09-05T13:29:02Z"}

If the process crashes immediately after that final event, the next process does not need to reconstruct history from natural language.

It already knows what happened.

Recovery is not the same as retry

This was an important distinction for us.

Retry means:

Run the operation again.

Recovery means:

Determine what already happened, reconstruct the workflow, and continue from a safe point.

Those are very different operations.

Imagine an implementation agent performs these steps:

1. modifies schema.ts
2. modifies router.ts
3. runs a migration generator
4. runs the type checker
5. writes an implementation summary

The process crashes during step four.

Simply retrying the entire agent invocation is dangerous.

The model might modify the same files again.

The migration generator might produce another migration.

The new session might arrive at a different implementation.

The existing changes might confuse its repository analysis.

Instead, recovery needs to start from durable facts.

The repository already contains modifications.

The state machine says we were implementing.

The event log says implementation had started.

There is no completion event.

The runner can inspect the workspace, restore the state machine, and decide which work needs to be continued.

This is much closer to how durable job systems work than how chatbots work.

Persist before crossing important boundaries

One practical rule emerged from this:

Important transitions must leave evidence.

Suppose specification approval changes what the agent is allowed to do.

Before approval, the agent can write inside the run directory but cannot modify product code.

After approval, implementation tools become available.

That boundary matters.

When a user approves a specification, we hash it:

const hash = sha256(specContents)
 
await writeFile('.agent-work/JOB-142/approved-spec.sha256', hash)

The approval event is also recorded before implementation begins.

Now consider recovery several hours later.

The runner can compare the current specification against the approved hash.

If the specification changed after approval, we know the current document is not what the user approved.

Without that tiny piece of persisted information, approval would exist only as something somebody remembers happening.

The same principle applies elsewhere.

Before entering implementation, persist approval.

Before reporting verification success, persist verification output.

Before declaring a run complete, persist the final state.

The more expensive or irreversible a transition is, the more important this becomes.

Idempotency matters more than clever prompting

Recoverable systems need operations that can safely happen more than once.

Agent workflows are no exception.

Consider this sequence:

await writeSnapshot(snapshot)
await appendEvent({ type: 'IMPLEMENTATION_COMPLETED' })

The process could die between those two writes.

Now the snapshot and event log disagree.

You cannot eliminate every possible partial failure, but you can design operations so that reconciliation is possible.

For example, artifacts can use deterministic paths.

Instead of creating:

research-1.md
research-2.md
research-final-final.md

the research step writes:

research/repository-analysis.md

Writing it again replaces the artifact for that phase.

Verification commands can also be derived from repository state rather than from previous conversation state.

If three packages changed, the runner computes which checks should run.

It does not ask the model whether tests were probably already executed.

This makes repeating verification relatively cheap and safe.

const plan = buildVerificationPlan({
  changedFiles,
  affectedWorkspaces,
})
 
for (const command of plan.commands) {
  await executeAndPersist(command)
}

The model can help repair a failure.

It should not be the authority that declares the repository valid.

Verification belongs to the runner

This became another strong boundary in the architecture.

An implementation agent should never be able to finish by saying:

Everything looks good.

That statement has almost no value.

The runner determines whether everything looks good.

It detects changed files and affected workspaces, then builds a verification plan.

Depending on the repository, that might include:

type checking
linting
format checking
framework specific checks
unit tests
integration tests
React Doctor
build validation

The output is persisted.

A failure becomes an event:

STATIC_CHECK_FAILED

or:

TESTS_FAILED

The machine then transitions to repair.

This loop is useful for recovery because verification is reproducible.

If the runner crashes halfway through verification, we do not need to trust a partially completed model conversation.

We can run the deterministic verification plan again.

There is a general lesson here:

The parts of an agent system that can be deterministic probably should be deterministic.

LLMs are valuable where interpretation is necessary.

They are a poor replacement for a state transition, a SHA256 hash, an exit code, or a test runner.

Subagents should return artifacts, not conversations

We also use specialized subagents.

For example, a bug hunt might launch separate agents to investigate database behavior, frontend state synchronization, caching, or test coverage.

A tempting implementation is to return their entire conversations to the parent agent.

We avoid that.

A subagent is expected to write a durable artifact and return something closer to:

type SubagentResult = {
  status: 'completed' | 'failed'
  artifactPath: string
  importantNote?: string
}

The useful output might live here:

.agent-work/JOB-142/subagents/cache-analysis.md

This has several benefits.

First, parent context stays small.

Second, another agent can inspect the same artifact later.

Third, the subagent session can disappear without losing its work.

Fourth, recovery does not depend on recreating a particular conversation tree.

It also creates a much cleaner interface between agents.

The parent does not need to know how the research happened.

It needs the result.

That is a normal software engineering principle, but it is surprisingly easy to forget when building with language models.

The filesystem becomes part of the protocol

Once artifacts are durable, paths stop being implementation details.

They become interfaces.

A research agent knows where research belongs.

A specification state knows where to find the research.

An implementation agent knows which specification is authoritative.

Verification knows where to write logs.

The CLI can expose commands such as:

agent-runner status JOB-142
agent-runner resume JOB-142
agent-runner approve-spec JOB-142
agent-runner request-changes JOB-142
agent-runner verify JOB-142

resume is particularly interesting.

Its job is not to send the last conversation back to the model.

Its job is to reconstruct the execution environment.

Conceptually:

const run = await loadRun(runId)
const snapshot = await loadMachineSnapshot(runId)
const machine = createActor(workflowMachine, {
  snapshot,
})
 
await reconcileWorkspace(run)
await reconcileArtifacts(run)
await reconcileEvents(run)
 
machine.start()

The exact implementation is more complicated, but the mental model is simple.

Persist the workflow.

Reconstruct it.

Reconcile external state.

Continue.

Recovery also requires limiting authority

There is another side to recoverability that is less obvious.

The more freedom an agent has, the harder recovery becomes.

If a research agent can edit arbitrary product files, install packages, modify Git history, run migrations, and spawn more agents, then reconstructing its partial execution is extremely difficult.

We therefore expose tools based on workflow state.

During research, an agent might have:

read files
search repository
inspect Git history
run safe commands
write research artifacts

During implementation it gains mutation capabilities.

During verification, the runner owns most execution.

This is not only a safety mechanism.

It reduces the number of possible states the system can end up in.

That makes failures easier to reason about.

A crashed research agent cannot leave half implemented product code behind because it was never allowed to modify it.

This is one reason I think capability design is underrated in agent architecture.

Permissions are also a state management tool.

What happens when the process dies?

Consider a real workflow:

research completed
specification created
specification approved
implementation completed
verification started
type checking passed
tests started
process crashed

A naive agent might restart from the original prompt.

Our runner already knows much more than that.

It has:

the machine snapshot
the approved specification hash
the implementation currently in the worktree
the completed research artifacts
the verification plan
the successful type check log
the event history

Recovery can therefore be conservative.

We can restore the workflow around verification and execute the required checks again.

We do not need another research pass.

We do not need another specification.

We definitely do not need another implementation unless verification finds a problem.

That sounds obvious when written down.

But it only becomes possible if those earlier phases were designed to leave durable outputs.

Recoverability cannot easily be added at the end.

It affects how every phase communicates.

The model is a worker, not the operating system

The biggest architectural change for us was conceptual.

Initially it is natural to think of an agent as the system.

The model receives the task.

The model decides what to inspect.

The model calls tools.

The model decides whether the job is finished.

Once we started caring about deterministic workflows and recovery, that model stopped making sense.

The runner is the system.

The state machine owns the lifecycle.

Artifacts own durable knowledge.

The event log owns history.

Verification tools own objective checks.

The permission layer owns capabilities.

The model is a worker used when the system encounters a problem that benefits from reasoning.

That can still be a very powerful worker.

During implementation it might have considerable freedom to explore the repository and write code.

During bug investigation it might form hypotheses and design experiments.

During review it might find architectural problems that static analysis cannot detect.

But it operates inside a workflow that exists independently of its current context window.

That is what makes the workflow recoverable.

What we are still working through

There are still difficult cases.

A shell command can have external side effects that cannot simply be inspected from the repository.

A process can crash at the worst possible point between an operation and the event recording that operation.

Parallel subagents introduce their own reconciliation problems.

Some commands are naturally idempotent. Others are not.

Worktrees make repository isolation much easier, but external systems such as databases, GitHub, cloud infrastructure, or third party APIs require additional strategies.

There is also a tradeoff between persistence and complexity.

You can record everything, but eventually you have created an event sourced workflow engine when all you wanted was an agent that fixes TypeScript bugs.

We are trying to keep the durable protocol relatively small.

Persist the state that changes what the system is allowed to do.

Persist outputs that would be expensive to recreate.

Persist evidence used to determine success.

Persist enough history to explain how the current state was reached.

Everything else can remain transient.

The broader lesson

Long running agents look less like chatbots the longer they run.

They start looking like distributed systems.

They have partial failure.

They have retries.

They have durable state.

They have external side effects.

They have workers that disappear.

They have operations that need to be idempotent.

They have events that can be observed in different orders.

They have state that must be reconciled after a crash.

The interesting part is that none of this requires making the model itself more autonomous.

In our case, we moved in the opposite direction.

The model became less responsible for orchestration and more focused on the parts it is actually good at.

Reasoning.

Investigating.

Designing.

Implementing.

Reviewing.

The boring parts went back to code.

That has made the system more predictable, easier to inspect, and much easier to imagine running for hours instead of minutes.

A recoverable agent workflow should be able to answer four questions after the entire process disappears:

What was completed?
 
What state were we in?
 
What side effects already happened?
 
What is the safest next action?

If the answer to those questions exists only inside the model's previous conversation, the workflow is not really recoverable.

It is just a chat session with a retry button.