Harness engineering framework: six control responsibilities for reliable agents

Key takeaways

An agent harness manages tools, context, state, permissions, and execution around the model. Start with a simple system and add agent steps only when evaluation shows they are needed. Put sensitive actions behind deterministic permissions, hooks, and approvals.

A component list tells you what exists. It does not tell you which part of the system owns a failure or a consequential decision. This harness engineering framework treats architecture as six control responsibilities: context, execution, authority, verification, recovery, and observability. The taxonomy is a design aid, not an industry standard or a performance formula. Its value is practical. Each responsibility gets an owner, a mechanism, and a question you can test.

What is harness engineering in simple terms?

Harness engineering is the work of designing the runtime control layer around a model. An agent harness manages tools, context, state, permissions, and the execution loop. Those controls turn a language model into an agent that can take steps and act through tools.

The broadest definition can include everything in an AI agent except the model. That scope is too loose for architecture work. You have to narrow it to the agent category and workload you are building. A coding agent, for example, needs controls that match its tools, workspace, state, and execution path. The same named component may own a different responsibility in another system.

That is why harness engineering architecture sits at the intersection of context engineering, evaluation, observability, orchestration, safe autonomy, and software architecture. We find the term most useful when it names the controls around behavior, not every line of application code. For a deeper treatment of the category, start with our harness engineering glossary.

A harness engineering framework has six control responsibilities

Use the matrix to assign each control surface an owner before choosing components.

An agent can see a tool without being allowed to use it without review. Tool visibility determines what the agent can see, while approval checks govern actions that require review. Sensitive actions need deterministic permissions, hooks, and approvals instead of voluntary model compliance.

The right harness exposes the controls your workload needs

We judge LLM harnesses by the controls they expose. The useful choice is the one whose controls match your workload and remain inspectable when the system misbehaves. Look for simple design, visible planning steps, and agent-computer interfaces that are documented and tested. Those properties make the execution path easier to inspect than a stack of hidden conventions.

Frameworks can accelerate initial development. Production systems may benefit from fewer abstraction layers and more basic components, especially when an abstraction hides tool calls, state changes, or control flow. This is not an argument against frameworks. It is a reason to check whether you can see and change the controls that matter.

That gives you four practical questions for a harness choice. Can you see the plan before tools execute? Are the agent-computer interfaces documented and tested? Can you inspect the basic components below the framework? Does the design stay simple enough to explain the path from a model decision to a tool action? A framework that speeds the first build may still create work later if the production controls you need sit behind opaque layers.

Complexity raises the engineering burden. One failed step can redirect an agent's trajectory and produce an unpredictable outcome. Multi-agent research systems have required careful engineering, comprehensive testing, detailed prompt and tool design, robust operations, and close work across research, product, and engineering teams. That vendor experience does not establish a universal architecture, but it makes a useful selection test: the number of interacting steps should be matched by enough visibility and control to diagnose them.

If you are comparing AI agent frameworks, compare their control surfaces before their feature counts.

How do you perform harness engineering?

Start with the simplest prompt that can do the work, evaluate it comprehensively, and add multi-step agency only when the simpler approach falls short. This keeps the architecture tied to a demonstrated need. It also gives you a baseline against which added loops, tools, and state can be judged.

A useful outer harness has two goals. It should improve the probability of a correct initial result, and it should provide a feedback loop that can correct issues before human review. Those goals require controls on both sides of an action.

Feedforward guides act before execution. They anticipate behavior and steer the agent before it acts. In an implementation, that responsibility can hold the instructions and constraints that shape the coming step.

Feedback sensors act after execution. They observe what happened and help the agent self-correct, especially when they produce signals optimized for LLM consumption. The sensor has to return something the loop can use, not merely produce a log for a person.

The balance matters. Feedback without feedforward can leave an agent repeating mistakes. Feedforward without feedback can encode rules without showing whether they worked. Treat them as separate jobs with different timing: one guides the action, and the other measures what followed. Evaluation then tells you whether that loop earns its added complexity.

Our agent loop guide goes deeper on the runtime levers inside that cycle.

Risky actions need a pause point and a bounded workspace

Consider an action that could charge a card, delete data, or change production state. In the documented approval lifecycle, the run reaches an approval point and pauses. The system surfaces the interruption, accepts an approval or rejection, and resumes the same run from its saved RunState.

That lifecycle is concrete:

1. Reach the action that requires approval.
2. Pause the run.
3. Surface the interruption for review.
4. Accept approval or rejection.
5. Resume the same run from RunState.

The same product documentation assigns different work to guardrails, approvals, and lifecycle callbacks. Guardrails check requests and outputs. Approvals pause risky side effects. Lifecycle callbacks support logging, tracing, and audit events.

A sandbox creates another boundary. It can separate a trusted application from a workspace used for risky or stateful work involving scoped files, command execution, artifacts, or resumable state. Approval controls whether a reviewed action continues. The sandbox bounds where the work occurs.

Permissions define the reachable blast radius

Approval happens at a decision point, but permission determines what is reachable before that point. An agent's tool-invocation blast radius is the set of systems, data, and actions available through its identity. Measure that reach through runtime tool paths and permissions, not only the scope presented at login.

Data flow needs its own control. Tool and application programming interface calls can transmit sensitive data outside an internal boundary. Identity-bound policies, approved tool catalogs, and outbound allowlists can constrain those calls. A tool being available does not mean every destination should be available through it.

Authentication is the third part of the boundary. Agent access should use short-lived, scoped, identity-bound, and context-aware tokens, not broad persistent access. Together, runtime paths, outbound policy, and scoped identity define a more useful authority map than a single permission label.

An authority review therefore starts from actual runtime reach. Follow the agent identity through the tools it can invoke, the systems those tools can contact, the data those calls can carry, and the token attached to the request. That sequence can reveal a broader reachable surface than the initial login scope suggests. It also prevents one control from standing in for another. A scoped token limits authentication, while a tool catalog and an outbound allowlist constrain available calls and destinations.

This is where a harness turns intent into enforcement. The question is not only whether the agent was told what to do. It is what the runtime permits the agent to reach and what the system can stop deterministically.

Can you provide an example of harness engineering?

A tool-using coding agent is a straightforward example. An agent application plans steps and calls tools that can read files, run commands, or edit code. Harness engineering decides how those tools, the loop, and the working context are supplied and controlled around the model.

The Claude Agent SDK is one named, product-scoped implementation. Its Python and TypeScript packages supply tools, an agent loop, and context management. It shows the control layer in concrete terms: the model is only one part of the running application.

A separate internal beta reported another scale of application. In that software product, Codex wrote application logic, tests, continuous integration configuration, documentation, observability, and internal tooling. It shows that harness work can surround a broad software workload, with tests and observability living alongside application code.

The two examples answer different implementation questions. The software development kit makes tools, the loop, and context management visible as supplied parts of an agent application. The internal beta shows a software workload spanning code, tests, configuration, documentation, observability, and tooling. They give you concrete surfaces to locate in the system you are evaluating.

These examples are useful because they expose implementation surfaces. When you inspect an agent harness, look past the model call and locate the loop, tools, context, tests, and visibility around the work.

Why is the final response not enough to evaluate an agent?

The final response hides the path that produced it. Diagnosing a failure requires the sequence of decisions and actions behind that response. A plausible answer can still follow a poor plan, choose the wrong tool, pass incorrect arguments, or take an invalid path.

End-to-end evaluation measures task completion, step efficiency, latency, and cost. Those measures tell you what the run delivered and what resources it used. They do not replace trajectory evaluation.

The distinction becomes useful during diagnosis. Task completion can show that the expected result arrived. Step efficiency, latency, and cost describe the run at an end-to-end level. If one of those measures changes, the action sequence gives you the detail needed to investigate the path rather than guessing from the response alone.

Reasoning and action evaluation can inspect plan quality, plan adherence, tool selection, tool correctness, argument correctness, and path validity. These measures expose where execution departed from the intended behavior. A task-completion score and a valid-path measure answer different questions, so both can remain visible without being collapsed into one verdict.

Tracing records what the harness did. Evaluation judges whether that behavior was correct, safe, and efficient. Keeping those responsibilities separate gives you the raw sequence for diagnosis and a defined method for judging it.

Turn failures into an evaluation system

An observed failure becomes more useful when it becomes a test case. Early evaluation work can capture failures, turn them into regression tests, and replace subjective impressions with metrics. The test then checks whether a later change brings the same failure back.

Build the evaluation system around five jobs. First, preserve real failure cases as a regression set. Second, cover the lifecycle dimensions that matter: quality, performance, responsibility, cost, production monitoring, and human-in-the-loop validation. Third, add safety measures for injection resilience, policy adherence, and consistency across user groups.

Lifecycle measures widen the view beyond answer quality. Performance, responsibility, cost, production monitoring, and human-in-the-loop validation can sit beside quality as separate evaluation dimensions. Safety measures add another focused job by checking injection resilience, policy adherence, and consistency across user groups.

Fourth, test memory as a behavior, not just a storage feature. Memory evaluation methods include before-and-after baselines and transcript replay with and without context trimming. They can track dropped constraints, repeated tool calls, and protected context lost under token pressure.

Fifth, use traces to locate expensive model calls. Tracing model-call usage can reveal costly steps that are candidates for optimization. This cost view belongs beside task quality and path quality, not in place of them.

Memory tests and cost traces also answer different failure questions. Transcript replay can expose context behavior under trimming or token pressure. Usage traces can identify a model-call step carrying unusual cost. Keeping both available helps you investigate the part of the harness that actually changed.

The resulting system connects production failures to repeatable checks. It also keeps distinct questions visible: Did the agent complete the task? Did it follow a valid path? Did memory preserve the needed context? Did it stay within policy? Which model calls carried the cost?

Read our agent harness guide, then inspect one runtime path in your own system. Locate its context, tools, permissions, approval point, evaluation, recovery mode, and trace before you add another layer of agency.