Open source AI harness: inspect the architecture before you choose

Key takeaways

• Choose for the workload, not the label.
• Inspect state, authority, traces, evaluation, recovery, portability, maintenance, and cost.
• A boundary checkpoint may replay completed work. Step-level durable execution can continue from the exact unfinished operation.

What is an open source AI harness?

An open source AI harness is the durable system around a model. It supplies project context, tools, permissions, workflow, and proof that the work is correct. The model is one part of the agent. The harness manages its execution environment, not just the prompt.

That boundary matters because an acting system needs more than instructions. A harness brings project context, tools, permissions, workflow, and proof around the model. The useful comparison is the surrounding architecture, even when repositories use nearby labels such as runtime, framework, or orchestrator.

Open source changes what a builder may control. Some published MIT or Apache 2.0 repositories provide access to the code. A candidate may run on infrastructure controlled by its operator, including air-gapped infrastructure, or accept a chosen model provider through the operator's API key. These are repository-specific properties to verify. They can matter to teams seeking transparency, customization, or less vendor lock-in, but they are not assumptions to make about every candidate.

Treat code, infrastructure, and provider choice as three inspection tasks. For code control, locate the published repository and its license. For infrastructure control, establish whether the candidate can run on systems you operate, including an air-gapped environment when that is required. For provider choice, determine whether the harness accepts the model provider API key you choose. A candidate may need to satisfy one, two, or all three goals, depending on why you are considering open source.

The ownership goal should also be explicit. Transparency asks whether the system can be understood. Customization asks whether it can be changed for the workload. Reduced lock-in asks what remains under the operator's control. The category can support those goals, but the repository under review still has to show how.

We therefore treat the label as an invitation to inspect the control surface. A credible candidate should make its context, tools, permissions, execution path, and checks legible enough to evaluate. The execution environment around the model is the thing you will operate after the demo works.

Are open-source agent harnesses available?

Yes. Open-source implementations exist for researchers and builders who want to understand how production AI agents work. But the available projects do not all occupy the same layer, so availability is only the start of the comparison.

Open Multi-Agent packages an orchestration runtime with tools, memory, checkpoints, traces, a command-line interface, and an offline Run Viewer. It can store traces locally for inspection in that viewer. A separate OpenTelemetry package is optional when centralized monitoring is needed. This is a concrete example of a repository spanning execution, state, recovery boundaries, and local inspection.

That package creates two specific diligence paths. If the work can be inspected locally, examine the traces in the offline Run Viewer. If monitoring must be centralized, inspect the optional OpenTelemetry package. The repository description establishes both paths without making them interchangeable.

Symphony occupies a more specific control-plane role. It is an open-source Codex-orchestration specification. Its agent orchestrator turns a project-management board, such as Linear, into a control plane for coding agents. That scope is useful when the workload begins with board-managed coding work, but it should not be mistaken for proof that the same architecture covers every agent workload.

For Symphony, the immediate question is whether a project-management board is the right control plane for the coding work. That is a different decision from choosing an orchestration runtime that already packages memory, checkpoints, traces, tools, and a local viewer.

HAL represents another layer. It is a holistic agent leaderboard for centralized and reproducible agent evaluation. Its repository was archived by its owner on July 1, 2026, and is now read-only. The evaluation purpose and the maintenance state are both material, but they answer different questions. One explains what the project was built to do. The other tells you what kind of repository relationship is possible now.

HAL presents a useful maintenance check because the repository state is unambiguous. Its evaluation role remains clear, while its archived, read-only status changes how it can enter an operating shortlist. Record both purpose and maintenance state instead of allowing the word "harness" to collapse them into one score.

These examples are not a ranking. They show why an open source agent harness search can surface runtimes, orchestrators, control planes, and evaluation infrastructure together. Before comparing candidates, decide which layer the workload requires. If you are still assembling candidates, our workload-first open-source agent harness shortlist can help narrow the field. Then bring each repository back to the same operating questions.

How should you choose the best AI coding harness?

The best AI coding harness is the one whose control contract fits the coding workload you must operate. Apply the same seven criteria to every candidate. Score each row from 0 to 2: 0 means the documented design conflicts with your requirement, 1 means it partly fits or leaves a material gap, and 2 means the requirement is met. A blank marks a row that current repository material does not answer.

Here is one clearly hypothetical worked row. Your workload requires an interrupted tool call to continue from the unfinished operation. Candidate A documents only boundary checkpoints, so "state and recovery" scores 0. Candidate B documents step-level journaling and continuation from the unfinished call, so it scores 2. A candidate that mentions recovery but does not explain its restart granularity stays blank until its current documentation settles the question.

The setup row deserves real weight. Open-source harness setup can take from minutes to hours depending on provider configuration. Cost also needs a workload-shaped measure. One published cost-performance harness ranks benchmark resolution per dollar and offers a tunable value score that blends capability and price. That is a useful method when its benchmark matches the decision. It does not turn a leaderboard into a universal answer.

For a more architecture-specific comparison, use the same rows while reviewing open-source agentic AI framework control contracts. Recovery deserves special attention because the word "checkpoint" can hide two very different restart behaviors.

What happens when the process dies?

Suppose a tool makes three calls and the process dies after the second. A boundary checkpoint returns execution to a stored boundary and restarts from there. If the completed calls were not recorded individually, recovery can run all three calls again. Parallel work has the same exposure: depending on the checkpoint implementation, some or all completed workers may redo their work.

Step-level durable execution uses a finer record. It persists each model call, tool call, sleep, and remote procedure call. Completed steps can return their journaled results, so the run continues from the exact unfinished step. The distinction is not whether a candidate advertises checkpoints. It is whether recovery restarts a boundary or resumes an individual operation. The boundary and step-level recovery mechanisms lead to different work after the same crash.

Write the required recovery point in operational terms. "Restart from the last boundary" accepts boundary replay. "Continue from the unfinished call" requires an individual record for completed operations. For parallel work, identify whether completed workers return recorded results or execute again after recovery. This wording makes the failure test concrete before a candidate is scored.

Durable state can also appear as an intent that survives outside the dead process. In the AXME crash-recovery example, an intent remains in PostgreSQL with an IN_PROGRESS state after the agent dies. When the agent listens again, the intent is redelivered and proceeds to COMPLETED.

Inspect a candidate by tracing one interrupted operation. Ask where the durable record lives, what has already been journaled, what is redelivered, and which completed work runs again. The answer describes the recovery contract more precisely than a checkpoint checkbox.

Tracing and evaluation close different gaps

Tracing tells you what the agent and harness actually did. Evaluation tells you whether that work was acceptable. A repository that supports one does not answer the question handled by the other.

Start with the trace question: what happened during execution? The record should let you inspect the observed path through the agent and harness. This is the practical job of agent observability. It makes the trajectory available for investigation instead of leaving only a final response.

Then ask the evaluation question at the right level: was this individual step acceptable? Was the complete trajectory acceptable? What about the whole run or a longer session? Evaluations can judge each of those units. A final output check alone does not substitute for a judgment about the steps and trajectory that produced it.

Useful evaluations usually grow from real traces and recurring failures. That creates a grounded engineering loop. A trace exposes observed behavior. A recurring failure becomes material for an evaluation. The evaluation then judges later steps, trajectories, runs, or sessions against that concern. Our agent evaluation notes use this broader unit of analysis because acting systems have consequential paths, not just answers.

Choose the evaluation unit from the failure you need to judge. A questionable operation creates a step-level question. An unacceptable path creates a trajectory question. The complete execution creates a run question. Behavior that must be judged over a session creates a session question. The harness should make the relevant unit available to the evaluation.

Improvement should remain a governed engineering loop, not automatic prompt rewriting. That boundary matters when a trace reveals something undesirable. The response is not simply to let the system rewrite its own instructions. The change belongs inside an engineering process that can judge the resulting behavior.

When comparing repositories, pair one observed-execution question with one acceptance question. "What did it do?" tests tracing. "Was the step, trajectory, run, or session acceptable?" tests evaluation. Keeping both questions visible prevents a detailed trace from being mistaken for a quality judgment.

How much authority should the harness hold?

The harness should hold only the authority its workload and production evidence justify. Sensitive actions need deterministic permissions, hooks, and approval mechanisms. Those controls should not depend on the model choosing to comply.

A production review can be organized around seven gates:

1. Inventory the system being considered.
2. Establish a behavioral baseline.
3. Compare declared access with observed access.
4. Set a blast-radius tolerance.
5. Check AI-specific detection coverage.
6. Establish response readiness.
7. Define re-approval triggers.

These gates produce operational outputs for a specific production approval. A security framework serves a different purpose: it explains how to build a program. The distinction helps keep a broad program document from standing in for an approval decision about one acting system.

The evidence bar should rise with autonomy. A read-only retrieval-augmented chatbot and an autonomous workflow agent should not face the same production standard. Their authority differs, so the evidence required for approval differs as well. The production evidence-gate approach makes that scaling explicit.

Apply the gates to observed behavior, not only declared capability. Inventory and declared access establish what the candidate says it can reach. A behavioral baseline and observed access show what appeared during operation. Blast-radius tolerance defines the acceptable consequence. Detection coverage and response readiness address how the operation will be seen and handled. Re-approval triggers define when that decision must be made again.

If your search starts with "agent harness open source," make authority the first test: locate the enforcement point. If a sensitive action is constrained only by a prompt, the model remains part of the permission decision. Deterministic permissions, hooks, and approval mechanisms place that decision in the harness.

Normal reliability checks miss agent-specific failures

An agent process can look healthy at the transport layer while its work has gone wrong. Circuit breakers, retry policies, and health checks based on HTTP status and latency are blind to some failures introduced by non-deterministic model reasoning.

Two failures make the gap concrete. A workflow can repeat an external action even while its requests succeed. A reasoning loop can keep consuming budget without producing useful progress. Neither failure has to appear as a bad status code or a latency spike.

Long-running recovery therefore needs controls that observe workflow state and side effects directly. The harness must distinguish a request that completed from an external action that has already happened, and it must expose whether the reasoning process is still making progress. Transport health remains useful, but it cannot answer those workflow questions.

When you test a candidate, interrupt a workload after it has produced a side effect. Inspect the stored workflow state, then determine whether recovery repeats the action or resumes after it. Run a stalled reasoning path as well and check whether the harness can expose the continuing loop before it spends more budget.

Take the scorecard to the repositories you are considering. Follow one real workload through authority, state, traces, evaluation, recovery, ownership, maintenance, and measured cost. Then interrupt the run and inspect what the harness can prove, replay, resume, and contain.