Harbour agent harness: evaluating the Harbor Framework

Key takeaways

• Harbor evaluates and optimizes agents with sandboxed tasks.
• Sandboxed evaluation isolates the task, exposes agent actions, and scores the result against a verifier.
• Structured frameworks fit deterministic work; lightweight harnesses may fit open-ended problems. Choose by complexity, token budget, and team preference.

Start with the right harbour agent harness

If you are evaluating a harbour agent harness, begin by separating Harbor Framework from the broader idea of an agent harness. Harbor Framework evaluates and improves agents. An agent harness is the surrounding system that helps an agent do work a single prompt cannot fully specify. For any same-name project, the Harbor or HARBOR label alone does not establish a shared purpose or architecture.

That distinction changes the adoption question. Harbor turns agent work into an evaluable task. A broader harness supplies the tools, knowledge, and guardrails around work. Before comparing features, identify the project you are looking at, then ask whether your work is ready to be expressed as an isolated task with a score.

Harbor's own description is deliberately evaluation-oriented: it evaluates and optimizes arbitrary agents and language models. That makes a Harbor investigation different from a search for a general-purpose agent shell. The first asks whether a task can produce a meaningful result under a verifier. The second asks what environment an agent needs to work at all. Both can matter in the same system, but they answer different engineering questions.

Those scopes also call for different inventories. A general harness decision starts with the tools an agent can call, the abstractions it works through, the repository knowledge it can reach, and the mechanical guardrails on its actions. A Harbor decision starts with the agent or language model you want to evaluate and the task against which you want to evaluate it. If your system needs both, name both layers in the design. The surrounding harness can define how work happens, while Harbor provides the evaluation framework for testing an agent against a specified task. Keeping those jobs explicit prevents a broad harness requirement from being mistaken for an evaluation requirement, or an evaluation framework from being asked to supply the agent's entire working environment.

What is the point of an agent harness?

An agent harness shapes the working environment, not just the words sent to a model. Prompt engineering optimizes a single interaction. Context engineering controls what the agent sees on a turn. Harness engineering shapes the whole environment in which the work happens.

For coding models, that environment shapes how the model interacts with repositories, tools, and execution environments. It is where a task stops being an isolated request and becomes work performed in a particular place, with particular tools and constraints. That is why the harness deserves attention even when the model is capable.

The distinction also prevents a common design shortcut. A better prompt can improve one interaction, and better context can change what the model sees on one turn. Neither choice by itself defines the tools, repository knowledge, or mechanical guardrails available across the working environment. A harness is the place where those parts are assembled for the work at hand.

Evaluation and observability belong in that environment when you need to see behavior rather than only a final answer. Structured evaluation layers, a CI-integrated harness, and observability can surface behavioral signals and trajectory drift before users encounter failures. Our agent harness notes focus on that control layer around the model: the system in which a model is allowed to act, not a claim that one model should be trusted on its own.

How does Harbor turn a task into a verdict?

Harbor specifies sandboxed agent tasks for evaluation and optimization.

The task-to-verdict flow

A sandboxed evaluation creates a boundary around behavior. The isolated environment limits side effects, while the action record makes the route to success or failure visible. Repeatability comes from running a specified task inside that boundary and scoring the result against a verifier. That combination matters when a final output is not enough to judge the run. You can inspect what the agent did as well as the score it received. The architecture keeps agents, tasks, and jobs separate, and it keeps trials separate from sandboxes, so the evaluation record does not collapse every part into one opaque run. Once you need that level of inspection, task design and verifier behavior become part of the adoption cost. A sandbox can contain execution, but the verifier still has to recognize the result the task defines.

Task design comes before agent testing

A score is only useful when the task is precise enough to test. Terminal-Bench task instructions use absolute paths, explicitly identify expected output files and formats, and include the information an agent needs to complete the task. These details make the target legible before any agent begins.

Terminal-Bench validation combines static structure and formatting checks, rubric-based autoreview, Docker builds, oracle checks, and nop checks. The point is not that every task needs every kind of check. It is that a final file alone may not settle whether a task was completed under the rules that define it.

One Harbor task implementation offers a useful preflight: prove that the oracle run succeeds, then prove that each negative control fails its intended criterion before connecting real agents. That test exercises the task and controls before agent behavior enters the picture. A result can then be read as an agent result without treating an untested task definition as ground truth.

The five validation mechanisms describe different kinds of checks: static structure and formatting, rubric-based autoreview, Docker builds, oracle checks, and nop checks. They are named parts of Terminal-Bench validation, not interchangeable labels for a single final-output check. Together, they show why verifier design is work in its own right.

The same discipline keeps the instruction itself useful. Use the exact output file and format when the task requires one. Put the necessary task information in the task. Do not let hidden assumptions become a second, informal evaluator after the run has finished.

What does a trajectory need to preserve?

The result is not the only record worth keeping. Agent trajectories represent reasoning differently. MiniSweAgent uses explicit THOUGHT fields. OpenHands embeds reasoning in tool calls. Gemini CLI records reasoning in token metrics. A trajectory format has to accommodate those differences without pretending that all agents expose the same kind of reasoning.

Action records vary as well. They can be bash strings, structured function calls, or implicit actions. Completion can involve multiple verification steps, or it can be represented as a single step. ATIF is intended to hold different patterns of reasoning, action logging, observability, and step granularity in one interoperable format.

That breadth matters when you are deciding what a record needs to reveal. A system may expose tool-call identifiers and security assessments, or it may retain minimal metadata. The right record is the one that preserves the actions and verification shape that your task needs to inspect. It should not force an agent's native trace into a false uniformity.

The difference in representations is not cosmetic. Explicit THOUGHT fields, reasoning embedded in tool calls, and token-metric recording are different records. Bash strings, structured function calls, and implicit actions are different action logs. An interoperable format has to make room for those differences while still allowing the trajectory to be carried between systems.

Reproducibility is also a record-keeping practice. The Baba Is Harbor benchmark keeps the exact jobs and commands behind every published number in RUNNING.md. Preserving the job and command gives a published result a concrete run record, rather than leaving it as an outcome with no route back to the work that produced it.

A stale verifier can make a correct run look wrong

Verifier logic needs the same scrutiny as the task. A reported Harbor Index issue found that all five named gaia2-* tasks gave a correct oracle solution a reward of 0. The reason given was stale verifier expectations: the expected events no longer matched runtime output.

That incident is specific to those five tasks. It still makes the boundary clear. A failing score can reflect verifier expectations that have drifted from runtime behavior, even when the oracle solution is correct. The issue record is a useful reminder that an evaluator produces a verdict only while its own expectations match the task it is judging.

This is why an oracle run and negative controls belong before a real-agent connection. They test whether the task's checks recognize the intended success and reject the intended failure. The gaia2 report does not describe every Harbor task. It does show the concrete cost of leaving verifier expectations stale.

Which agent harness fits your task?

Harbor is a framework for evaluating and improving agents. The comparison below keeps the adoption questions concrete without declaring a universal winner.

Task complexity, token budget, and team development preferences remain part of the choice. The durable question is whether the work needs a bounded task, an action record, and a verifier score. If it does, the structured evaluation path is worth examining. If the work is creative and open-ended, a lightweight harness may fit better.

Deeper iteration adds its own cost curve

The cost of an evaluation harness is not limited to the coordinating layer. Automated harness engineering must contend with heterogeneous actions, sparse and noisy evaluation signals, very long trajectories, and difficult outcome attribution. Iterative harness evolution also adds benchmark execution, trajectory analysis, and workspace management.

These are different sources of overhead. Heterogeneous actions make a single execution pattern hard to assume. Sparse and noisy signals make outcomes harder to read. Long trajectories make more of the run relevant to inspection. Difficult outcome attribution makes it harder to connect a change in the harness to a later result. Benchmark execution, trajectory analysis, and workspace management add continuing engineering and compute work to that problem.

Compute has its own shape. Separating harness and sandbox layers can let sandbox compute start only when it is required and give different tasks different capacities. A coordinating harness can be lightweight while a sandbox needs substantial processing power and memory for work such as data analysis, software builds, or document conversion.

That is the final adoption boundary. Harbor earns its overhead when the task needs a sandboxed, verifier-scored evaluation record. When defining that record, start with our agent evaluation guide: define the task set, scoring method, and regression gate before treating a score as a decision.