Key takeaways
- Pydantic AI already gives every agent a light harness.
- The separate Harness package can iterate faster and carries its own release policy.
- Treat issue-stage capabilities as dependencies to test, not package guarantees. Price the completed workflow, not just tokens.
The useful boundary is not "framework versus no framework." It is the layer of harness your workload requires. Package documentation, repository instructions, and issue records answer different questions. Read them that way, then test the behaviors that determine whether the agent can complete work reliably and at a cost you can sustain.
Where should you start with Pydantic AI Harness?
| Project record | What it establishes | Decision it answers |
|---|---|---|
| Official Harness page | Pydantic AI Harness is the official capability and harness library for Pydantic AI | Am I looking at the official package? |
| Repository AGENTS.md | Dependency changes should be required, made with uv, and linked to an issue. Changes to pyproject.toml or uv.lock need dependencies:approved; a later push clears approval | What controls apply when contributing dependency changes? |
| Issue #149 | GraphAgentState serialization and resume_from_state() are tracked as foundational checkpointing infrastructure that requires a Pydantic AI core change. No branch or pull request was shown | Does the checkpointing dependency sit in the package or in issue-stage core work? |
A checkpointing dependency belongs against the issue record, not the package name or contributor approval rules. Issue #149 places GraphAgentState serialization and resume_from_state() in foundational infrastructure that requires a Pydantic AI core change, with no branch or pull request shown. That is the relevant boundary when durable checkpointing would shape the production design.
What does Pydantic AI already include?
Every Pydantic AI agent already has a light harness: a typed agent loop, model support, user-defined tools, and structured output. The official documentation says that baseline is sufficient for simple agents.
Pydantic AI Harness is maintained as a separate package so its capabilities can iterate faster. That creates a practical starting point. Define what the light harness already covers for your agent before you add another layer. For a broader view of the surrounding design choices, our agent harness field guide frames the harness as the part that decides how work is controlled, checked, and allowed to continue.
Use those four parts as the baseline, not as a generic feature count. The typed loop, model support, tools, and structured output describe what every Pydantic AI agent starts with. The official guidance draws the sufficiency line at simple agents. Once the workload needs more, name the missing behavior before selecting another package or building a wrapper. A specific gap is easier to evaluate than a broad desire for a "more complete" harness.
Release changes follow a stated migration policy
The separate package has a two-part maintenance policy. Patch releases are not intended to break existing behavior. When a breaking change does occur, it is documented in release notes with migration guidance.
Those commitments give an update review two inputs: the release type and its release notes. The first sets the compatibility expectation. The second identifies a breaking change and provides the migration guidance for it. Keep both in the update record so a patch expectation is not mistaken for a claim that behavior can never change.
Pydantic AI Harness is maintained separately so its capabilities can iterate faster. Version policy therefore remains part of operating the package after the first adoption decision. Record the release type, the relevant release note, and any migration guidance before moving the update through your normal workload checks.
What must production readiness cover beyond model quality?
| Boundary | Defined behavior | Adjacent mechanism | Scope |
|---|---|---|---|
| Model reliability | A more intelligent model does not automatically make an agentic service more reliable | A harness governs how agent intent becomes durable work | Test the service boundary, not model capability alone |
| Verification loop | Make a change, run tests, lint, or builds, parse failures, inject them as context, fix the problem, and repeat | Execution starts the check; parsed feedback drives the next correction | Require the active cycle to continue until verification passes |
| Shell access | Gives the agent somewhere to execute commands, but does not by itself provide an iterative verification-and-fix cycle | A full verification loop adds failure parsing, feedback, correction, and repetition | Decide whether command access is enough or the workload requires verified correction |
| LSP diagnostics | Provides passive language-server feedback rather than active test execution | Tests, lint, and builds exercise a different part of the failure path | Decide whether passive diagnostics satisfy the check your workload needs |
| Tool Error Recovery | Handles tool exceptions rather than failures returned by a test or lint run | Verification failure handling parses the failed check and feeds it back into correction | Keep tool exceptions and failed verification as separate acceptance cases |
| Stuck Detection | Identifies repetitive patterns rather than verification failures | A verification loop responds to the output of a failed check | Test repetitive behavior and failed verification as different failure modes |
| Growing-system demands | Larger agent projects demand more sophisticated tools, coordination, context, orchestration, execution, safety checks, fallback plans, and memory | The governing harness layer must cover the demands that matter to the workload | Reassess the required layer as the project grows |
| Function-level authority | Which functions may the agent invoke, and what authority applies to each one? | Having a tool available does not answer the authority question | Record the allowed function boundary and the authority attached to it |
| Bounded execution | What contains the run, and what happens when it reaches that boundary? | Stuck detection covers repetitive patterns, not the full execution boundary | Record the execution boundary and the behavior at that boundary |
| Interruption | Where can an operator stop the run before another consequential action? | Tool Error Recovery handles exceptions, not the operator's interruption path | Record the interruption point and what the operator can stop there |
| State and recovery | What state persists, and what can resume, restart, rewind, or recover? | Checkpointing and conversation-state issues describe proposed work at an issue stage | Record the durable-state and recovery path without treating an issue title as shipped behavior |
Test the operating boundary with one failure path
- Establish the starting boundary. Record which tools the agent may use, which functions it may call, and where its authority ends. Add the execution boundary and the point where an operator can interrupt the run. These are acceptance conditions for the architecture. They are not implied by the package name or by access to a shell.
- Run the complete verification cycle. Have the agent make the change and run the selected check. When the check fails, observe whether the failure is parsed, returned to the agent as context, corrected, and tested again. Continue until the check passes or the run reaches its boundary. That sequence is the behavior under evaluation. A command that merely executes is not the complete cycle.
- Test adjacent mechanisms separately. Give passive language-server diagnostics their own result. Do the same for a tool exception and a repetitive pattern. Those mechanisms answer different failure questions, so one passing result should not stand in for another. This separation also shows which layer owns the response when a check fails, a tool throws an exception, or the agent repeats itself.
- Stop and recover. Interrupt the run at the defined point, then record what state remains and what can resume, restart, rewind, or recover. Durable checkpointing and conversation-state requirements belong in the issue-stage comparison before they enter the operating design.
- Write the result as an architecture decision. Mark each required behavior as owned by the light baseline, Pydantic AI Harness, or your wrapper. Leave any behavior without an owner as an open design gap. Then price the architecture that actually passed the workload, including retries, review, latency, and wrong actions, rather than pricing an assumed capability set.
Treat any required behavior with no named owner as an architecture gap. Put that gap in an agent run contract and make it an agent evaluation target before you compare costs or approve the design.
Keep proposed capabilities in a separate dependency register. For each hard dependency, record the issue number, label, required upstream change, and visible branch or pull-request state. Issue #149, for example, places durable checkpointing work behind a Pydantic AI core change and showed no branch or pull request. That gives the wait path a concrete trigger without turning proposed work into a package promise.
Full-workflow economics decide whether the harness pays
Do not price a harness from license or token cost alone. Additional agent steps, retries, critiques, and larger contexts can raise inference, tool, and review costs. The size of that increase depends on the workload, so the same architecture can present a different economic case when the work changes.
Measure cost beside success rate and latency. A cheap attempt tells you little when the work needs several attempts, tool calls, or reviews before it is complete. Use three outcome units:
- Cost per correctly resolved case. This counts the work that reaches the required result, not every attempt as an equal success.
- Cost per completed workflow. This keeps all required steps inside the economic boundary.
- Cost of a wrong action. This makes failure part of the calculation instead of treating it as a side note.
Turn those units into one workload calculation. For the workload you observed:
- Add the inference, tool, and review costs created by the steps, retries, critiques, and context in the case.
- Add engineering and operations, infrastructure, retrieval, and observability where they contribute to that architecture's cost.
- Divide the workload total by correctly resolved cases.
- Divide the same total by completed workflows.
- Record wrong-action cost separately. Place success rate and latency beside the three results.
Run that calculation separately for each architecture. One path may add retries and review; another may need more engineering or infrastructure. Keep each cost on the path that creates it, then compare success rate, latency, correctly resolved cases, completed workflows, and wrong actions side by side. That exposes whether an apparent saving at the model call simply moved into execution or operations.
There is no universal largest hidden cost in an agentic project. Inference, engineering and operations, infrastructure, tool calls, retrieval, observability, and human review can all be significant. The full-case economics depend on the intended architecture and completed workload.
Open-source licensing is one input. Engineering effort, orchestration overhead, and operational needs still belong in the comparison. Use the full cost of a completed case for the architecture you intend to operate.
Should you adopt, wrap, or wait?
Start with the light baseline, then follow the first branch that matches your workload.
- Can the light harness cover the whole job?
- Yes: adopt the light baseline. Pydantic AI's existing light harness is sufficient for simple agents. Use it when the typed agent loop, model support, user-defined tools, and structured output cover the job.
- No: continue to the dependency check. Name the missing behavior before you choose another layer. The separate Harness package can iterate faster, but its separate package identity does not answer whether that missing behavior exists. Keep the requirement concrete enough to test against a package record, an issue, or a wrapper design.
- Is a named issue-stage capability a hard dependency?
- Yes: wait for that dependency, or change the design so it is no longer required. These issue records show the capability stage:
- Execution Checkpointing & Resume was tracked under a label for proposed or added standalone capabilities, with no branch or pull request shown.
- Resumable browser sessions and sandbox state were tracked as a new feature or request and foundational infrastructure, with no branch or pull request shown.
- Background agent execution with asynchronous results was tracked as an agent-level feature and new feature request, with no branch or pull request shown.
- Saving, rewinding, and forking conversation state were tracked under a label for proposed or added standalone capabilities.
- No: continue to the harness-layer check. An issue title is not package behavior, but it does not block a workload that does not depend on that capability. Keep the issue out of the adoption case unless the planned workflow actually requires it.
- Does the job need control or execution behavior beyond the light baseline?
- Yes: assess a wrapper. The harness layer a team needs can matter more than a feature grid. Put each missing behavior against the production-readiness boundary: verification, function-level authority, bounded execution, interruption, and state or recovery. Assign each behavior to the baseline, the package, or the proposed wrapper. If no layer owns a required behavior, the design is not ready for that workload.
Price the wrapper through the full cost of a completed case. Engineering effort, orchestration overhead, and operational needs remain part of the framework cost. Compare architectures with success rate and latency beside cost per correctly resolved case, cost per completed workflow, and the cost of a wrong action. Our AI agent frameworks comparison covers the broader cross-project choice.
The wrap path is workload-specific. It does not establish a global rank for Pydantic AI Harness or any alternative. It asks whether an added layer can supply the behavior the light baseline does not cover, and whether the completed-case economics support operating that layer. Licensing alone cannot settle the answer because engineering effort, orchestration overhead, and operational needs remain part of the cost.
The wait path is narrower. It applies when the workload depends on one of the named issue-stage capabilities. The adopt path is narrower too. It applies to the simple-agent boundary stated in the official documentation. Keeping those paths separate avoids treating package identity, a feature proposal, and a workload result as interchangeable proof.
Limits of this decision guide
Your workload still needs its own measurements. Package identity, maintenance policy, and issue metadata do not determine the success rate, latency, or completed-case cost of a specific run. Measure those values on the workflow and architecture you intend to operate.
Apply the production-readiness matrix to one real workflow. Name the required behavior, mark any issue-stage dependency, and calculate the cost per completed case. Then use our agent harness field guide to examine the five levers that decide what ships.