How it works
A macro evaluation exercises a meaningful slice of the operating system around an agent, then judges the combined outcome. The unit under test is not one prompt, tool call, or model response. It is the behavior of the harness across a complete scenario: receiving work, assembling context, granting authority, invoking tools, updating state, handling failures, and producing an observable result.
A useful macro evaluation usually follows a short control loop:
- Define an operational scenario with initial state, permissions, dependencies, and a clear completion condition.
- Introduce realistic variation, such as ambiguous input, a provider failure, stale state, an interrupted run, or a misleading tool result.
- Execute the system through its normal orchestration path rather than bypassing controls for the test.
- Capture outputs, state transitions, approvals, tool effects, recovery actions, and traces.
- Judge the system against outcome-level criteria: correctness, containment, recoverability, policy compliance, cost, and operator legibility.
The evaluator may combine deterministic checks with human or model-based judgment. Deterministic checks should own facts such as whether a forbidden write occurred or a checkpoint was persisted. Judgment is better reserved for questions such as whether the final result was useful or whether an escalation gave the operator enough context.
Why it matters in an agent harness
Agentic systems fail through interactions. A model may choose a reasonable action while the harness supplies stale context. A tool may return a valid response while the orchestration layer associates it with the wrong task. Every component can appear healthy in isolation while the overall run produces an unsafe or unrecoverable outcome.
Macro evaluation exposes these composition failures. It asks whether the complete control structure works when local behaviors interact. That makes it especially useful for testing boundaries between model judgment and deterministic enforcement.
For control, a macro evaluation can verify that approval policies remain effective across retries, delegation, and resumed execution. Checking the approval function alone does not prove that every write path actually reaches it.
For reversibility, it can test whether interrupted or partially completed work can be identified, resumed, rolled back, or compensated without duplicating effects. The important result is not merely that checkpoints exist. It is that recovery from those checkpoints preserves the intended state.
For observability, it can determine whether an operator can reconstruct what happened from the artifacts the system really emits. A complete trace is still inadequate if identities, decisions, authority changes, or external effects cannot be connected into one legible account.
For permissions, it can test the effective authority of the whole execution path. The model-facing tool list may look narrow while delegated workers, inherited credentials, or indirect calls create a larger blast radius. Macro evaluation measures what the system can actually cause, not only what one interface declares.
For failure containment, it can introduce a provider outage, malformed result, or confused-deputy attempt and observe whether the harness stops safely. A strong outcome makes the failure visible, preserves authoritative state, and prevents unrelated work from being corrupted. Continuing with plausible but degraded output should count as failure when the operator cannot tell that the system lost a required capability.
Macro evaluations are therefore expensive relative to unit checks. They cover fewer cases and can be harder to diagnose. Their value is different: they test whether locally sensible mechanisms compose into dependable operation.
Macro evaluation vs trajectory evaluation
The distinction changes where you place evaluation effort. A trajectory evaluation judges the sequence of reasoning-visible actions within a run. A macro evaluation judges what that run means for the surrounding system and operator.
| Axis | Macro evaluation | Trajectory evaluation |
|---|---|---|
| Primary unit | End-to-end system scenario | Action or decision sequence |
| Main question | Did the system produce a controlled, recoverable outcome? | Did the agent take an acceptable path? |
| Typical evidence | State transitions, external effects, approvals, traces, recovery results | Messages, tool calls, intermediate decisions, ordering |
| Best at finding | Composition failures and control gaps | Poor action selection and inefficient or unsafe paths |
| Main limitation | Broad failures can be difficult to localize | A good-looking path may still produce a bad system outcome |
The two evaluations should inform each other. Use the macro result to establish whether the system met its operating contract. Use trajectory evidence to explain why it succeeded or failed. Do not let a coherent trajectory excuse an unauthorized effect, and do not treat one successful outcome as proof that the path was safe.
The Rifty take
We optimize macro evaluations for failures that cross boundaries: model to tool, tool to state, worker to orchestrator, and automation to operator. We accept lower test volume in exchange for scenarios that expose real authority, external effects, and recovery behavior. A system does not pass because it completed the task; it passes when completion remained controlled and legible.
Implementation checks
- Define the system outcome before selecting component metrics.
- Include initial state, authority, dependencies, and stop conditions in every scenario.
- Exercise normal approval, checkpoint, retry, and delegation paths.
- Record external effects separately from generated text.
- Test at least one interruption and one degraded dependency.
- Make unauthorized or untraceable effects automatic failures.
- Verify that recovery does not duplicate irreversible actions.
- Preserve enough trace data to localize a failed macro evaluation.
- Pair broad scenarios with narrower regression evaluations after diagnosing a defect.
- Re-run scenarios when permissions, orchestration, state handling, or recovery contracts change.
Frequently asked questions
What should a macro evaluation measure?
A macro evaluation should measure outcome correctness, policy compliance, external effects, recoverability, failure containment, and operator legibility across an end-to-end scenario. Its criteria should reflect the system's operating contract, including what must stop, what may proceed, and what evidence must remain after execution.
How is a macro evaluation different from an end-to-end test?
A macro evaluation adds behavioral and operational judgment to end-to-end execution. An end-to-end test often verifies that a workflow reaches an expected result. A macro evaluation also asks whether authority was respected, failures were contained, state remained recoverable, and the resulting evidence supports operator review.
Can deterministic checks be used in a macro evaluation?
Yes. Deterministic checks should verify concrete facts such as permission violations, duplicated effects, missing checkpoints, invalid state transitions, and absent audit records. Human or model-based evaluators can assess usefulness and judgment, but they should not replace exact checks for properties the harness can verify directly.
When should macro evaluations run?
Run macro evaluations before releasing changes that alter orchestration, permissions, state management, delegation, recovery, or tool effects. They also belong in periodic reliability exercises. Use smaller regression checks for frequent feedback, then run selected macro scenarios when a change could affect behavior across component boundaries.