How to choose an open-source agentic AI framework by its control contract

Key takeaways

Choose an open-source agentic AI framework for the workload, not the category label. Put your top two candidates through the same small task. Compare persisted execution, tool authority, approval gates, trace records, behavioral evals, and crash recovery before you commit.

Choose an open-source agentic AI framework for the work at hand

Start with the shape of the work. A framework can be a good match for one workflow and an awkward match for another, even when both projects are called "agentic."

AI workflows use predefined graphs of executors and edges. That structure supports deterministic orchestration, predictability, and testability for work that is well defined and repetitive. If your task already has a known sequence, that fit deserves a place on your requirements list.

AI agents address a different shape of work. They use dynamic autonomy for open-ended, adaptive problem-solving. That flexibility comes with unpredictability and higher resource consumption. The decision is not whether dynamic behavior sounds more capable. It is whether the work requires it.

Write the workload in plain terms before opening a feature matrix. Is it a predefined graph, or does it require open-ended adaptation? That answer narrows the useful field without pretending that a category settles the implementation details. Our guide to AI agent architectures can help you describe that shape before you compare frameworks.

Once the workload is clear, the shortlist can reflect stated fit. It still cannot tell you how a candidate behaves when execution is interrupted or a tool needs approval. That takes a trial.

Which open-source agentic AI frameworks belong on the shortlist?

A useful agentic AI frameworks list is a set of plausible candidates, not a ranking. The AI agent framework open-source field includes application frameworks, runtimes, harnesses, workflow frameworks, orchestration frameworks, and software development kits. Those categories help you find candidates with a relevant stated fit.

This starting field covers nine distinct positions from an official framework overview:

• LangChain is an MIT-licensed, open-source large language model application framework positioned for fast prototyping of complex agentic workflows.
• LangGraph is an MIT-licensed, open-source agent runtime positioned for complex agents that require precision. If the agentic AI framework LangGraph is on your list, that is its stated fit, not a verdict on your control requirements.
• Deep Agents is an MIT-licensed, open-source agent harness positioned for long-running workflows.
• CrewAI is an MIT-licensed, open-source multi-agent orchestration framework positioned for rapid prototyping of role-based agent workflows.
• Microsoft Agent Framework is an MIT-licensed, open-source multi-agent orchestration framework positioned as the unified successor to AutoGen and Semantic Kernel for the Microsoft stack.
• LlamaIndex Workflows is an MIT-licensed, open-source agent workflow framework positioned for document-centric, event-driven multi-agent systems.
• Google ADK is an Apache 2.0-licensed, open-source agent development framework positioned for teams using Google Cloud Platform and seeking opinionated agent runtimes.
• OpenAI Agents SDK is an MIT-licensed, open-source multi-agent workflow software development kit positioned for tightly scoped assistants and delegation workflows.
• Mastra is an AI agent application framework with partial open-source status, positioned for TypeScript teams building production custom agents.

For readers looking specifically for a Google agentic AI framework, Google introduced Agent Development Kit at Google Cloud NEXT 2025 for end-to-end development of agents and multi-agent systems. Keep that introduction scope distinct from any operational capability you have not observed in your own trial.

Narrow the field to the two candidates whose positioned fit comes closest to your workload. A license, category, or stated use case can guide that first cut. Then require both candidates to show how they handle the controls your application needs.

The category differences also prevent a false like-for-like comparison. A runtime positioned for precise complex agents, a harness positioned for long-running work, and a software development kit positioned for tightly scoped delegation begin from different promises. Keep those promises in the candidate notes, but do not turn them into scores. The same task will show whether either candidate meets your acceptance requirements.

You can make the first cut with three questions. Which stated use case most closely matches the work? Does the license or partial open-source status fit your project? Does a positioned ecosystem fit, such as Google Cloud Platform, the Microsoft stack, or TypeScript, matter to the implementation? These questions reduce the field without claiming that one category is universally better.

The broader AI agent frameworks guide develops the harness lens across framework choices. For the immediate decision, two candidates and one small task are enough to expose the control contract.

Test the control contract on one small task

Run the same small task in your top two candidates and trace both. This becomes useful when an application needs durable state, retries, multi-agent delegation, human-in-the-loop steps, and consistent tracing. The point is to observe operating behavior under the same task, not to compare two polished demos with different demands.

Define the acceptance requirements before either run. Keep the task and the six control surfaces fixed for both candidates. Record what each system actually does, including a failed run, instead of giving credit for a capability name. This makes the comparison symmetric while leaving room for each framework to expose its controls in a different way.

Fill this matrix with observations from each run:

What must survive a crash?

Chat history is not an execution record. It does not establish which actions ran, and it does not show whether a retry will duplicate a side effect. A durable agent needs an execution journal, idempotent tool boundaries, versioned prompts and tools, durable approvals, and intentional crash-recovery tests.

Place these recovery requirements in your agent harness, alongside the state and tools involved in the run.

Each part answers a different recovery question. The execution journal establishes which actions ran. Idempotent tool boundaries are required at the points where a repeated call could matter. Versioned prompts and tools identify the operating inputs attached to a run. Durable approvals preserve approval state. The intentional crash test shows what the system does when execution is interrupted instead of ending normally.

That distinction changes the trial. Do not stop after confirming that the conversation reappears. Interrupt the run after an action has started, restart it, and inspect what the system does with completed work and pending work. The acceptance requirement is not a restored transcript. It is recovery without replaying the external effect.

AXME documents a concrete recovery sequence. An intent remains IN_PROGRESS in PostgreSQL after the agent dies. After restart, the intent is redelivered with a maximum of three delivery attempts and then proceeds to COMPLETED. This example names the persisted state, the interruption, the delivery bound, and the completed state. Those are observable surfaces you can look for in another runtime without assuming it implements the same design.

Eino exposes a related set of control responsibilities. Its Runner manages agent lifecycle, context initialization, checkpoint persistence, and interrupt recovery. Again, the value for selection is in the named surfaces. Your trial can inspect where checkpoints live and how interruption recovery behaves.

During the failure test, capture the state immediately before interruption and the state after restart. Look for the status of the pending intent, the delivery attempts after restart, and the final completion state when the framework exposes them. Then inspect whether any tool call, mutation, approval, or message was repeated. These observations answer the recovery requirement directly.

Session memory can still be useful, but it answers a narrower question. The crash-recovery decision depends on execution journaling, idempotent tool boundaries, durable approvals, and the behavior observed after an intentional interruption. A candidate earns that row from the failure test, not from the restored conversation alone.

How should approval scale with risk?

Require approval when an agent action is irreversible, costly, regulated, or has a high blast radius. That puts human attention where an incorrect action carries meaningful consequences. It also gives your framework trial a concrete approval requirement instead of the vague question, "Does it support humans in the loop?"

In Microsoft Agent Framework, requesting user approval for a function call is a human-in-the-loop pattern. Test whether the run requests approval for the function call and capture that event with the rest of the execution behavior.

Not every low-risk action needs the same approval intensity. One sampled-approval pattern approves all high-risk actions and 5-20% of low-risk actions. That pattern includes exception monitoring and periodic audits of auto-approved actions.

Turn those boundaries into two test cases. First, use an action that your policy classifies as high risk and observe whether the run pauses for approval. Second, use a low-risk action and observe whether the configured sampling policy is represented in the control surface. Record only what happens. A settings screen or capability label does not replace the run itself.

Write the policy before testing the framework. Mark actions that are irreversible, costly, regulated, or high in blast radius as requiring approval. For lower-risk actions, record the sampling rate, exception monitoring, and audit policy you expect the harness to support. The test can then show whether approval behavior follows the policy instead of leaving the reviewer to judge a run by feel.

Approval is part of recovery too. If a process crashes after approval, durable execution should preserve completed approvals so recovery does not repeat them. That is why the approval and crash rows sit beside each other in the matrix while remaining separate observations.

Evals turn trial failures into regression tests

The first failed run is useful when it becomes a test. Investing in evals early turns failures into test cases, lets those tests prevent regressions, and replaces subjective impressions with actionable metrics.

Use the trace from each candidate as the record of what happened. Run the agent, apply checks to the recorded behavior, and turn a manual fix into a regression test. This behavior-grounded loop gives you more than a final answer to compare. It lets you examine the trajectory that produced the answer and keep a known failure from quietly returning after a change.

Tool behavior belongs inside that evaluation loop. Effective agent-tool development includes building and testing tool prototypes and running comprehensive tool evaluations with agents. For framework selection, that means the tool call in your small task is part of the test, not incidental plumbing.

Add every observed failure to the candidate's trial record. Then rerun the checks after changing the prompt, tool, or framework configuration. The useful comparison is whether each candidate lets the failure become a repeatable check with an actionable result. That is stronger than choosing the run that looked better once.

Keep the run record, the check, and the manual fix connected. The run provides the observed behavior. The check grades it. The manual fix becomes a regression test that can run again after a change. If a candidate makes that loop legible, you can compare its behavior across changes with metrics instead of relying on a remembered impression.

What can a broad integration surface expose?

A broad integration surface creates more places to inspect. Microsoft Agent Framework, for example, documents an agent harness for long-running work and workflow APIs for execution and state. Its integrations cover models, tools, context, middleware, evaluation, and user interface components.

That breadth maps directly to control questions. Where does execution state live? Which tools receive authority? What context reaches the model? Where can middleware enforce a boundary? How does evaluation receive a run record? Treat each integration point as part of the harness you are adopting, because each one can affect what the complete system can do and what you can inspect.

External services also carry concrete risks. Microsoft documents that its framework's integrations with external services can expose API keys through configuration or logs. External-tool failures or unavailability can also affect agent performance. Put credential handling and unavailable-tool behavior into your observations when either is relevant to the task.

Those risks create two practical checks for an integration-heavy candidate. Inspect configuration and logs for exposed API keys. Then make an external tool unavailable during the task and record how the run behaves. These checks do not decide whether a broad surface is good or bad. They show whether the harness makes two documented risks visible enough to manage.

Apply the control-contract matrix to your top two frameworks with the same small task. Keep the observed traces, failures, approval events, and recovery results beside each acceptance requirement. If you want to place those findings in a broader selection model, read our guide to choosing AI agent frameworks by the harness.