Open Agent Specification: Enabling Cross-Framework Comparison of AI Agents

Open Agent Specification is a framework-agnostic declarative language for defining AI agents and multi-agent workflows, enabling portability and interoperability across agent frameworks. It provides common abstractions for control flow, data semantics, and tool integration so that workflows built in one framework can run in another.

Abstract

The proliferation of agent frameworks has made it difficult to evaluate AI agents fairly across runtimes, because frameworks differ in their abstractions, execution semantics, prompt handling, and tool integration mechanisms. We present a cross-framework evaluation methodology for agentic systems based on a shared declarative representation and a standardized evaluation harness. To enable this methodology, we introduce Open Agent Specification (Agent Spec), a framework-agnostic declarative language for representing AI agents and workflows with common components, control and data flow semantics, and validation support. Using Agent Spec as the common representation layer, we study four distinct runtimes (LangGraph, CrewAI, AutoGen, and WayFlow) across three different benchmarks (SimpleQA Verified, 𝜏2-Bench and BIRD-SQL). Our results show that, even when starting from the same agent specification, runtimes differ meaningfully in accuracy, latency, and execution behavior due to framework-specific differences. We also release supporting tooling, including a Python SDK (PyAgentSpec), a reference runtime (WayFlow), and adapters for popular frameworks (e.g., LangGraph, AutoGen, CrewAI), to make such cross-framework comparisons easier to reproduce and extend.