Steering Agent Behavior via a Domain Expert-Driven Alignment-to-Optimization Bridge
Wesley Pasfield (University of San Diego and Databricks)
Architectural Patterns & Composition Evaluation & Benchmarking
Summary
A system that makes agent behavior steerable by bridging domain expert trace labels to calibrated evaluation judges and optimized prompts, improving performance by 15.7%.
Description
Aligning compound AI agents with domain expertise typically requires manual prompt engineering or scorer design that drifts from actual expert quality criteria. We present a system that makes agent behavior steerable: the only manual step is for domain experts to label traces. From those labels, an automated bridge produces a calibrated evaluation judge (via MemAlign), an optimized system prompt (via GEPA), and composable agent skills (via GEPA's optimize_anything). Because the judge is calibrated to expert feedback before optimization begins, every downstream change reflects the expert's definition of quality, and all artifacts are versioned in MLflow for auditability. We demonstrate the bridge using a baseball hitting-analysis assistant with graph-enforced tool routing, per-thread conversation memory, and parallel tool execution. In this example, the bridge culminates in an agent that outperforms the original by 15.7% as evaluated by the aligned judge.
