Aug 16, 2026 · 2 min

Vargo

Agent routing hits an identification wall · verified execution reaches the deployable ceiling

Agent Evaluation · Decision Science

Built with Python · ALFWorld · Agent routing · Verifier cascade · Reproducible evaluation

Bar chart showing that task-side features, 2026 routing methods, deployment bandits, and activation probes fail to identify the per-task winning agent configuration before execution — Every tested pre-execution signal falls short of reliably identifying the per-task winner

WHY

The best combination of agent scaffold, memory, retry, and verifier changed from task to task. Could task text, embeddings, or hidden states identify that best configuration before any candidate was run? If not, the practical question became how much of the performance gap could be recovered by executing and verifying a limited number of candidates.

Evaluation matrix 134 × 29

ALFWorld tasks × agent configurations

Oracle headroom up to +29 pp

Best per-task configuration vs. one fixed setup

Verified execution 54% → 85%

Recovered oracle gap; 100% in the K=11 deployable pool

HOW

Replace winner prediction with bounded execution and verification

The system stops asking which configuration looks best and instead asks whether a candidate has produced a verifiable success.

01 Task and candidates
Hold the task fixed and enumerate the reasoning, memory, retry, and verification configurations that can execute it.
02 Fixed execution order
Run candidates under a bounded budget rather than training another selector on an unstable ranking signal.
03 Outcome verifier
Check the environment result after each attempt and distinguish certified success from plausible-looking traces.
04 Early stop
Return the first verified success together with its execution record; otherwise expose the exhausted budget.

RESULT

Two different decision problems

Approach	Signal available	Observed outcome
Pre-execution routing	Task text, embeddings, hidden states, bandit feedback	Could not identify the per-task winner
Verified execution	Actual candidate outcomes under bounded budget	Reached the deployable ceiling at K=11

Capture curve showing the percentage of oracle gap recovered as the verified execution budget increases from 3 to 13 agent configurations — Corrected ALFWorld matrix · anchor-failure subset n=42 · 90% task-bootstrap intervals

Contribution

Built the 134-task × 29-configuration evaluation matrix and separated oracle headroom from what a deployable selector can actually identify.
Implemented routing baselines, activation probes, contamination controls, and verified-execution capture evaluation; the resulting paper is under review at TMLR.

Evidence

Private project