The world will spend over a hundred billion dollars on AI agents this year alone. A big portion of that will be wasted on excessive reasoning chains. To curb this, we trained a decision model as a proof of concept for precise, automatic effort selection.
The router reads the task before an agent starts and sets the reasoning effort. Trained on DeepSWE and tested on Terminal-Bench 4.0, it matches the pass rate of high effort at 15% lower cost. Picking max effort is even worse. It costs 69% more than our router for a 0.3% better score.
This router was trained specifically for GPT-6 Astra. However, the recipe we developed applies to any model. It requires under a thousand agentic trajectories and trains within a single hour.
Results
In the chart below, each point on the line is GPT-6 Astra running every task at one effort level. The highlighted point is the router, which picks the effort for each task. Its score, cost and time come from the same recorded runs, using each task’s results at the effort the router picked.
Terminal-Bench 4.0
Router trained on DeepSWEOn Terminal-Bench 4.0, the router sent about two-thirds of the tasks to high effort. It also finished tasks about 7% faster than high effort. Medium effort costs about the same as the router, but solves fewer tasks.
The result holds in the other direction. Trained on Terminal-Bench 4.0 and tested on DeepSWE, the router scored slightly above high effort at 13% lower cost and in 9% less time.
Hard is the wrong question
The obvious approach is to estimate how hard a task is and spend more on the hard ones. But a hard task can fail at any effort: it may depend on a fact the agent can’t find, a service that isn’t available or a requirement it misread. More reasoning just buys a longer attempt at the same mistake. Meanwhile, a short request can hide a detail that only a careful pass will catch.
What matters is how the outcome changes with effort. Some tasks succeed no matter the effort, and some fail no matter the effort. Extra reasoning only pays off on the tasks in between, and those are what the router learns to find.
To learn them, we use recorded runs of GPT-6 Astra on DeepSWE, where every task was attempted several times at different efforts. A single attempt is a noisy label, so we estimate each task’s success rate at each effort, smoothing the counts with a uniform prior so that a handful of attempts never looks like certainty. Attempts that returned no grade are left out. The training target weighs the gain in success against the extra cost:
The price weight sets how much one point of success is worth in extra spending. It comes from the training data, not from what users pay. The router learns to rank tasks by this value, and a threshold turns the ranking into a decision.
The model
The router is built on Laya, an open-source decision model that pairs the ModernBERT-large text encoder with a small decision head. We freeze the first 22 encoder blocks and the decision head and train low-rank (LoRA) adapters in the last six: about 770,000 parameters.
It reads only the task prompt and turns it into two kinds of features: a pooled representation of the text, and the decision head’s answers to a few fixed questions about whether the task needs more effort. Each question is asked with its options in both orders and the answers are combined, so option order can’t sway the result.
During training, the model also learns to predict 80 properties of each task, rated in advance by a hosted decision model. The ratings teach the encoder what to pay attention to. The router never calls that model at inference, and it never waits for the agent to try the task.
Small kernel ridge regressors map the features to two estimates of the value of more effort: one through the task properties and one directly from the prompt. Each is averaged over 16 regularization settings, so neither hinges on a single tuned value. The router blends the two and compares the result with a threshold to set the task’s effort.
The score ranks tasks. It is not a calibrated probability of success. Each task is decided on its own, so other tasks in a batch never change its effort. The router doesn’t rewrite the request, propose a solution or retry a failure. It sets the starting effort and leaves the rest to the agent.
Training for transfer
A router is only useful on tasks unlike the ones it learned from, so we train on one benchmark and test on the other. Nothing from the test benchmark enters training, neither its outcomes nor its property ratings.
Within the training set, we split tasks into three folds, keeping related tasks together. A small predictor is fitted on two folds and scored on the third, and we backpropagate through it into the adapters. Rotating the held-out fold rewards features that carry over between groups of tasks, not features that only fit one split.
The loss combines the error in predicting the value of more effort with an error on the task properties, plus a penalty that keeps the features close to where they started. We stop after a set number of updates, fixed before training, and fit the final regressors on the full training set.
Long prompts are split into windows of up to 4,096 tokens and combined in proportion to their length, so the end of a request is never cut off. Training took no new agent runs: it reuses the trajectories we had already recorded.
Training recipe
- Base model
- Laya (ModernBERT-large encoder), pinned revision
- Trainable weights
- Rank-8 LoRA in the final 6 encoder blocks · 771,072 parameters
- Frozen weights
- First 22 encoder blocks and the decision head
- Training signal
- Outcomes and attempt costs at different efforts, and 80 property ratings, from the training benchmark only
- Optimization
- AdamW · learning rate 0.003 · weight decay 0.01
- Stability
- Gradient norm clipped to 1 · features anchored to their initial values
- Context
- 4,096-token windows · every prompt token kept
- Readout
- Kernel ridge · 4 × 4 regularization settings
- Decision
- One effort per task · threshold 0.45
What we learned
Keeping the decision narrow is what made it learnable. Two runs of the same agent at different efforts differ only in the reasoning budget, so each task’s label measures exactly what the router has to predict.
More elaborate designs didn’t help. We tried routing directly with the hosted model, readouts on a frozen encoder, other semantic views, separate cost models and longer adapter training. None consistently beat this simple design on both cost and pass rate.
Moving to a new model changes only the data. A router learns how one model responds to more reasoning, so each model needs its own, trained on its own trajectories. The architecture, objective and training procedure stay the same, and the property ratings can be reused, since they describe tasks rather than models. The cost mostly comes from the new data, and the threshold has to be chosen again.
Prompt caching
Until recently, changing a model’s reasoning effort in the middle of a conversation invalidated the prompt cache. The whole context had to be processed again at full price, which made switching effort expensive after the first request.
OpenAI now lets developers change GPT-6’s reasoning effort mid-conversation while keeping the prompt cache. See OpenAI’s documentation
Codex and ChatGPT Work also keep the cache when reasoning changes mid-conversation, as OpenAI’s Tibo announced on October 23. Read the announcement
That matters for the router today. In a real session, each new request can be routed on its own: a quick follow-up gets less effort and a demanding one gets more, without paying to reread the conversation. We urge other labs to follow suit.
What’s next
In a critique of a related product from another company, developer and YouTuber Theo Browne raised an objection: a prompt alone rarely shows how complex a task is. That depends on the codebase, the tools the agent can use and the system around it. Read Theo’s post
We agree. This router decides from the first prompt. It has learned which kinds of tasks tend to reward extra effort, but it can’t see what the agent will find in the repository.
The next step is to let the agent look first and adjust its effort as tool output and test results come in. With the cache preserved, each adjustment is cheap.
We’ll compare that approach with the current router and with fixed effort, counting the cost of inspection and retries alongside pass rate and time, to see when gathering more information pays for itself.
This is research, not a shipped feature. We’ll keep developing the method, train routers for more models and later bring it to Rogue’s products.