Enterprise AI agents have a routing problem. Give one access to a library of two thousand tools, ask it to download a dataset, transform the data and generate a report, and watch it drown in options. Feed the entire tool catalogue into the prompt and you burn through hundreds of thousands of tokens before the agent even picks its first move. Researchers at Alibaba built SkillWeaver to solve exactly that bottleneck, and their approach slashes token consumption by more than 99% while lifting accuracy at the same time.

The framework doesn’t rely on a bigger, more expensive model. Instead, it changes how the agent thinks about the task before it ever touches a tool. That distinction matters, and it’s the reason SkillWeaver is worth paying attention to if you’re building anything that resembles a production agent.

Why tool routing breaks at scale

Most current tool-use frameworks handle routing as a one-shot decision. The agent scans its skill library, picks a tool, executes, repeats. That works when you have twenty tools. It falls apart when you have two thousand, which is increasingly the norm as agents plug into ecosystems like the Model Context Protocol.

The naive fix is to stuff every tool description into the LLM’s context window and let the model figure it out. Alibaba’s team tested this with Qwen-Max, one of their larger models. Despite its strong reasoning capability, it only picked the correct tool category 21.1% of the time when flooded with options. Context length exploded to roughly 884,000 tokens per query. That’s not a workflow you can afford to run at scale.

Traditional ReAct-style agent loops did worse. On the benchmark used by the researchers, ReAct achieved 0% decomposition accuracy. Its iterative loop collapses multi-step plans into isolated actions, so a request that needs three coordinated tools becomes three disconnected attempts that never form a coherent plan.

How SkillWeaver works

SkillWeaver reframes tool selection as compositional skill routing. Rather than asking “which tool fits this step,” it asks how to break the whole task into atomic sub-tasks, map each to the best available skill, and wire them together into an executable plan. The framework does this in three stages.

Decompose

An LLM acts as a task decomposer. It reads the user’s request and produces a sequence of atomic sub-tasks, each solvable by a single skill. A request like “download the dataset, transform it, and create visual reports” becomes three discrete steps rather than a vague monolith.

Retrieve

An embedding model compares each sub-task against the full skill library and pulls a shortlist of candidate tools per step. The Alibaba team used all-MiniLM-L6-v2 paired with a FAISS index, both off-the-shelf components. Swapping in BGE-base-en-v1.5 boosted accuracy without any fine-tuning.

Compose

A planner evaluates the retrieved candidates for inter-skill compatibility. Does the output of tool A actually flow into the input of tool B? It then produces a final execution plan as a directed acyclic graph, which lets independent branches run in parallel where possible.

The trick that makes it work: Skill-Aware Decomposition

Here’s where the framework gets clever. LLMs are notoriously bad at guessing the exact vocabulary that a specific tool library uses. Ask a model to describe a step and it might say “fetch data from remote source” when the actual skill in your library is called api-client or http-fetch. That mismatch tanks retrieval accuracy.

Skill-Aware Decomposition, or SAD, fixes this with a feedback loop. The LLM drafts an initial plan. The system runs a preliminary retrieval to find loosely matching skills. Those retrieved skill names then get fed back to the LLM as hints, and the model rewrites its decomposition using vocabulary that actually aligns with the library.

The numbers make the case. With the lightweight Qwen2.5-7B-Instruct as the decomposer, a vanilla setup hit 51.0% decomposition accuracy. Activating SAD lifted that to 67.7%. Running SAD with the larger Qwen-Max model pushed accuracy to 92%. On the hardest queries, those requiring four or five distinct skills chained together, SAD improved accuracy by 50%.

Bigger models aren’t always better

One of the more counterintuitive findings from the paper concerns model size. In a vanilla setup without SAD, a 14-billion parameter model performed worse than the 7-billion parameter model. The reason: bigger models tend to over-decompose. They break simple tasks into microscopic, unnecessary steps that don’t map cleanly to any real tool.

Once SAD kicked in, the retrieved tool hints anchored the larger model back to reality. Accuracy jumped. The lesson is worth sitting with. Aligning your agent with the actual vocabulary of your tool library often matters more than paying for a larger, more expensive LLM.

Token savings and what they mean in practice

SkillWeaver’s targeted retrieve-and-route approach compressed context consumption from roughly 884,000 tokens down to about 1,160 tokens per query. That’s a 99.9% reduction. For any team running agents at scale, that translates directly to lower API costs and faster response times.

Think about what that means for economics. If an enterprise workflow previously chewed through 100,000 tokens per execution, SkillWeaver could bring that under 1,000. Multiply across millions of queries a month and you’re looking at a serious shift in what agentic AI costs to operate.

The benchmark and its limits

To evaluate SkillWeaver, the researchers built CompSkillBench, a custom benchmark of 300 multi-step queries across varied difficulty levels. They sourced 2,209 real tools from the public MCP ecosystem, spanning 24 functional categories including cloud infrastructure, finance, and databases. That’s a realistic proxy for what an enterprise agent might actually face.

Retrieval isn’t perfect, though. Even with a solid bi-encoder, the correct tool lands in the top 10 candidates about 70% of the time, but only ranks first around 37% of the time. To close that gap, most teams will need a secondary cross-encoder or LLM-based reranker to re-sort those top candidates.

What developers should know before implementing

The source code isn’t released yet, but SkillWeaver was built on reproducible, off-the-shelf components. The SAD prompt templates are shared in the paper. You can implement the framework using LangChain, LlamaIndex, or plain Python with a FAISS index and an embedding model of your choice. Embedding and indexing 2,209 skills took the researchers 15 seconds. Retrieval added under 15 milliseconds per query.

One caveat: SkillWeaver handles routing and planning, not error recovery. If an API call fails halfway through the DAG, the chain breaks. Any production deployment needs its own retry logic, fallback paths, and handling for malformed outputs on top of the compose stage.

The real takeaway for agent builders

The most useful insight from SkillWeaver isn’t the 99% token reduction, impressive as that is. It’s the diagnosis. Task-decomposition granularity, not raw model size, is the actual bottleneck in accurate tool routing. Teams pouring budget into frontier models to solve agent orchestration problems may be optimising the wrong variable. Smarter retrieval architecture and vocabulary alignment can outperform brute force at a fraction of the cost, and that shift changes how you should think about building agent infrastructure from day one.