Agentic data analytics with Claude sounds simple in theory. Point an agent at the warehouse, let a stakeholder ask a question in plain language, get a number back. The number arrives fast, formatted well and stated with confidence. The hard part is knowing whether it is right.
Anthropic’s own data science and data engineering team now automates roughly 95% of internal business analytics queries with Claude, at around 95% aggregate accuracy. That freed the team to work on causal modelling, forecasting and machine learning instead of answering the same questions week after week. The interesting part is not the headline number. It is what had to be built underneath it, because the first version of that system scored below 21% on their own evaluations.
Why analytics agents behave differently from coding agents
Coding is an open ended solution space. Creativity is rewarded, and tests, type signatures and documentation act as natural guardrails. If the agent hallucinates, something usually breaks loudly.
Analytics offers no such feedback loop. There is often exactly one correct answer, drawn from one correct source, and no compiler or test suite waiting to flag a wrong result. A query that joins the wrong table still runs. A number that measures the wrong thing still looks reasonable. That is why Anthropic frames accuracy as a context and verification problem rather than a code generation problem. Once you know which entities a question maps to and how to work with them, writing the SQL is the trivial part.
The three failure modes behind most wrong answers
Anthropic isolated three attributes that explain the overwhelming majority of inaccurate responses.
- Concept to entity ambiguity. A mature warehouse holds hundreds of plausible fields for a single business concept. Ask for active users and the agent has to decide what counts as active, whether fraudulent accounts are included and which lookback window applies. Nobody wrote those rules down, so it guesses.
- Data staleness. Sources, definitions and schemas change constantly. Documentation and agent knowledge drift, and the answers turn subtly wrong while still looking plausible.
- Retrieval failure. The right table exists, properly annotated, and the agent still fails to find it in a search space of millions of fields.
The agentic analytics stack layer by layer
Each layer of the stack exists to attack one or more of those failures.
Data foundations collapse the ambiguity
Dimensional modelling, shift left testing and freshness checks all still apply. What changes is the consumer. Your data model is no longer read by a data scientist who can smell a bad number. It is read by agents acting for people who cannot validate the underlying logic at all.
The most effective fix is fewer, more heavily governed models. Curate a small set of canonical datasets that are clearly owned, consumption ready and discoverable, then aggressively deprecate the near duplicates. If revenue resolves to one governed dataset instead of forty candidates, most of the ambiguity disappears before the agent ever searches. Rollups and caches still matter for cost and performance, but they should derive mechanically from the canonical models rather than sit beside them as alternatives.
Governance only holds when it is enforced. Route the agent structurally to the canonical layer, fail CI on changes that bypass it and require downstream teams to build on the governed layer or explain why not. Anthropic also colocates nearly all data code in a single repository, so a modelling change that would break a downstream dashboard gets flagged and fixed in the same pull request.
Sources of truth in descending order of trust
Above the warehouse sit the reference surfaces the agent consults. The semantic layer ranks first. When a question maps cleanly to a defined metric, the agent calls a function and receives the same number every other surface in the company produces. Anthropic’s skills structurally require the semantic layer to be tried first.
One experiment failed here and it is worth remembering. Bootstrapping the semantic layer by having an LLM auto generate metric definitions from raw tables and query logs produced plausible looking definitions that encoded exactly the ambiguities the layer was meant to remove. It scored net negative against a smaller, human curated layer. Generate the documentation with Claude, but let a human own the definition.
Below that sit lineage and the transformation graph, which let the agent reason about which upstream models feed a concept and which are deprecated. Then comes the query corpus, and this is where intuition breaks down. Giving the agent raw retrieval access to thousands of historical queries moved accuracy by less than a point. The last layer, business context, is the one most teams skip. Without indexed docs, roadmaps, decision logs and an org chart, the agent answers what was asked rather than what was meant.
Skills as procedural knowledge
If sources of truth are declarative knowledge, skills are procedural. A skill in Claude Code is a folder of markdown the agent reads on demand, and it is the single biggest accuracy lever in the whole stack. Without skills, accuracy never exceeded 21%. With them it sits consistently above 95% in aggregate and near 99% in some domains.
Anthropic uses pairwise skills. A knowledge skill acts as a thin router that says try the semantic layer first, and if there is no coverage, here are roughly thirty reference files describing the tables, joins and gotchas for this domain. That router is the answer to retrieval failure, because it narrows a million field warehouse to a few dozen curated files before a query is written. A runbook skill encodes the process a senior analyst would follow, including reusable patterns for retention curves, rate decomposition and funnel analysis.
Skill maintenance is not optional. Anthropic watched offline accuracy drift from 95% to 65% within a month before treating documentation as an engineering problem. Colocating skill markdown with the transformation models fixed it, helped by a review hook that flags any reporting model change without a matching skill edit. Around 90% of data model pull requests now carry a skill change in the same diff.
Validation, ablations and the value of null results
Offline evals are question and answer pairs. Anchor them to a snapshot date or a stable fact table so ground truth cannot drift, store results like telemetry rather than test logs, and gate each domain launch on a pass rate threshold before stakeholders ever see it. A few dozen evals per topic hits diminishing returns.
Structural decisions get settled by ablation. Hold the eval set fixed, vary one component, compare pass rates. The most useful result was negative. When the agent got grep access to thousands of dashboard, transformation and notebook SQL files, transcripts confirmed it read them, yet accuracy barely moved. The answer was present in the corpus about 80% of the time and that made no difference to whether the agent got it right. The bottleneck was never access to prior work. It was structure.
Online, adversarial review by a second Claude instance added 6% accuracy at the cost of 32% more tokens and 72% higher latency. Every response carries a provenance footer showing the source tier, data freshness and model owner. A footer reading raw table with unknown freshness tells the reader to verify before forwarding. A scheduled agent scans stakeholder channels for correction language, drafts a one line fix to the relevant reference doc and opens a pull request tagged to the domain owner.
Where the wider industry lands on this
The same pattern shows up outside Anthropic. Benchmarks from dbt Labs found that questions covered by a well modelled semantic layer approach 100% accuracy because query generation is deterministic, while text to SQL stays flexible but fragile. Analysis of thousands of incorrect queries attributes roughly 81% of failures to schema level errors such as wrong column selection and semantic misinterpretation, not syntax. Better models do not remove that gap, because the model is not the constraint.
What to build first, and the failure nobody has solved
Starting from zero, a handful of canonical datasets, a few dozen offline evals and a thin knowledge skill capture most of the upside. Everything else is what you add once those exist. Then calibrate against your own situation. How technical is the audience reading the output, how fast is your data model changing, how much latency and token cost will you trade for accuracy, and how comfortable are you giving one agent broad context versus scoping several narrower ones.
Worth sitting with, though, is the failure none of this fully catches. The silent one. The answer is wrong, it looks entirely plausible, and it gets forwarded into a board deck without a single objection. Provenance footers, human sign off on leadership bound work and daily KPI sanity checks against a blessed dashboard reduce the odds, but no team has a robust solution yet. Which quietly raises the bar for the humans in the loop rather than lowering it. The skill that matters most in an agentic analytics setup is no longer writing the query. It is recognising when a number deserves a second look.
