Generated by Codex with GPT-5

What happened

Anthropic’s official Claude blog published the engineering write-up How Anthropic enables self-service data analytics with Claude, a June 3, 2026 post from its Data Science and Data Engineering team about making agentic business analytics reliable enough for routine internal use.

The post is interesting because it rejects the simplest story about analytics agents. The hard part is not usually writing SQL. It is mapping an ambiguous business question to the one governed metric, table, grain, filter, time window, and ownership boundary that will make the answer correct. In normal analytics work, a human analyst carries that context: which revenue table is canonical, which user definition excludes abuse, which dashboard is blessed, which migration changed a field, and which stakeholder phrase refers to which launch. An agent pointed at a warehouse does not inherit that tacit map just because it can generate syntactically valid queries.

Anthropic frames the production problem around three failure modes. Concept-to-entity ambiguity makes the agent choose the wrong field or definition among many plausible ones. Staleness makes once-correct schemas, docs, metrics, and skills drift as the business changes. Retrieval failure means the right answer may exist somewhere in the warehouse, docs, dashboards, or query history, but the agent cannot find and apply it reliably. The architecture is built around narrowing those spaces before the model writes a query.

The reported internal result is substantial: Anthropic says Claude now automates about 95 percent of business analytics queries at roughly 95 percent aggregate accuracy. The number is less important than the mechanism. The system works because the team treated analytics agents as data infrastructure with governance, evaluation, CI, ownership, and operational telemetry, not as a generic chatbot with database credentials.

The architecture

The first layer is ordinary but non-negotiable data engineering. Anthropic emphasizes canonical datasets, dimensional modeling, shift-left tests, freshness checks, completeness checks, ownership, lineage, and maintained metadata. This matters more for agents than for expert analysts because the agent is often serving a user who cannot evaluate the correctness of the underlying query. If “active user” resolves to forty plausible tables and definitions, the model has already been set up to fail.

The next layer is a hierarchy of sources of truth. The semantic layer is the preferred path: if a question maps to a defined metric or dimension, the agent should call that layer rather than hand-writing its own interpretation. When the semantic layer does not cover a question, lineage and transformation graphs help the agent understand which models feed which concepts, which tables are deprecated, and where grains differ. Historical SQL is treated more cautiously. Anthropic found that giving an agent direct retrieval over thousands of old queries did not materially improve accuracy, because the agent could often see a relevant precedent and still fail to map the new question to the right entity.

That negative result is one of the post’s strongest engineering lessons. More context is not the same as better structure. Query history becomes useful only after it is distilled into curated domain references, reusable analysis patterns, and explicit routing rules. In other words, the system does not ask the model to discover the shape of the data estate from raw artifacts on every request. It gives the model a maintained operating manual.

Skills provide that procedural layer. In Claude Code, a skill is markdown that the agent loads on demand. Anthropic uses paired skills: a knowledge skill routes the agent to the right domain references, while an analysis skill describes the workflow a senior analyst would follow. The agent is instructed to clarify the request, prefer the semantic layer, fall back to curated references when needed, run the query, and send the result through an adversarial review step before answering. Skills also bundle common analysis patterns, such as retention curves or funnel analysis, so repeated questions do not require rediscovery.

The maintenance strategy is as important as the runtime path. Anthropic colocates data models, semantic definitions, dashboard definitions, and skill documentation in the same repository. A model change that would invalidate a dashboard or doc can be caught in CI, and a review hook flags reporting-model changes that do not update the relevant skill file. The post says about 90 percent of data-model pull requests now include a skill change. That turns skill maintenance from a separate documentation chore into part of the data model’s change contract.

Evaluation and operations

Anthropic’s eval approach is built around the same assumption: correctness depends on the data model, not just the model. Offline evals are question-and-answer pairs, but they need stable grounding. If a metric changes every day, the eval must pin a snapshot date, query a stable fact table, or grade the query logic instead of the live number. Each run is stored with skill version, git SHA, model ID, pass/fail details, token count, and wall-clock time, which makes eval history queryable as production telemetry rather than disposable test output.

The team uses dashboard-derived evals for common stakeholder questions, long-tail evals generated from business context and table docs, and correction-derived evals harvested from real user feedback. Launches are gated by domain, so a team cannot publish an analytics agent for a business area until that area’s eval slice clears a threshold. This is a useful organizational pattern: ownership of data quality, reference docs, and eval performance stays close to the domain.

Ablations keep the design honest. Anthropic reports that exposing raw query corpora moved accuracy by less than one point, even when the needed answer was present much of the time. That pointed the team away from broader retrieval and toward stronger structure. Other experiments also produced negative results, such as over-expanding docs until they became harder to use, or replacing the adversarial reviewer with a cheaper model and losing most of the accuracy benefit.

Online validation adds production controls around the agent’s actual answers. Adversarial review improved eval accuracy, but with notable token and latency costs. Provenance footers expose whether an answer came from the semantic layer, curated references, or raw exploration, plus freshness and ownership signals. Data quality checks catch cases where the agent used the right field but the underlying pipeline is wrong. Passive monitoring tracks how often answers resolve through the semantic layer and how often users correct the agent with phrases that indicate wrong tables, filters, or assumptions.

The correction loop is especially practical. A scheduled agent scans stakeholder channels for correction language, drafts a small reference-doc fix, and opens a pull request for the domain owner. The path is deliberately simple: edit markdown, review, merge, and sync the updated skill to the surfaces where analysts and business users ask questions. The unresolved risk is silent failure, where an answer looks plausible and nobody corrects it. Anthropic mitigates that with provenance, human sign-off for leadership-bound analysis, and daily sanity checks for top KPIs, but the post is clear that this class of failure is not fully solved.

Why it matters

The broader takeaway is that analytics agents are a forcing function for better data governance. A human data scientist can often work around messy lineage, tribal metric definitions, stale docs, and ambiguous dashboards. An agent serving non-expert users amplifies those weaknesses. If the warehouse has no single source of truth, the model will invent one by choosing among plausible candidates. If docs are stale, the agent will confidently operationalize stale knowledge. If evaluation is not tied to domain ownership and CI, quality will decay after launch.

The post also gives a useful counterweight to context-maximalist agent design. Giving the model more files, more queries, and more warehouse access did not solve the central problem. What helped was making the correct path narrower: canonical models, semantic-layer-first routing, curated references, skill-triggered procedures, adversarial review, and feedback loops that update the instructions as the data estate changes. The model remains important, but the system’s reliability comes from constraining and maintaining the environment around it.

For teams building internal agents, the implementation lesson is concrete. Start with a small set of canonical datasets, a thin knowledge skill, and a few dozen evals for a domain that matters. Measure whether the agent finds the semantic layer, whether its answer matches a stable ground truth, and whether users correct it in production. Then invest where the failures actually occur. If the problem is ambiguity, clean up the model and docs. If the problem is retrieval, add curated routing. If the problem is drift, put skills and metadata under the same CI discipline as data code.

Takeaway

Anthropic’s post is less about replacing analysts than about turning repeated analytics work into a governed agent workflow. Claude can answer many routine data questions, but only when the system gives it a maintained map of the business, a ranked set of trusted sources, a procedure for resolving ambiguity, and a way to learn from corrections without letting docs rot.

The engineering pattern generalizes beyond analytics. Production agents need more than tools and context windows. They need canonical sources, procedural skills, evals that reflect real work, cost and latency measurements, provenance, ownership, and a maintenance loop that keeps their world model synchronized with the systems they operate over.