Artificial Analysis Rebuilds AI Intelligence Index Around Private, Real-World Agent Tests

Artificial Analysis has released Intelligence Index v5, a major revision of its composite system for comparing language models across reasoning, knowledge, mathematics, programming, and agentic work. The update makes 60% of the index depend on private test sets or held-out answers, a shift intended to reduce training-data contamination and make scores harder to optimize directly.

The index combines 9 evaluations into a single score while preserving four broad category weights: agents at 30%, general capability at 30%, coding at 20%, and scientific reasoning at 20%. Artificial Analysis describes the measure as a practical synthesis rather than a definition of intelligence, an important distinction because benchmark scores capture performance on selected tasks rather than a model's full capabilities.

The central change in v5 is the move from widely circulated public benchmarks toward private evaluations and held-out answers. Artificial Analysis introduced FrontierCode 2.0, Cognition's private coding benchmark, which grades whether a repository maintainer would merge an agent's proposed change. It also added Terminal-Bench-Science, a set of 70 agentic scientific research tasks performed in a terminal environment.

These additions measure work closer to deployment: modifying real codebases, completing multi-step terminal tasks, and conducting technical research. VentureBeat described the overhaul as a replacement of popular benchmarks with real-world tests, while Artificial Analysis framed the change as an effort to keep its index useful as models become better at fitting public evaluations. Private testing improves resistance to memorization, but it also reduces outside reproducibility because researchers can't inspect every task or answer.

V5 adds 2 new benchmarks and upgrades 3 existing evaluations. The changelog identifies FrontierCode 2.0 and Terminal-Bench-Science as new components, while HLE-Diamond replaces Humanity's Last Exam. Other updates include refreshed agent and science evaluations, with the exact composition reflected in the index's methodology and model pages.

The revision follows several months of incremental changes. Version 4.1 shifted more weight toward agentic workloads, version 4.1.1 upgraded grader models and τ³-Banking, and version 4.3 upgraded Terminal-Bench to 4.0 while adding AutomationBench-AA, a private workflow-automation test. In v4.3, Claude Fable 5.1 led on AA-Briefcase and SciCode, while GPT-6 Astra led on Terminal-Bench 4.0 and AutomationBench-AA, illustrating why a single total can conceal sharply different capability profiles.

The index gives developers and buyers a common reference for comparing quality, cost, and task performance. Artificial Analysis also presents scores alongside weighted cost per index task and provider metrics such as speed and latency, allowing users to weigh capability against operational expense rather than treating the highest score as the automatic choice.

A model's composite result can differ substantially from its performance in a specific job. Artificial Analysis's provider comparison ranks Claude Opus 5.5 ahead of GPT-6 Astra across several professional categories, while other index views show different leaders by evaluation. A separate leaderboard listing 245 models places GPT-5.6 Sol at 58.9%, Claude Opus 5.5 at 57.6%, and Claude Sonnet 5.5 at 56.0%, but that table compiles provider and secondary reports and doesn't determine the official rankings.

For builders, v5 makes benchmark selection more closely match the work agents are expected to perform. Repository-level coding, terminal operation, scientific investigation, and workflow automation test persistence, tool use, error recovery, and task completion, not just isolated question answering. That makes the index more relevant to teams choosing models for software engineering or research operations.

The tradeoff is transparency. Private datasets protect evaluation integrity and make score chasing harder, yet they prevent independent reproduction of much of the result. Users should therefore read the per-task breakdown, cost data, and methodology alongside the headline score. The index is best treated as one decision input, with domain tests added for deployments where failure costs matter.

Artificial Analysis v5 reflects a broader transition in AI evaluation from static public exams to controlled tests of useful work. The emphasis on private data, agentic environments, and merge-ready code responds to models that increasingly operate through tools and across multiple steps rather than only generating standalone answers.

The next pressure point will be trust. Evaluation providers must show enough methodology for users to understand what a score means while withholding enough test data to preserve its value. Artificial Analysis's continuing version updates show that benchmark systems are becoming maintained products, with new tasks, grader changes, cost tracking, and revised weighting. For model vendors and buyers alike, leaderboard position now matters less than whether performance survives private, task-specific evaluation.