The cost-efficiency benchmark for AI coding models

A coding score from independent benchmarks

Each model gets a coding score from 0 to 100, measured independently by Artificial Analysis:

Effort level: more thinking isn't always worth it

Many models have an effort setting (low, medium, high, xhigh, max). It controls how long the model thinks before it answers. The price per token doesn't change, but a higher level uses more tokens, takes longer, and usually gets a somewhat higher success rate. For strong models the gain shrinks fast: some score best at medium, and a few get slightly worse at the top level.

So each model is ranked once, at the level with the lowest cost per finished task: past that point, the extra thinking costs more than it saves in fixes. Tap "Use at" in the table to see every level.

How much a level thinks: Artificial Analysis times every level (seconds until the answer starts × tokens per second ≈ thinking tokens). Where a model has that for two or more levels, we use the measured gap (square-rooted, since those test prompts are short). Otherwise a level gets a typical ratio: low 0.35×, medium 0.6×, high 1×, xhigh 1.7×, max 2.5× the thinking of high.

• Token use can differ up to 23× between the lowest and highest effort of the same model. Artificial Analysis
• At max effort, GPT-6 Astra uses about 27K output tokens per task and Claude Fable 5.1 about 78K, for the same score. Artificial Analysis
• Extra effort barely helps strong models (+0.6 points for GPT-5.5 from low to xhigh) but lifts weak ones much more. Digital Applied
• Claude Opus 5 scores highest on FrontierCode at medium effort. SitePoint

Brand-new models don't wait

Artificial Analysis usually publishes a new model's general Intelligence Index within a day, but its Coding Index can take longer. A model released in the last 60 days without one gets a provisional estimate (marked ^*) from the models with the most similar Intelligence Index. It's replaced automatically once the real score is out, and it never sets the quality bar.

Your kind of work

Reading a repo, building a new API service, changing legacy code and chasing a production bug through logs and traces are different jobs. Developers spend about 58% of their time understanding code, not writing it. At Anthropic, the daily uses are debugging (55%), code understanding (42%) and new features (37%). Pick your kinds of work under "Coding tasks" and each model is scored on the benchmark built for that work:

These benchmarks measure a success rate on that kind of task: the share of codebase questions answered correctly, of real incidents diagnosed, of repo changes that pass the hidden tests. That rate replaces the general coding score as the chance a model gets your task right first time. For LMArena WebDev, which is a rating, it's the chance developers prefer the model's web app over an average model's. A model that isn't on a benchmark yet is estimated from its general coding score using the fit across measured models, minus one standard error, so an unproven model doesn't get the benefit of the doubt. It's marked "estimated" in the table. With nothing picked, the general coding score is used.

Why dedicated benchmarks: an earlier version mixed general benchmarks per kind of work. Checked against SWE-Atlas, the mix agreed with real codebase-understanding results at only r = 0.34, worse than the plain coding score (0.61). So each kind of work now uses a benchmark made for it. SWE-Atlas scores include the agent harness (Claude Code, Codex, Mini-SWE-Agent). Part of the DeepSWE scores are the vendors' own reports; independent runs are used where they exist.

Only near-top models qualify

What a finished task costs you

cost per task = (price × tokens + fix cost × failure chance) ÷ success chance

The cheapest model per token isn't the cheapest to work with. A weaker model gets more tasks wrong, and every miss costs you a re-prompt, a review and a fix, plus another run's tokens. We treat the coding score as the chance a model gets a task right first time (81 ⇒ 81%). Dividing by that chance counts the retries.

• Tokens per task: 270K. Artificial Analysis measured real agent tasks at $4.10 for Claude Opus 4.7 and $4.82 for GPT-5.5. At their list prices that's about 270K tokens a task. Source
• Thinking share: 40%. A typical multi-file agent task reads about 1.5M tokens (90% from cache) and writes about 40K. Output costs about 5× input and cached reads about 0.1×, so the model's own output is roughly 40% of the token bill. That's the part that grows with the effort level. Source
• Your hour: $93. A US software developer's median wage is about $65 an hour, and wages are about 70% of what an employer pays in total. BLS wages · BLS employer costs
• Minutes per failed attempt: 20. In 20,574 real sessions, 91.5% of problems were fixed only after the developer explicitly corrected the agent: spot it, re-prompt, re-review, about 12 minutes. The rest need hands-on work: METR measured 26 minutes to make even test-passing agent code acceptable, 42 minutes on average. Sessions study · METR
• Why this matters: 66% of developers name "almost right, but not quite" AI code as their top frustration, and 45% say debugging it takes longer. Stack Overflow 2025

Cursor models

Who is left out, and why