Comparison

AI model comparison, October 2026: benchmarks, costs, and practical choices

A detailed guide to GPT, Gemini, Claude, GLM, Kimi, DeepSeek, and Qwen, with reported benchmark scores, worked costs, and a practical evaluation plan.

By Saud AlotaibiPublished 3 October 202614 min readSources checked 3 October 2026
The takeaway

Compare exact versions and matched evaluations, then choose by accepted outcomes, total cost, and the work you actually do.

The most useful question in AI this week is what you get for the money, time, and supervision you put in. A model can lead a coding benchmark and still be awkward for your document workflow. Another can cost less per token while using so many extra steps that the saving disappears. The releases arriving at the end of September make those tradeoffs especially interesting.

This guide compares GPT-6.1 Sol, Gemini 4 Argon, and Claude Sonnet 5.5, then puts Chinese models alongside established competitors using results published by a benchmark operator. It includes actual reported scores, the settings behind them, and worked examples you can adapt. Sources were checked on 3 October 2026. Benchmark results belong to the named publisher; Fieldnote has not run an independent head-to-head test.

My editorial position for Saud Alotaibi’s Opinions is straightforward: choose a model for a recurring job, and make it earn its place with observable results. The tables below narrow the search. The evaluation plan near the end explains how to turn that search into a decision.

What is available, and what is still a rollout promise?

Before comparing intelligence, check whether the product is usable in your environment. An announced price or a research score does not establish that your account can access the release.

Swipe or scroll sideways to see every column.

ReleaseAccess reported by its providerDecision for a reader today
GPT-6.1 SolWork, Codex, and API; ordinary Chat availability is separate [1]Confirm the interface and account you intend to use.
Gemini 4 ArgonPhased access beginning with trusted cyber defenders; wider availability follows [2]Verify access before building a delivery plan around it.
Claude Sonnet 5.5Claude Platform and major cloud platforms [3]Check your platform’s model identifier and configuration.

For someone trying to finish a project this week, availability is part of capability. A promising release can belong on a watchlist while another earns the immediate workload. Keep both decisions in view: what should I use now, and what should I revisit when access changes?

A common benchmark: coding results from DeepSWE

The table reproduces selected Datacurve DeepSWE v1.1 results: 113 tasks, mini-swe-agent, displayed update 22 September 2026 [4]. Checked on 3 October, the retrieved snapshot omits Sol, Argon, and Sonnet 5.5; their newer provider claims follow separately.

Swipe or scroll sideways to see every column.

Model in the operator’s snapshotReasoning effortPass rate, with displayed uncertaintyAverage cost per attempted task
GPT-6 Astraxhigh74% ±3%$4.43
Gemini 3.8 Flashhigh74% ±1%$2.36
Claude Opus 5max74% ±4%$11.84
GLM-5.3max69% ±3%$3.99
Kimi K3max69% ±5%$4.65
GLM-5.3 Flashmax63% ±4%$0.24
DeepSeek V4 Promax63% ±6%$1.67
Qwen 3.8 Maxxhigh57% ±3%$3.73

Source: Datacurve [4]. Costs are per attempted evaluation task, not token-rate quotations. Different effort settings mean different compute budgets; displayed uncertainty is preserved.

Three things matter when reading this table. First, a rounded tie does not establish equal performance on every task. Second, overlapping uncertainty calls for caution about small gaps. Third, price and capability form a tradeoff: a lower score can be acceptable for an easier workload if the outputs clear the same quality bar. Decide what failures cost before chasing the cheapest row.

The snapshot also gives Chinese models a concrete place in the comparison. Evaluate the exact model and configuration shown. A family name is too broad to tell you what a particular release will do.

New release claims: useful evidence, separate comparisons

These results come from provider announcements. They are worth studying, but the rows do not form a shared leaderboard. Each describes its own evaluation, benchmark version, and comparison. A missing score means it was not verified for this guide; it does not mean zero.

Swipe or scroll sideways to see every column.

Publisher and modelEvaluationReported resultHow to interpret it
Google: Gemini 4 Argon [2]DeepSWE v1.177.9%A provider-reported result, separate from the operator snapshot above.
Google: Gemini 4 Argon [2]AutomationBench51.3%Business workflow evaluation; it cannot be averaged with coding scores.
OpenAI: GPT-6.1 Sol [1]DeepSWE v1.1Matches Astra at roughly one-fifth the costRelative claim; the retrieved text does not give an absolute Sol score.
OpenAI: GPT-6.1 Sol [1]AutomationBench2.2 percentage points above Opus 5.5 at medium effortA setting-specific comparison.
Anthropic: Sonnet 5.5 [3]Terminal-Bench 4.070.6%; Opus 5.5: 66.4% at xhighA terminal-task comparison under the published configurations.
Anthropic: Sonnet 5.5 [3]FrontierCode 1.1, main46.2% at max; 52.1% at xhighMore reasoning did not produce the highest score.
Anthropic: Sonnet 5.5 [3]OSWorld 2.180.1% partial scorePartial credit is distinct from full task completion.

The FrontierCode result is a particularly useful reminder to test settings. Anthropic’s footnote says extra review at max sometimes produced timeouts or changes beyond the task’s scope [3]. That has a practical lesson: a capable agent still needs a bounded assignment. More activity can create more work for the reviewer.

Argon’s reported coding result warrants close attention. It does not establish a verified numerical gap against the operator’s earlier rounded Astra result, because the two publications are different evaluation reports. To make that comparison defensible, obtain the exact configuration, task set, scoring rules, and matched run data. The same standard should apply to every provider.

What the benchmark actually measures

A coding score can describe repository changes, terminal operations, or whether a proposed patch meets a merge standard. Those are related skills, but they reward different behavior. A model that explores broadly may uncover a useful improvement while losing points for changing something the assignment did not request.

DeepSWE’s v1.1 methodology grades committed code in a fresh, isolated environment and fixes dependency drift and flaky tests [5]. That distinction matters: what survives outside the agent’s working session is what a user ultimately receives. When reading any coding result, look for a verifier that checks delivered behavior, then inspect whether its tasks resemble your language, repository size, and acceptance rules.

Computer-use results need an additional check. OSWorld’s authors distinguish binary completion from partial reward [6]. Imagine an agent that collects all the information for a form, fills most fields, and then fails to save it. Partial credit can show substantial progress. The user’s job remains unfinished. Decide which metric reflects the consequence you care about before interpreting a percentage.

Percentage points and relative percentages also differ. Moving from 50% to 55% is a gain of five percentage points and a relative increase of 10%. Neither establishes a 10% reduction in your review time. Avoid translating one measure into another without evidence.

A useful benchmark note should answer six questions:

  • Which exact model or snapshot was evaluated?
  • Which benchmark version, subset, and scoring rule were used?
  • Which agent, tools, permissions, and prompts were supplied?
  • What effort, token, time, and retry budgets were allowed?
  • Who ran the evaluation, and where are the methods or trajectories?
  • What uncertainty and known limitations accompany the score?

When a headline omits those details, treat it as a reason to investigate. It has not yet supplied everything needed for a purchasing or deployment decision.

Pricing: equal token rates can produce different bills

These are the rates published in the cited announcements, in US dollars per million tokens, checked on 3 October 2026 [1–3]. Argon’s rates are announced introductory pricing; its broader rollout remains phased.

Swipe or scroll sideways to see every column.

ReleaseStandard inputCached input or cache readOutput
GPT-6.1 Sol [1]$2$0.10$10
Gemini 4 Argon [2]$2$0.10, calculated from announced 95% discount$10
Claude Sonnet 5.5 [3]$2$0.20$10
Claude Opus 5.5 [3]$4$0.20$20

Cache creation, minimum eligible prefix size, retention, context tiers, platform charges, and other conditions can alter a real bill. Confirm those details with the endpoint you will use. A cache-read rate alone does not establish that two providers’ caching arrangements are equivalent.

Here is a deliberately simple calculation, not a measured model run. Assume 100,000 uncached input tokens and 10,000 billable output tokens. At $2 input and $10 output per million, the token bill is $0.20 + $0.10 = $0.30. That arithmetic is identical for the first three rows under those assumptions. It says nothing about how many tokens each model would actually need.

Now suppose 80,000 input tokens qualify for cache reads, while 20,000 are uncached. At a $0.10 cached rate, input costs $0.008 + $0.04 = $0.048. Add the same output and the total is $0.148. At a $0.20 cache-read rate, the analogous total is $0.156. Both examples exclude cache creation and other charges, and assume identical billable usage. They illustrate a rate difference rather than a complete provider cost comparison.

The more useful calculation includes failed attempts:

Cost per accepted task =
  (all model charges + tool charges + review cost + repair cost)
  / number of accepted tasks

For another hypothetical example, Model A costs $0.20 for each of 100 attempts and produces 60 accepted results. Its model-only cost per accepted result is $20 / 60, about $0.33. Model B costs $0.30 per attempt and produces 90 accepted results: $30 / 90 is also about $0.33. The cheaper attempt has not produced a cheaper accepted outcome. If A also needs substantially more repair, the practical balance changes again.

These are invented scenarios for explaining the calculation, not measured scores for any named model. In your own comparison, count every retry and inspect the actual billing fields, including reasoning usage where applicable. Never substitute visible answer length for billed output.

Chinese models: compare versions, deployment, and the job

The operator table provides specific entries for GLM, Kimi, DeepSeek, and Qwen. Preserve those identifiers when discussing the results. Kimi K3’s row does not establish a score for K2.8 Preview. Qwen 3.8 Max’s coding result does not evaluate Qwen Image 2.1. A release that creates images belongs in a different comparison, with criteria such as prompt adherence, editing consistency, and usable asset quality.

There is a concrete version trap in DeepSeek’s documentation. The older V4 Flash identifiers remain accepted, but their requests are now served by V4.1 Flash through the current Flash service [7]. Historical V4 Flash benchmark results therefore cannot automatically describe what an application calling that old name receives today. Record the underlying version as well as the request identifier.

For GLM-5.3, Z.ai documents text-only input, a one-million-token context window, and always-enabled reasoning with low, high, and max settings. Applications that disable thinking need a configuration change before migration [8]. A large context window will not make a text-only endpoint read a photograph. Convert or supply the appropriate evidence, or select a compatible multimodal model.

For any Chinese or Western provider, check the same operational questions: API compatibility, hosting location, retention terms, limits, commercial licensing where relevant, and the tools your application requires. Open weights, a hosted API, and a coding subscription are different deployment arrangements. Verify the arrangement for the particular release rather than assuming it from the company’s nationality.

For Saudi readers, Arabic deserves a place in the evaluation itself. Use Modern Standard Arabic, the register your audience uses, and mixed Arabic-English material. Include names, dates, numerical expressions, technical vocabulary, and right-to-left presentation. An English coding score cannot establish Arabic editorial quality.

A practical shortlist by workload

The following is editorial guidance for choosing what to test, not a declaration of measured winners. Keep the candidate set small enough to investigate failures properly.

Swipe or scroll sideways to see every column.

Work you need to finishUseful starting pointEvidence that should decide
Bounded repository changesAvailable coding models, including Sol and Sonnet; add a relevant Chinese entryPassing behavior, sensible scope, review burden, and cost per accepted patch.
Source-based reports or document draftsAccessible candidates with the required document toolsCorrect citations, preserved qualifications, missing-information handling, and usable output.
Multi-step office workflowsAgents compatible with the actual applicationsFull completion, saved state, permissions, and recovery from interruptions.
High-volume structured extractionCandidates that support your required schemaField accuracy, invalid-output rate, cost, and slowest-run latency.
Arabic or bilingual publishingModels available in your intended workflowMeaning, idiom, terminology, factual fidelity, and layout reviewed by a fluent reader.
Following the frontierAdd Argon to the watchlist until access is confirmedMatched evaluation details and tests once the intended endpoint is available.

A model can be an excellent choice for a specific row without becoming the default for everything. Routing easy jobs to one system and difficult cases to another can be sensible, provided the routing rule itself is checked. Count the escalation cost too.

Run a small comparison you can trust

Start with around 30 representative tasks as a practical pilot, not a statistically conclusive study. Include straightforward work, difficult cases, and examples where information is absent or contradictory. Use material you are authorized to process. Keep a separate holdout set if you tune prompts during the pilot.

Write acceptance criteria before seeing the outputs. For a coding task, require the relevant behavior, working checks, and a patch within scope. For an article, require accurate factual claims, primary sources that support them, useful analysis, and no invented experience. For extraction, define exact field-level checks. A beautifully worded answer should not compensate for a wrong number.

Run the candidates with the same evidence and comparable tools, permissions, and budgets. A common agent tests model differences more directly; each provider’s native product tests the overall experience you would buy. Either is useful if you explain which comparison you performed. Repeat difficult tasks to expose variability, and preserve failures rather than selecting attractive examples.

Review the outputs without model labels when possible. Log acceptance, billable usage, elapsed time, retries, tool failures, and minutes of repair. Track serious errors separately; an average quality score can conceal the one failure that makes a workflow unusable.

Swipe or scroll sideways to see every column.

Item to recordWhy it belongs in the log
Model identifier, provider, date, and effortMakes later reruns interpretable.
Task and evidence versionPrevents comparing changed inputs.
Accepted result and reason for failureConnects performance to an observable standard.
Input, cached, and output billing fieldsSupports a real cost calculation.
Time to usable result and reviewer minutesCaptures the work after generation.
Retries, tool errors, and serious mistakesExposes instability hidden by averages.

Choose the least expensive setup that reliably meets your quality threshold and time requirements. Rerun the difficult slice after a substantive model or configuration change. Expand the sample when the result is close or the stakes warrant it.

Saud’s view: make the recommendation earn its precision

The exciting development is that more providers now deserve a serious place in the evaluation. The useful next step is to make recommendations narrower and better supported: this model, at this setting, for this workload, with these tools and this acceptance rule.

That is the standard I want this publication to use. Benchmark tables should show their dates and methods. Cost claims should count the work needed to reach an accepted result. Comparisons should help readers build a shortlist and understand what evidence could change it. As new releases arrive, the conclusion may change; the discipline behind it should remain recognizable.

Update note: This guide replaces the earlier short comparison with published benchmark results, pricing arithmetic, version caveats, and an evaluation plan. Source-check date: 3 October 2026. It remains a researched comparison and editorial analysis, with no independent Fieldnote testing claimed.

Sources & further reading