Scott Feinberg
technologyaisoftware-engineeringagents

From Model Intelligence to Harness Intelligence

There's a marketed shift happening from model intelligence to harness intelligence when it comes to day-to-day utility. The scaffolding around the model — the agent loop, the tool integration, the sandbox, the context management, the interface you actually touch — has become the thing that determines whether you get useful work out of the session.

I can take a merely good model, put it in a great harness, and get excellent output. I can take a genuinely excellent model, put it in a bad harness, and get something I'd rather not have paid for. I do both of these every week, and the harness wins nearly every time.

That has a consequence for the "we need to slow down frontier model development" conversation. There's so much alpha opportunity in moving harnesses forward that the marginal frontier model matters far less than it used to.

The numbers on this are pretty stark

OpenAI's own ARC-AGI-3 results sheet for GPT-6 Astra, submitted September 2, lists two scores for the same model: 62.71% with the standard harness and 99.95% with the provider-adapter harness OpenAI submitted alongside it. Same weights, same games, 37 points apart. What the adapter does is preserve the model's internal reasoning state across requests, and on the same 167 games it ran 3.66× faster on 49% fewer tokens.

It's not a one-off:

  • On July 29, OpenAI published that GPT-5.6 Sol went from 13.3% to 38.3% on the ARC-AGI-3 public set with no change to the weights. Two API settings did it.
  • ARC Prize verified Claude Opus 5 at 30.16% on the public set on July 24. On August 21, NVIDIA reported the same model at 100.00 on the same set. The weights did not change.
  • A research harness called Schema reached about 99% on the public set with an Opus 4.8 and Fable 5 pairing, against 42.83% for the same pairing scored on its raw output.

ARC-AGI-3 public set scores for three models under a standard harness versus a tuned or provider-specific harness, with no change to model weights

Worth flagging that the 100s are self-reported and haven't been verified on the private set. But the direction is not in dispute.

Coding benchmarks say the same thing with less drama. On SWE-bench Pro, Claude Opus 4.5 reaches 45.9% under a standardized scaffold and 55.4% under Claude Code, and adding a single search subagent to otherwise identical infrastructure flips the ordering between MiniMax 2.5 and Claude Opus 4.6. The Holistic Agent Leaderboard reports single-model swings of up to nearly 48 percentage points across scaffolds, which dwarf the two-to-four point shifts that papers report as meaningful model advances.

So when the harness moves the number by 25 to 70 points and the model generation moves it by two, I know where I want to spend my time.

Why this makes the safety conversation weird

The pacing push is real and it's broad. Over 1,200 employees across the leading AI companies have signed a petition called "Pacing the Frontier" asking Washington to back an international throttling framework, and the signatories openly acknowledge that no individual lab can slow down alone under competitive pressure. Altman has said OpenAI welcomes safety requirements for frontier labs, and Amodei has put out a three-step plan to temper the pace without giving up commercial advantage.

I'm not going to argue about whether any of that is sincere. The structural problem holds either way.

Current responsible-scaling policies gate deployment decisions on direct-API benchmark scores, and those scores are format-contingent measurements that shift substantially with administration format, where structure-destroying scaffolds change the effective format without the evaluator knowing. Which means the safety regime is calibrated on the naked model while the capability people actually deploy lives in the wrapper. You can pace pretraining to a crawl and the deployed capability frontier keeps moving, in a layer nobody is holding hearings about.

That's the part I'd call safety theatre. Not that nothing is improving — Astra went from 78.5% to 100% on ExploitBench and cut hallucination from 12.2% to 4.2%, so that argument doesn't hold. It's that we've all agreed to regulate a number the wrapper decides. Meanwhile executives and policymakers are increasingly reading the safety rhetoric as a cover for regulatory capture or marketing, and the White House dismissed the warnings and doubled down on beating China.

My stack right now

This is a mixture of life stuff, coding, and running businesses.

Tool Spend What it's for
Claude $100/mo Pro (down from $200) UX design
ChatGPT / Codex $200/mo Max Coding workhorse and reviewer
Google / Antigravity $20/mo, free with my cell plan Image generation
Muse Spark / Muse Code cancelled, was $50/mo —
Muse for Humans free Lightweight agent errands
OpenCode Go / Zen $10/mo plus usage Best harness I use

My AI stack plotted by model quality against harness quality

Claude — $100/mo Pro, downgraded from $200

Highlights:

  • Design with Claude Design, which is a generally great tool, as is the way they handle artifacts
  • Systems planning and UX research with Fable. I've genuinely found Fable rewarding for interactive tasks and for UX reviews
  • Cowork for personal research and work
  • Remote control of Claude Code
  • Portable MCPs. Claude Code gets all your MCPs from your web experience, which is pretty convenient

Lowlights:

  • Claude Code. I can no longer use it as my primary coding workhorse or even my reviewer. It's too verbose and the code isn't high quality
  • Claude Memory generally works against me, not with me. I haven't found it genuinely useful yet

I'm keeping Claude mainly just as my "UX Designer." The things I kept are the ones where the harness is the product. The one I dropped is the one where I was paying for the model.

ChatGPT / Codex — $200/mo Max

Highlights:

  • GPT 5.6 Sol is my code reviewer and coding workhorse. This is where I'm spending most of my time coding
  • Codex is generally a great harness. For interactive coding it's a fair bit slower than GLM and Muse, but it's generally right the first time

Lowlights:

  • Codex Remote feature is unusable
  • ChatGPT "Work" isn't as well implemented as Cowork, especially on web
  • Crappy MCP integration

Google / Gemini / Antigravity — $20/mo Pro plan I got free with my cell phone

Highlights:

  • It is free
  • Did I mention it's free?
  • Nano Banana is awesome, and the best image generation I've done is via Gemini

Lowlights:

  • Using it

Antigravity is a really bad harness. The model itself, Gemini Flash 3.8, is genuinely good — I've used it with OpenCode and within my own projects and it performs. But the harness is effectively worthless for anything of value. Same weights, and it may as well be a different product.

Muse Spark / Muse Code — cancelled, was $50/mo

Highlights:

  • The Muse model is genuinely great. Within OpenCode it actually works really well. At or above GLM 5.3 Flash, much faster and about as good as GPT Sol

Lowlights:

  • The Muse Code harness is genuinely useless. The sandbox is so tightly wound that I'm constantly having to disable it to do anything useful

This one is really sad, because I was really hoping that Muse would become my coding workhorse. It was that good via OpenCode. Given the cost versus GLM, though, it doesn't make sense to leverage.

Muse for Humans — free

Highlights:

  • Free
  • The dedicated MicroVM architecture is pretty interesting, and gives you a lightweight, secure computer in the cloud
  • The ability to give it credentials to log into websites is pretty sweet

Lowlights:

  • It's a little lazy. It's not a very thorough model, so I ask for things like "go read my email and pull everything out about this one company" and it only pulls the first 50 results

For individuals who want an OpenClaw-like experience without paying for it or setting anything up, Muse is genuinely really useful.

OpenCode Go / Zen — $10/mo plus usage

Highlights:

  • Genuinely the best harness I use. Best interface, great design, including cost tracking front and center
  • It sips tokens compared to every other harness
  • Totally open. Bring your own tokens, bring your own providers, whatever you want

Lowlights:

  • No remote control feature
  • No portable MCP feature

That's the whole gap list, and it's a $10 tool beating the $100 and $200 tools on the dimension I care most about.

AI models within products

I'm directly leveraging GLM Flash 5.3 and Gemini 3.8 Flash within actual products, the first via OpenRouter and the latter via GCP directly. I'm not seeing any real difference between them in functionality for agentic use. GLM Flash 5.3 is just cheaper, while Gemini is easy for an app running on Cloud Run.

I'm also making use of Moonshine for running voice models cross-platform, on device. It works really well, especially when for safety purposes you can't have things going off-device.

ElevenLabs was a major unlock too, but more for test data. I needed a wakeword acoustic classifier for the Moonshine build but didn't have a bunch of audio. Six dollars later, I had 13k samples and a 98% accurate acoustic classifier (and a few days of experimentation).

What I'm looking at

For cost reasons, I'm considering a z.ai subscription to double down on GLM Flash for my workhorse coding via OpenCode. That would replace the Muse Spark and Antigravity usage for coding. The GLM Coding Plan starts at $18/month for Lite, $80 for Pro and $168 for Max, and it runs inside OpenCode, Claude Code, Cline, Cursor and 20-odd other tools. Flash lists at $0.15 per million input and $0.50 output, roughly a tenth of what the GLM-5.3 flagship costs at $1.40 and $4.40, now that the launch promo has ended.

What I'm doing there isn't shopping for a better model. I'm picking the cheapest adequate model and pointing it at the best harness I own.

The industry numbers back that up more than I expected. On the Artificial Analysis Coding Agent Index, GPT-6 Astra scored 67.0, roughly level with the Fable models, while matching Fable 5's score at less than half the cost per task thanks to token efficiency. On the broader Intelligence Index, Astra came in at 61.2 against 60.9 for Sol. That's three tenths of a point between one flagship generation and the next, against 37 points from swapping a wrapper.

Sources

← Back to all posts

Get notified of new posts

No spam. Just new articles when they come out.