Who Pays When the Model Is Wrong?

S Ravi Kumar November 15, 2026 11 min read 4 views (4 unique)

When an AI model is wrong, the customer pays twice, and nobody gives the time back. So I gave one brief to nine models in the same framework and counted: the same job took twenty times the tokens from one model to another.

Back in July I was fed up with going through the build cycle again and again without any output.

The cycle ran inside TechieFlow, the framework that runs my AI coding agents: it gives each agent its task, its rules and its checks. For weeks I had been inside one loop. The agent builds. Something is wrong. I point at it. The agent fixes it. Something else is wrong. A new top model had just been released, so I gave it the whole framework and asked one thing: how do I make this fix cycle shorter?

Somewhere in that conversation I lost my patience and asked it who was responsible for the time I had already lost. It answered honestly. It could not give me my time back, and it could not give me my money back.

It was right, and the answer stayed with me. I pay a hundred dollars a month for that model. When it is wrong I pay a second time: in tokens for the repair, and in my own evenings for finding the mistake. The model does not pay. The vendor does not pay. When my hosting provider is down for an hour, I get a credit. When my model is wrong for an hour, I get an apology.

So the question in the title has a short answer. You pay. I do not think that changes soon. This post is about what I did once I accepted it.

What I could not answer

By then I had been building with coding agents for about a year. In September I wrote about that year: twenty one repositories, one shipped. I had strong opinions about which model was good. I had proof for none of them.

Is the new model better than the last one on my work, or does it only feel better in its first week? Is the expensive one worth it for this job, or only for the hard ones? When a build goes wrong, is it the model or my instructions? When a vendor publishes a benchmark chart, what does it say about my code?

Every answer I had was an impression. And the model's review found the same weakness in the framework itself: too much of it was prose. Rules written as sentences, which an agent can read, agree with and not follow. That same month an agent marked its own work as verified. Nothing had checked it.

I was paying for mistakes, and I could not even say how many.

What I built instead of trusting

Two changes, and everything else follows from them.

Rules became scripts. A rule that matters is no longer a sentence the agent is asked to respect. It is a script that runs and says pass or fail. TechieFlow has fifty of them today. The agent that builds no longer decides whether its own work passed.

Everything is recorded. Every run writes a row: which command, which model, how long, how many tokens. Every mistake writes a row: what was wrong, who found it, what it cost to find and to fix. They are plain files inside each project. TfLens, the dashboard I wrote about in October, reads them.

This is what those files hold as I write this, across five public projects of mine (TechieFlow itself, TfLens, TechieBlog, Chatur and TrBlazeUI), since August:

Runs recorded 846
Tokens the models wrote 74.8 million
Different models used 12
Mistakes recorded 1,493
Found by a check written as code 819
Found by me 500
Found by a second agent that reviews the first one's work 119
Found in use, after release 31

I want to be exact about what this did and did not do. The fix cycle has not gone away. 500 times, I was still the one who found the mistake. What changed is that I can see the cycle now: per project, per model, per kind of work. And I can test the thing I could never test before, which is the framework itself. When I change a rule, the runs that follow tell me whether it helped.

It also made one experiment possible.

One brief, nine models

I ran it live, on a Sunday stream in October.

One brief, for a diary app. One framework. One job: write the requirements, the architecture and the screen mockups, and stop. Nine models, each in its own folder, with no help from me. Afterwards a script counted the result from each folder's own record.

Model Where it ran Minutes on the job Tokens written Requirements Screens
Claude Opus 5.5 Claude Code 18 431,696 97 20
Claude Sonnet 5.5 Claude Code 8 127,185 93 20
Claude Haiku 5.5 Claude Code 10 797,077 69 20
Kimi K2.7 Code OpenCode 52 141,515 81 15
DeepSeek V4 Pro OpenCode 11 38,452 94 20
GLM-5.3 OpenCode 27 192,546 61 22
GPT 6 Luna OpenCode 165 146,239 76 20
Qwen3.8 Max OpenCode 63 131,061 99 19
Qwen 3.8 27B, on my own laptop OpenCode, local 15.5 hours of sessions 79,042 77 14

"Requirements" is the count of numbered requirements each model wrote, and "screens" the count of mockup files. The model on my laptop has no timed run, only the hours its sessions stayed open.

Four things in that table that no benchmark had told me.

The same job took twenty times the writing, depending on the model. 38,452 tokens at one end, 797,077 at the other. Same brief, same framework, same instructions. If you pay by the token, that is a twenty-fold difference in the bill for one piece of work.

The cheapest model per token wrote the most and delivered the least. Haiku is the small, cheap one in its family. It wrote 797,077 tokens and produced 69 requirements. Sonnet, one step up, wrote 127,185 and produced 93. The price per token is not the price of the job, and only the price of the job is on your bill.

The top model was not the obvious choice either. Opus wrote more than three times what Sonnet wrote and took more than twice as long, for four more requirements. That may be worth it. I will come to why I cannot tell you from this table.

The model on my own laptop was inside the range. A 27-billion-parameter open model, running on a MacBook, produced 77 requirements. The hosted models produced between 61 and 99. What it could not do was finish in an evening: it took fifteen and a half hours of sessions across a week.

What the table does not say

It does not say which model was good.

The table counts cost and volume: minutes, tokens, requirements, screens. Ninety-nine requirements are not better than sixty-one if thirty of them are padding. I left quality out on purpose, and I want to say why.

Cost and volume can be measured. Quality, for the most part, cannot, because it belongs to the person judging. What I call a good requirements document, another architect will call bloated, and a third will call thin. Each of us brings our own habits to the judgement. I do not believe there is a standard that settles it, and I am careful with anyone who publishes a single quality score for work they have never seen: mine.

One part of quality is not taste, though, and that part I do measure. Whether the thing that was built does what the requirement says is a yes or a no, and a script can answer it. That is what the 819 in the first table are: mistakes caught by a check, not by an opinion. My nine-model test stopped at documents, where there is nothing to run. The same test carried through to working code would show how far each model drifts from the requirement it was given, and that is a number. It is how I will compare the next new model with the one I work with every day.

So my position is this. Measure everything that can be measured. Say plainly that the rest is your own judgement. Do not let a chart pretend to be either.

What I decide from my own numbers

The model follows the job. Thinking and planning go to the top model. Building and fixing go to the one below it. Running checks again goes to the cheaper one. Every run records which model did it, so the choice itself can be checked later.

A new model does not get my work because of its launch chart. I will not run nine models again. A new model goes up against the one I already work with every day: the same task, the same framework, one evening. Then I look at the two rows and keep whichever earned it.

I know what I would do with a stronger machine. If I had hardware that ran that open model at a usable speed, I would not be paying for a coding subscription at all. I did not get that from a review. I got it from watching it produce 77 requirements on my own laptop, slowly.

When I buy a coding plan, I buy the flat one with a high ceiling, not the one with a small monthly allowance. The table shows why: the same job can cost twenty times the tokens depending on the model, so nobody can tell you in advance what an allowance will buy. I learned it the paid way. I had a second, cheaper plan, OpenCode Go Plus, for the open models in that table. One afternoon of real work on two projects used the month's allowance. I did not continue it. A plan you have to ration is a plan you stop building on.

What your own system needs, at the least

You do not need my framework. TechieFlow is public and you are welcome to it, but any set-up that does these five things will do.

  1. One fixed task from your own work. Not a puzzle from the internet. A real brief you understand well enough to judge. Keep it and do not change it.
  2. The same framework for every model. Same instructions, same tools, same checks. Otherwise you are comparing two set-ups and calling it two models.
  3. A record written by the machine, not from memory. Time, tokens and what was produced, for every run, with the model's name on every row. I did not do that from the start, and it shows: only about a third of my recorded mistakes can be laid at a named model.
  4. Checks as code for whatever can be checked, and a note of who found each mistake. The day most of your mistakes are found by you, you are the test suite.
  5. When a new model arrives, put it against the one you work with every day. Not a grand comparison each time. Two models, the same task, the same framework, and you keep whichever earns it.

None of it is difficult. It is only work that nobody else will do for you, because nobody else is paying.

Where this leaves me

The model was right that evening. It cannot give me my time back, and nobody else can either. I still pay when it is wrong.

But I no longer buy on trust or on benchmark scores. Every model I use has earned its place on my own work, in my own numbers, and the next one will have to do the same. Nobody is accountable to you for a model's mistakes. The nearest thing to accountability you will get is a record of your own, and the freedom to move the work when the record says so.

If you take one thing from this, do not start with a framework. Start with one evening. Pick a task you finished last month, one you know well enough to judge. Give it to the model you pay for and to one other, with the same instructions. Write down three numbers for each: how long it took, how many tokens it wrote, and how many things you had to fix afterwards. That is your first record. The next time someone shows you a benchmark, you will have something of your own to put beside it.

Rate this article
0.0 · 0 ratings One rating per email — no sign-in needed
SR
S Ravi Kumar Author of this post.

More Posts

Comments (0)

No comments yet

Be the first to share your thoughts on this post.

Leave a comment

No account needed — just your name and email.

Your email is never published — it is used only for confirmation and moderation.
Generated and checked by this site — no third-party service.
Comments appear after email confirmation and moderation.
An unhandled error has occurred. Reload ×