The mistakes
What I got wrong
Four things I published that turned out to be wrong, two bugs that had been quietly ruining measurements, one test I set myself and failed, and one correction that went too far and had to be walked back. Two of the corrections made my work look better and two made it look worse, which is roughly what you would expect if the checking is real. They are on their own page because there are a lot of them, not because I want them out of the way.
The big one
A setting that was 48 times too large
The circuit is asked to do two things at once: obey the battery's physical limits, and among the plans that do, find the cheapest. I set the penalty for breaking a rule about 48 times larger than the entire spread between the cheapest and most expensive legal plan. So the machine was being asked to find a legal plan and paid almost nothing for finding a cheap one.
Fixing that one number, with the circuit and the tuning untouched, improved how often it lands on the best plan by a factor of 440. The right value can be worked out from the problem in advance rather than tuned by hand, which is what makes it worth writing down.
The damage was bigger than one number. I had been showing a chart of how the quantum method scaled, and it was measured entirely through the bad setting. On that chart the advantage shrank as problems grew, and several points sat below random guessing, meaning the circuit was doing worse than nothing.
Re-running it moved the same measurements to 4 to 20 times better than chance at every size, with no decline as the problem grows, and turned finding the exact best plan at 14 qubits from something that happened twice in nine tries into something that happened eight times in nine.
One thing that chart is not claiming. Beating random guessing is a low bar here: the ordinary classical solver returns the exact best plan for every one of these problems in microseconds. This measures whether the quantum state gathers around good answers at all, which is a different and much smaller question than beating a classical computer.
I had been showing that chart as something I had learned about quantum computing. Most of it was something I had learned about my own settings.
The other end of the chart
Zeros, a ceiling, and a tool that never worked
The old chart had a second problem, at the opposite end from the weight. It read the answer by sampling 4,096 times, and anything rarer than one in 4,096 comes back as a flat zero. Nineteen of its thirty-six measurements were exactly zero. The declining trend I was describing was not the quantum method getting worse. It was my measurement hitting its own floor.
The fix cost one extra calculation: read the exact probability off the simulator instead of sampling it. Same runs, same seeds, same problem sizes. The numbers that had read zero turned out to span 4e-12 to 7e-02.
While checking the corrected numbers I looked at how much tuning effort the runs had actually used. The answer came back at exactly the maximum I had allowed. A number sitting precisely on its own ceiling is not a measurement of anything; it is the ceiling.
So the figures from that sweep are floors rather than final answers, and the pages that quote them say so.
Underneath that was worse. The tool computing the measurement had never once worked above the smallest problem size. It was building something that doubles in size with every qubit, so it ran the machine out of memory partway through. From outside, that looks like a long job crashing rather than a broken instrument, which is why it survived so long.
Rewriting that one calculation took it from failing at 14 qubits to finishing 22 in about five seconds. It now checks itself against the old method before the study spends anything.
And then I over-corrected
Finding an error is not the same as knowing what it broke
Finding that ceiling felt like it settled something, so I took the next step without checking it. One of my earlier conclusions rested on runs that had been sitting on the cap, and I wrote that the conclusion was therefore not supported. That was an overreach, and it took 120 more runs to find out.
The first half of what I had said was right. The budget really was binding: the original runs hit their limit on ten seeds out of ten. Lifting it twenty-five times really did help, by a margin clear enough to measure.
It just did not help nearly enough. The gap the result had to close was about three times larger than everything the cap had been hiding. So the conclusion survived, and the number it rested on moved by an amount that changed nothing.
The giveaway was that the extra budget went unused. Given twenty-five times the allowance, the optimizer spent 38% of it and stopped, which means it was no longer being cut off by the limit at all. Whatever holds it back now is something else. There is a cheap check hiding in that: if you raise a limit and the extra allowance goes unspent, the limit was not what was binding.
One run out of the 120 did clear the bar, and it came from the arm with the smallest budget, not the largest. If money had been the problem, the well-funded runs would have cleared and the starved one would not. It happened the other way round, which is what a lucky start looks like rather than a resource limit.
So there are two mistakes stacked here, and the second one is mine as much as the first. Finding the ceiling was right. Assuming it knocked down the conclusion built on top of it was not. A measurement stuck against its own limit tells you the number is imprecise. It does not tell you the number is pointing the wrong way, and the only way to find out which was to raise the limit and look.
A test I failed
The instance I named first
Before putting a circuit on real hardware I wrote down what would justify it: a specific version of the problem, named in advance, and a bar the simulation had to clear first. It did not clear. Twelve combinations of tuning strategy and setting, some with ten times the computing budget of the others, all landed just under: 0.075 against a required 0.078.
Several of those same strategies clear the bar on two other versions of the problem, one of them ten times out of ten. So it is not that the method fails. The version I happened to name first is just the hardest of the three.
What I wanted to do was point at one of the versions that passes and call that the answer. Naming it beforehand is what stopped me, so I did not run it. That question is still open, and it should be re-measured at the corrected setting, since the runs that failed were scored through both of the bugs above.
Retractions
Four claims I had to take back
Each of these was published before it was checked. The repository keeps every one where it was made, marked wrong, rather than deleting it.
- I blamed a difference between two circuits on the number of qubits. My own earlier data does not support that.
- I argued the machine could not have drifted between runs, using a comparison that was never valid. Drift is real, and worth 43% of that difference.
- I said the variation between runs was larger than the machine's own wobble within a run. The two are too close to tell apart with the data I have.
- My first error bars used a method that added noise the measurement does not contain, which made my own result look weaker than it was.
Fixing that last one made my result eight times stronger, which was an awkward thing to find. I could have left the worse number up and looked modest.
One more, and it is the one I mind most
A second test in the same plan asked how much the machine varies within a single job. I had written it as needing a spread “comparable to or larger than” the effect, without saying what comparable meant, and then picked a number for it after seeing the data. That is exactly the move that writing the plan down beforehand is supposed to prevent.
Fixing the wording would not have saved it either, because the test never had the power to answer the question: a spread estimated from a single pair of runs carries about 76% uncertainty, so it straddles any sensible threshold. Re-run later with ten repeats, it came back genuinely undecided rather than wrong.
Why this page exists
The record is the point
None of this had to be published. Most of it I found myself, after the original claim was already up, and nobody would have known if I had quietly edited the numbers. But a result you cannot check is not worth much, and the checking only means something if it can come back negative. Two of these corrections made my own work look better and two made it look worse, which is roughly what you would expect if the checks are real.