The experiment did not owe me a happy ending

The question I actually asked

“Why is there no improvement on B?”

That was my question during the development comparison on 12 September. B was the candidate approach being tested against a baseline. The point of the exercise was to find out whether the added machinery delivered enough benefit to justify itself.

The answer was not especially comforting. Some early tasks left very little room for improvement. Some changes helped both sides. The candidate's stronger execution records did not automatically make its answers better.

This is the part of building an AI system that does not fit neatly into a launch announcement. You can spend substantial effort making something more structured and discover that the structure has not improved the outcome you hoped to improve.

The conversation continued through broader testing, a diagnosis, a shared change and a final decision. The useful story is that sequence, including the disappointing parts. It would be much easier to write an enthusiastic paragraph about the checks that passed and leave the comparison out. It would also miss what the day taught us.

The first broader result went the wrong way

In the broader development run, the baseline completed twenty-four of twenty-eight executions successfully. The candidate completed twenty-two. The candidate also used roughly three times as many input tokens.

Those numbers described a small authored development set. They were not a verdict on every possible UBIKITOS implementation, nor a general ranking of models. But they were enough to challenge the particular performance story being explored.

There was a narrower positive result: the candidate's custody records checked out. In plain language, the system had a consistent account of its executions. That matters when trying to understand what happened. It does not turn a lower task-success result into a higher one.

The conversation made that distinction explicit. The engineering around recording and controlling execution had value to investigate, but better records were not evidence that the candidate solved more tasks.

For a founder, that is an awkward but necessary separation. The parts that work deserve to be understood without being recruited to defend a claim they did not test.

Sometimes the work is right and the finish is wrong

The failure diagnosis found a particularly instructive pattern. Some executions had reached the correct state, then used another call checking it and exhausted their allowance. Other failures concerned how the final response was presented.

That is a useful reminder that an AI workflow is more than the quality of its intermediate reasoning. It needs to complete the task within the available resources and return a result in the expected form.

A hypothetical assistant might make the requested update correctly, then fail to provide the required confirmation. From one perspective the underlying work happened. From another, the workflow did not satisfy its completion contract. An evaluation needs to be clear about which outcome it measures.

The diagnosis suggested a shared limitation: the prompt did not clearly show the remaining tool and model-call budget. The next test gave both sides that information while keeping the limits fixed.

That was a change grounded in an observed problem. It was not permission to erase the earlier scores or quietly redefine what success meant after seeing the result.

A useful improvement belonged to both sides

On the small follow-up, showing the remaining budget helped both the baseline and the candidate. Each moved from one successful execution out of four to four out of four in those cases.

That was useful. It was also shared.

The distinction matters because a development process can accidentally attribute every improvement to the preferred design. If both approaches receive a clearer instruction and both improve, the instruction may be valuable without establishing a unique advantage for the candidate.

This is where fair comparison becomes a practical discipline rather than a slogan. The baseline needs the improvements it is entitled to receive. Otherwise the experiment risks comparing careful treatment on one side with avoidable disadvantages on the other.

The small follow-up was not large enough to justify a broad conclusion. It gave us a reason to test the shared budget information across a wider development set. The historical scores stayed intact, and the untouched evaluation remained closed.

A useful fix had been found. The larger question was still waiting for an answer.

The stopping rule needed to survive the score

Before the next screen, the conversation set practical requirements. The candidate needed a meaningful improvement in success, no regression across task families, acceptable token overhead and consistent custody checks.

The exact thresholds were recorded before the result. That matters because a stopping rule is only useful if it remains a stopping rule when the outcome is inconvenient.

The completed screen contained one hundred and twelve executions across the two sides. Both achieved forty-eight successes out of fifty-six. The candidate used about 2.69 times the input tokens of the baseline. Its recovery family scored six out of eight, compared with seven out of eight for the baseline. Its custody checks were consistent across all fifty-six candidate runs.

The practical continuation screen failed. The candidate had not delivered the required success improvement or resource profile.

These were development findings with a limited, authored task set. They did not settle every scientific question. They did settle the immediate decision that had been defined for this candidate: stop tuning it for the claimed performance benefit.

A result can narrow the project without erasing it

My reactions in the chat were not a sequence of polished product-management statements. I challenged the lack of improvement and kept asking what came next.

The answer eventually became more useful because it became narrower. Preserve the results. Close this candidate's performance-lift work. Investigate whether the governance and auditability capabilities solve an operational problem beyond a simpler control.

That last question still needed external evidence. When asked for a customer example, I said I had no clients. It would be wrong to turn the resulting workflow proposal into a customer success story. It was a direction for discovery, not proof of demand.

This is a difficult transition in a technical project. A disappointing performance comparison does not automatically identify the right business. It removes one unsupported reason for continuing and leaves other questions to be tested on their own terms.

A clearer record of execution might matter in a consequential workflow. Whether someone needs that capability, and whether the cost is worthwhile, remained open.

What was worth preserving

The completed work included the result reports and an evidence-preservation inventory. The decision was candidate-specific. No holdout was opened, and no general claim that UBIKITOS improved model performance was justified by this screen.

A separate experiment on 13 September would later raise its own evidence-retrieval question. Its result cannot be inferred from this one. The development screen described here has its own recorded conditions and outcome.

Looking back, the useful achievement on 12 September was a decision supported by an uncomfortable answer. The comparison did not deliver the hoped-for advantage. It did reveal specific weaknesses, identify a shared improvement and establish a reason to stop this line of tuning.

The next step was to look for a concrete workflow problem and assess whether the remaining capabilities mattered there. The experiment had done its job by making that decision harder to evade.

It did not owe me a happy ending. It owed me a result I could use.