Behind the builder / 6 September 2026
The better score came with an asterisk worth reading
A fairer UBIKITOS development comparison showed a recovery signal, higher token use, and grounding problems. The promising score still needed a better next test.
The rerun was fairer
September sixth completed a repaired development comparison across twenty-one tasks and forty-two paired executions. The fairness problems addressed in the rerun mattered because a comparison is only useful if the two sides are being judged under appropriate conditions. The treatment scored 61.90 percent on the run’s success measure against 47.62 percent for the baseline, a descriptive difference of 14.29 percentage points. That was an interesting signal. It was not yet a confirmatory finding or a general claim about the whole platform.
The details made the result more useful than the headline alone. Recovery cases favoured the treatment, while the earlier reasoning advantage disappeared and evidence-grounded research became worse. The treatment also used materially more tokens and produced more unsupported claims. The next step was not to publish a victory over the baseline. It was to strengthen the promising bounded behaviour, repair the regressions, freeze the treatment, and only then consider the preregistered confirmatory evaluation. The untouched holdout remained closed.
One score can hide several stories
An aggregate score is helpful for seeing a broad pattern. It can also conceal where the pattern comes from. In this run, the strongest signal was concentrated in stateful workflows and recovery. The three recovery cases went from zero successes in the baseline to three in the treatment. That is worth investigating, and the sample is very small. It supports a specific development question about recovery; it does not establish that the treatment is better at every kind of task.
The research regression matters just as much. A system that improves one family while making more unsupported claims elsewhere presents a tradeoff, not a universal upgrade. The builder needs to understand the distribution of outcomes. Which tasks benefited? Which became worse? Did the improvement rely on more resources? These questions turn the result into a guide for engineering. Ignoring them would make the headline simpler while making the next development decision less informed. The asterisk is not an obstacle to the story. It is where the story becomes useful.
Recovery is a practical problem
Stateful work involves more than producing one answer. The system may need to preserve a decision, use a tool, interpret the result, and continue appropriately when something goes wrong. Recovery is therefore a meaningful place to look for value in UBIKITOS. A system that handles interruption more reliably could reduce a real supervision burden. That potential is more specific than saying the model has become generally smarter, and it connects naturally to the earlier work on persistence and controlled continuation.
A hypothetical workflow might need to resume after a failed information request without losing the task’s scope. Another might need to recognise that a previous action remains uncertain and avoid repeating it blindly. Those examples explain why recovery quality matters; they are not additional results from today’s evaluation. The actual run supplied a small development signal in that direction. The next experiment needed to test whether the signal remained under a fixed treatment and appropriate conditions, rather than expanding the claim faster than the evidence could support.
Fairness is not a decorative label
The rerun repaired fairness conditions, verified provider and model records, and kept the evaluator’s expected answers away from the executor. Those details matter because an apparent advantage can arise from unequal information or resources rather than the mechanism being studied. A fairer comparison narrows those alternative explanations. It does not automatically turn a development set into an independent confirmation. The role of the data still matters, including whether it has influenced the design being evaluated.
Development runs are useful for learning what to change. That is precisely why they are weaker as final proof of the changed design. The builder sees the failures and adapts. Over time, the system may become particularly well suited to those cases. A separate holdout provides a different test once the treatment is fixed. Keeping it closed preserves that role. Opening it early would produce more information now while making the later claim harder to interpret. The decision to wait was part of protecting the quality of the eventual answer.
More tokens belong in the comparison
The treatment used materially more tokens. That does not automatically invalidate its usefulness, but it belongs in the account. A better result may be worthwhile at a higher cost, depending on the task. It may also be less attractive if the extra resource use produces only a narrow or inconsistent gain. The founder needs the tradeoff, not just the success percentage. Performance without resource context can make an engineering improvement look more efficient than it is.
The unsupported claims are a separate concern. A system intended for consequential work should not hide a grounding regression behind a higher aggregate score. The next treatment needed to improve evidence use and reduce those claims while preserving the recovery behaviour that looked promising. That is a more demanding engineering target than simply maximising the visible total. It asks the system to become useful in the right way, including the quality of its explanations and the reliability of the statements a user might act on.
The next version needed to stop changing
Freezing the treatment before a confirmatory run was part of the planned sequence. A moving target makes the result difficult to interpret because different tasks may effectively be testing different systems. A fixed treatment gives the evaluation a stable subject. If the run then reveals a weakness, the next change can be made explicitly and evaluated separately. This is not an argument against iteration. It is a way to keep iteration from dissolving the meaning of the comparison.
For a founder, the discipline can be frustrating because there is almost always one more improvement that seems worth making. The development result provides direction: focus on governed stateful tool use and recovery, repair grounding, then freeze. That is a useful boundary around the next work. The project does not need to solve every possible weakness before asking a confirmatory question, but it does need to decide exactly what is being tested. Otherwise, the result becomes a record of an evolving experiment rather than a clear test of a defined system.
The promising part deserved a better test
September sixth ended with a stronger development signal and several reasons not to oversell it. The recovery result was encouraging within a tiny sample. The aggregate difference was descriptive. Other task families did not share the gain, resource use increased, and unsupported claims needed repair. No customer, market, generation-wide, or causal claim followed from the run. The next meaningful result required a frozen treatment and the appropriate confirmatory procedure.
This is the kind of day that makes a founder diary more interesting than a leaderboard screenshot. The score improved, but the details changed what the improvement meant. UBIKITOS had a more specific direction to pursue: dependable stateful work and recovery, with better grounding. That is a useful outcome even before a larger claim is available. The asterisk did not ruin the better score. It explained how to turn the score into a better next experiment, which was the job the development run was supposed to do.