Recursive Self-Improving (RSI) approaches like Poetiq's Metasystem render static benchmarks obsolete (Benchmarks Are Dead). To quantitatively measure progress, we use Augur. Augur dynamically maps exactly where a model or agent's knowledge ends and its reasoning breaks — turning those boundaries into an un-memorizable benchmark and the precise training data required to improve it.

• Enterprises deploying LLMs. Which model should run your underwriting, your trial documentation, your customer operations? Public leaderboards cannot answer these questions for your specific domain. Augur generates a benchmark for your domain. It shows where each model will succeed and where it will fail. It shows where an open-source model matches a frontier model and where it falls short. And because the benchmark is dynamically regenerated, it keeps identifying the best models as new models ship.
• Model builders and fine-tuners. The highest-value training data is the data your model gets wrong. Typical bulk-labeling vendors cannot find it, because most of what they label your model already knows. Augur finds the places where your model fails, whether due to missing knowledge or reasoning errors. These two types of data are a natural fit for pre-training and fine-tuning in the first case, and RL post-training in the second. And, if you are already fine-tuning, it will tell you what your last training run silently broke — and hand you the data to restore it.
• Us. Augur closes the loop our RSI work opened: map a model's boundary, generate the exact data to move the boundary, train with that data, and re-measure — all autonomously. We use the maps to decide which model to trust inside every harness our Metasystem builds. Each run compounds a proprietary asset: an atlas of where every major model breaks, refreshed as fast as models ship — and the exact domain-specific data to lift any model's performance.

Because Poetiq's Metasystem works across models, it is continuously and implicitly mapping the capability space of every model it touches. Augur is that capability made explicit and packaged as a product. The map it produces is a per-model, domain-specific chart of where knowledge ends and where reasoning breaks. If you are fine-tuning an open-source model, it tells you exactly the data needed to close the gap.

Simply asking a question to two LLMs and measuring their differences does not reveal their real inconsistencies. Relying on simple factual questions overlooks how LLMs are used in practice. Augur instead measures the whole process of answering a real question. Specifically, it separates three dimensions that ordinary benchmarks blur:

• Knowledge: Does the model know the fact, from memory, with no tools?
• Tools: When models disagree, can a live search resolve it, or is the fact genuinely contested?
• Reasoning: When a question requires several facts combined into one answer, does the model compose them correctly?

The third dimension is the one that matters in practice. It is what separates a model that scores well on model cards from a model you can build agents and multi-step workflows on. Two models can agree on every individual fact and still diverge when those facts must be combined, which is precisely what agentic work needs. Augur finds these points automatically: it generates questions and sharpens them until the model under test breaks. How much sharpening that takes is itself a measurement of the model's robustness.

Since the probes are dynamically generated rather than drawn from a fixed set, the result is a benchmark that cannot be memorized, because it adapts to the model under test, and it measures far more than whether a fact can be recalled.

Here is a simplified schematic of a probe Augur creates:

Q: Consider the chief executives of companies A, B, C, D & E at the start of YEAR-Y. How many were still in the role N years later?

A model can answer each underlying question correctly, who each CEO was and when each departed, and still miscount the composite, returning 3 where the verified answer is 2. Every fact is in place, but the composition fails.

What happened? Not a knowledge failure, but a reasoning failure. This form of error, where every atomic fact is answered correctly, but the composed answer is wrong, is exactly the task modern RL post-training needs: a prompt with a verifiable, exact-match reward that the current policy sometimes solves.

Diving a bit deeper, Reinforcement Learning (RL) fundamentally works by trial and error. If a model doesn't know a fact, it can't stumble upon it by accident, meaning the reward is always zero, and the model learns nothing. But when a model holds the atomic facts and simply fails to compose them, it becomes an ideal candidate for RL.

Each Augur outcome provides useful information: knowledge failures become supervised fine-tuning data, reasoning failures become RL training data, and contested questions are dropped to avoid polluting either dataset.

Augur yields raw data for model inspection and improvement. Each point that Augur returns serves three roles:

1. A test question. Hard, discriminating, and almost impossible to have leaked into training — it was just generated.
2. A training example. It sits at the model's capability boundary. This is the single most efficient kind of training data for moving the boundary, as it can help with either fine-tuning or RL post-training.
3. A guide to general capability boundaries. Collectively, these insights reveal the specific categories of errors a model will encounter within your domain, moving beyond the mere identification of isolated failures.

The training-example role deserves particular attention from anyone fine-tuning. More teams are fine-tuning open models to cut cost and raise throughput. Fine-tuning and post-training that injects new knowledge or capability tends to degrade other capabilities (catastrophic forgetting), in domains both near and far from the tuning target. Augur finds the points where knowledge was lost and surfaces the exact training data to recover it.

The data Augur provides combines well with parameter-efficient fine-tuning approaches like Low-Rank Adaptation (LoRA), for both supervised fine-tuning and RL. Augur plus LoRA creates a highly modular, cost-effective architecture: a single base model that dynamically swaps lightweight adapters tailored to the exact capability maps Augur generated for each of your domains.

When we test our Augur benchmarks, the same pattern repeats: on easy questions many frontier models appear similar; but push the models to answer real questions and the differences emerge. As an example, we looked at how DeepSeek-V4 fared against multiple closed models from OpenAI. The findings below come from examinations across music, law, software APIs, video games, and hardware specifications.

• Knowledge gaps appear, but not all of them are real. As we show in Figure 3, model differences caused by a genuine knowledge gap varied widely by domain: 68% of hardware's single-fact disagreements, 52% in video games, 51% in software, 34% in law, and 30% in music. The remainder were less distinct: either the fact itself is contested across sources, or the model's answer is unstable (if you resample it, it flips). This is uncertainty, not necessarily a conflicting stored fact. The distinction is important in how you use (and fine-tune) the model.

• Tools resolve some of the differences, though not all. If we are willing to bear the expense of LLM tool calls, such as web search, single-fact disagreements in the technical fields can be dissipated. However, even with tools, 27% of the disputed facts in music and 9% in law and software remained.
• On real questions, reasoning divergence dominates. When questions required understanding multiple facts, disagreement was more frequent, and it usually was not about missing information. Of all the model disagreements we surfaced in a domain, the share where the models agreed on every underlying fact yet still reached different answers was the largest single cause: 58% of law's disagreements, 48% in video games, 43% in music, 40% in software, 32% in hardware. See Figure 4.

• The depth of similarity of two models is measurable, for any domain. Some pairs of models rarely showed disagreement; others came apart as soon as two facts had to be compounded — and the extent varied heavily by topic. The share of Augur questions that never produced a disagreement at all ranged from about a fifth (hardware) to two-fifths (music). So when you decide which model to use, two models shouldn't be considered “interchangeable.” “Interchangeable” is not a property of two models; it is a property of two models on a specific task. If you change the task, re-measure. Once it is measured, you can route between models on price, latency, or any metric you choose.

K3 has been collecting accolades across the industry; we highlight a few insights from the Kimi K3 case study we did. On factual knowledge, ask it a simple question and its recall matches the major frontier models across most of the domains we tested, though not all (climate change being a notable exception). In contrast, when examining reasoning, differences emerge across multiple domains.

Under compositional pressure, where several facts must be reasoned into a single answer, K3 diverges, and when it does, more than 75% of the time it knows all the facts. It has the pieces, but assembles them incorrectly. That is the finding that matters for anyone deploying it for real use, and it is overlooked by any test that only checks recall.

How does K3 compare to the previous Kimi generation? It is a genuine improvement over K2.6, but a specific kind of improvement. It is measurably better on some knowledge (civics/law for example) and roughly unchanged on pure puzzles and on obfuscated C and Python, where K2.6 was already near the ceiling. Genuine knowledge gaps fell by about three-quarters from K2.6 to K3 in the domains tested. It did not, however, provide the improved multi-fact reasoning you would expect from the stated public benchmark gains.

So how would you make K3 better? Let's look at a common use case.

We pointed Augur at a typical financial task: answering the type of questions an analyst will ask when looking at the corporate records of the largest technology companies. Augur probed the models about their executives and leadership histories, their acquisitions and abandoned deals, and the direction of their reported results.

On executive careers, leadership transitions and strategic pivots, K3 holds the facts equivalent to frontier models, but diverges almost entirely on composing them: counting transitions across companies, summing tenures, and tracing a multi-step chain to yield a single answer. Acquisitions look much the same, with only a small residue of genuine gaps on the specifics of individual deals. In neither case is missing knowledge the culprit; K3's command of the corporate record is broad and current. What increasingly separates K3 from the frontier is not what it knows, but how it composes what it already knows.

Augur, with no human in the loop, produced a targeted dataset: the specific facts K3 is uncertain about or likely wrong on, and the specific reasoning chains where K3 holds every fact and still lands on inconsistent results — e.g. five-right-facts-one-wrong-answer examples, ready for RL. That dataset is what you would post-train K3 on to close the gaps, and it is also the same dataset you would use afterward to check that it worked.

Just as important is what Augur withholds. Some apparent failures are contested ground: the sources themselves disagree, and there is no single right answer. (Ask which of two close quarters had the lower earnings, and the “answer” depends on which source you trust.) Augur flags these and holds them back, so the data it emits reflects real errors rather than artifacts of bad questions or bad data. A pipeline that cannot make that distinction will happily train a model on someone's typo.

Augur operates autonomously, pinpointing exactly where models fail by classifying breaks into missing facts, contested sources, or reasoning failures. It discards contested items while transforming knowledge gaps into fine-tuning data and reasoning errors into RL tasks.

For external developers, Augur serves as an automatic real-time benchmark generator for any vertical, and a high-signal source for identifying the specific data needed to label and improve model capabilities.

Within Poetiq, Augur closes our Recursive Self-Improving (RSI) loops. It generates the high-value data required to advance our Metasystem in any domain, while the Metasystem in turn optimizes Augur's own probing efficiency.