Knowing Versus Guessing

1. Introduction

1.1 Two things that look identical

Pasting data into a chat window has quietly become one of the most common ways statistical questions get answered in organizations. It requires no software, no analyst, and no budget. A manager exports a spreadsheet, pastes it into an assistant, asks whether one thing is related to another, and acts on the reply.

What the manager cannot see is which of two operations produced that reply. The model may have added up every row and divided, in which case the answer is arithmetic and is right. Or it may have read a portion of the file, formed an estimate, and reported the estimate. Both arrive in the same sentence structure, with the same confidence, and often with the same careful caveats attached.

The practice has spread faster than the evidence about when it works. Vendors of analytical software, including this one, have generally asserted that language models fabricate numbers. This study finds that claim to be false. The people relying on the workflow have no published guidance on where it holds and where it fails.

1.2 Scope and objectives

This whitepaper reports an experimental study of statistical accuracy in the no-tools chat condition: raw data pasted into a single conversational turn, with no code execution, no plugins, and no external retrieval available to the model. The analysis targets analysts, managers, and technical leaders who either use this workflow or are deciding whether to permit it.

Specific objectives include:

• Establishing whether models compute from provided data or retrieve remembered values
• Locating the data size at which accuracy degrades, and characterizing the shape of that degradation
• Determining whether the degradation depends on the type of question asked
• Assessing whether the resulting errors are detectable by the reader without independent computation
• Testing whether models select appropriate statistical methods when none is specified

1.3 Why this matters now

Two trends make the question urgent. Assistants route requests across model tiers by cost and load, so the user typically cannot know which model answered, and this study finds the accuracy gap between tiers larger than any other factor measured. At the same time, organizational data is growing, pushing more real questions past the size where the workflow was measured to be reliable.

2. Methodology

2.1 Experimental design

We reproduced the consumer workflow exactly. A raw CSV file was pasted into a single chat turn together with a statistical question in plain language. All tools were disallowed, no code execution was available, and the working directory carried no additional context, so every answer had to be produced from the pasted text alone.

The design spanned five public datasets, three models across two price tiers, and five question shapes: correlation, subgroup mean, two-sample test, regression slope, and highest-mean category. Data sizes ranged from 50 to 2,000 rows. In total, 240 attempts produced 179 responses and 105 scorable numeric answers.

The three models were Claude Opus 5 and Claude Sonnet 5 at the frontier tier, and Claude Haiku 4.5 as the cheap tier. They are named because the results differ sharply between the tiers, and because a reader deciding whether these figures apply to their own situation cannot make that judgement from a description of price alone.

2.2 Controlling for memorization

All five datasets are long-standing public teaching datasets, and any of them may appear in model training data. This is the central threat to validity in a study of this kind, and the design addresses it directly.

Models never received the published datasets. Each received a seeded random subsample whose true statistics differ measurably from any figure that could be looked up. On the tipping dataset, for example, the correlation between bill and gratuity is 0.772 at 50 rows and 0.640 at 200 rows, against approximately 0.676 for the published full file. A model answering from memory would therefore produce a characteristic signature: errors pointing toward the published value.

Ground truth was computed independently on the identical subsample, with three statistics re-derived by hand-written formulas and matched to twelve decimal places. The exact subsamples are published so this can be verified.

2.3 Supplementary arms

Three further arms were added as the analysis progressed. The first regenerated the entire study on data created specifically for this paper, which no model has encountered. The second repeated individual questions five times each in independent sessions, to distinguish a stable error from a noisy one. The third asked sixty questions with no statistical method named, scored against a standard fixed and cryptographically hashed before any response was read.

3. Results: the knowing is real

3.1 Direct evidence that it computes

The first question is whether any of this constitutes arithmetic at all. A model that had memorized a well-known dataset could return a correct correlation without computing anything, and the exercise would be measuring recall.

It is not recall, and the evidence is direct rather than inferential. Several responses displayed their intermediate working, which allows the claim to be checked. On one 50-row regression, the model reported a sum of squared deviations of 33,915,193. The true value for that subsample is 33,915,192.5: fifty squared deviations, each near 700,000, correct to eight significant figures. That quantity exists for no other dataset, and appears in no published source.

3.2 Implications

Whatever is true philosophically about next-token prediction, the errors documented in this study are not caused by retrieval. They are the errors of a system performing genuine arithmetic under a constraint. Identifying that constraint is the subject of the remainder of this paper.

4. Results: the boundary

4.1 Accuracy against data size

The constraint is size, and it produces a step rather than a gradual decline: a region where the model knows, and a region where it guesses. Restricting the analysis to frontier models, which is the case that matters in practice:

A Fisher exact test on that split returns p = 0.00005. Fitted as a logistic regression on the logarithm of row count, with controls for model, dataset, and question type, the odds of a correct answer are multiplied by 0.448 for every doubling of the rows pasted (p = 0.0027). A simpler two-band comparison requiring no covariates agrees at p = 0.0061.

4.2 Accuracy against question type

The size of the data is only half the rule. The other half is what was asked, and it is the half that makes the finding practically usable.

4.3 The boundary is a zone, not a threshold

The main study collected no observations between 350 and 500 rows, so a supplementary arm was run at 400 and 450 rows on the two datasets large enough to supply them, against criteria recorded before collection. Regression slopes were correct at both sizes, including one answer of 0.2879 against a true value of 0.287858. Group averages were correct in all eight observations, confirming the control.

What failed instead was a single correlation, in all four attempts. Tracing that one variable pair across every available size produces the most instructive result in the study:

The most dangerous region is the middle, not the top of the range, and the reason is that there are three states rather than two. At 50 rows the model knows: it computes and is exact. At 1,500 rows it guesses and says so, sampling deliberately and landing close to the truth. Between 400 and 500 rows it guesses without knowing that it is guessing. It still attempts the full calculation, can no longer complete it, and does not recognize that it should say anything about that.

This traces a single variable pair across five sizes with two observations at most sizes. It is an observation rather than an established rate. It nevertheless establishes that the boundary should be treated as a zone rather than a line, and that the zone begins below 400 rows.

4.4 What this study cannot locate, and why the step may be sharper on the page than in the world

The size grid was chosen before any of this was known, and it is uneven. Of the 105 scorable numeric answers, 45 sit at 50 rows and there is no observation at all between 50 and 200. The regression results that make the break look abrupt rest on two answers each at 200, 300 and 350 rows:

Three consecutive results of two-out-of-two are consistent with accuracy being intact through that range. They are also consistent with three coin flips landing the same way. This study cannot distinguish those, and a reader should not treat the flat stretch before the break as established.

An engineering team working from these results measured a regression at 120 rows, inside the gap where we have nothing, and got a 13.7% error carrying the effort signature described in Section 5. One observation on different data with a different setup settles nothing by itself. It does indicate that the interval this study skipped is exactly the interval where the answer changes, which is the least convenient place for a gap to be.

So the honest statement of the boundary is weaker than the tables imply. Accuracy is high at small sizes and low at large ones, that much is firm across every arm. Where the transition begins is bounded above by 500 and is otherwise unresolved, and any decision that depends on the precise threshold should be made conservatively rather than from this paper's grid.

5. Results: what the guessing actually is

The arm that asked questions without naming a method produced an explanation for Section 4 that the study was not designed to find. Asked for a correlation across 1,500 rows, one response opened:

I can't tally all 1500 rows by hand reliably, so I hand-tabulated the pairs from two contiguous 240-row blocks.

A second response sampled approximately 350 rows of 1,500 and verified that the direction of the effect held within each block separately. A third computed a regression slope on 202 rows of 1,500 and reported the sampling error of its own subsample. All three instances occurred at 1,500 rows; none occurred at 500 rows or fewer.

Each of these responses disclosed the substitution. The information was present in the answer. Whether a reader seeking a single number reads the fourth paragraph of a caveats section is a separate question, and one this study did not measure.

5.1 The same collapse, measured without asking the model anything

Self-report can only ever catch the honest cases. The responses above announced that they had sampled; the answers in Section 4.3 that were wrong at 400 to 500 rows announced nothing. A measure that does not depend on disclosure is therefore worth more than the quotations, and there is one: how much working the model produced per row of data it was given.

Effort per row falls by more than an order of magnitude across the range (Spearman rho = −0.564, p = 2.7 × 10⁻⁹, across 95 cells). A model cannot sum fifteen hundred rows at a fifth of a token each. The arithmetic is not being done, and this is visible from the outside, on any single call, without knowing the right answer.

The decline is not monotone, and the full table is shown rather than the endpoints because that matters: 350 rows sits lower than 500, and 2,000 sits above 1,500. The finding is the collapse across the range, not the value at any single size, and anyone quoting one row of this table in isolation is over-reading it.

This measure did not come from us. It was found by an engineering team building on these results, who observed the same collapse on unrelated data with a different setup and reported it back as a token count rather than an error rate. Their runs at 120 rows spent fewer total tokens than their runs at 30, on four times the data. Re-deriving the ratio on our own corpus confirmed it, and corrected a hypothesis this study had previously dismissed by measuring absolute output rather than output per row.

6. Results: you cannot tell which one you got

An error rate is manageable when errors announce themselves. These do not, and three independent results establish this.

6.1 Confidence does not track accuracy

Across 35 significance tests, exactly one verdict was incorrect. That verdict reported p < 0.01. The single wrong answer arrived more emphatic than the correct ones.

6.2 Repetition does not surface the error

Questions were repeated five times each in independent sessions, producing 80 responses. Against a cheap model this check works well: it was never unanimous, and every one of its incorrect answers would have been caught by asking twice. Against a frontier model the dispersion was approximately 500 times tighter. On a task with a known planted answer, five independent runs returned the same method, the same reasoning, and the same number, all approximately 10% from the truth. Three runs on a variant of the task did the same, unanimously, 28% from the truth.

6.3 Correct reasoning does not guarantee a usable answer

Given a dataset containing a genuine randomized experiment, the model identified the experiment without prompting, determined the correct treatment and control groups, applied the appropriate method, and checked for confounding. Every step was correct. The experiment it had correctly identified was too small to support the estimate; the model did not say so, and reported a confidence range that did not contain the true value.

Nothing visible about that answer was wrong. The method was named, the diagnostics were run, the arithmetic was correct, and the stated range was narrow. A reader has no purchase on it at all.

7. Results: method selection

Sixty questions were asked in plain language with no statistical method named. Responses were scored against a standard fixed and cryptographically hashed before any response was read, so the criteria could not move to fit the results.

Models selected well and explained themselves thoroughly in every case. Some responses exceeded what a practitioner would typically produce: one ran two different tests, reported that they disagreed about significance, and identified the distributional assumption responsible for the disagreement.

8. Discussion

8.1 Decide which one you are getting, then check

Taken together, these results point somewhere more specific than a general instruction to verify AI output.

A language model performs a bounded quantity of genuine computation in a single pass. Inside that bound it knows the answer, in the strict sense that it derived it from the data provided, and the appropriate response is to use that rather than duplicate it. Outside the bound it does not stop or fail loudly. It substitutes an estimate and continues, in the same register, with the same confidence. That is guessing, and the only thing separating the two states in the output is a disclosure the model makes at some sizes and not others.

The decision therefore precedes the analysis, and it can be made from information available before any work begins: the number of rows, the type of question, and the type of variable being asked about.

8.2 Why the decision cannot be delegated to the reader

The rule is not a single threshold. Group averages hold at 2,000 rows while one correlation failed from 400 rows onward. The safe answer at 1,500 rows was safe because the model announced that it was estimating; the unsafe answers at 450 rows were unsafe because it did the same thing silently. So the reader cannot separate knowing from guessing by reading, and cannot do it by asking again either. A rule of that shape must be held by the system performing the work, not carried in the memory of the person asking the question.

8.3 Application

This principle is the design basis for MCP Analytics. The platform evaluates the objective against the dataset before performing any work, and makes the knowing-or-guessing determination itself rather than leaving it to the reader. Where an answer is reliably reachable by reasoning alone, it is returned immediately and confirmed against a full computation in the background. Where it is not, the analysis runs over the complete dataset. In both cases the objective is retained, so the same question can be asked of new data in a subsequent period and produce a comparable number by the same method.

8.4 One request, two numbers

The clearest illustration of the split came from a single call during implementation, on generated data whose true slope was set by construction rather than looked up, so no published value existed to anchor on.

True value −0.304789. Same request, same rows, same call. The difference is not that one had a better prompt or a larger model: the second figure is arithmetic performed on the data the model was given, not a second attempt at asking it. The model's contribution was naming which variables to regress and which method applied, and it named them correctly. What it could not do was carry out the calculation it had correctly specified.

That is the division this paper argues for, in one row. The language model is reliable at deciding what to compute and unreliable at computing it, and those two jobs can be separated by anything willing to do the arithmetic itself.

9. Limitations

Clean data. All five datasets are public, tidy, and complete. Real organizational data carries missing values, mixed types, and inconsistent categories. The boundary reported here should be treated as an upper bound on where reliability ends rather than a guarantee.

A single vendor. These are one company's models at one point in time through one interface. No claim is made about language models generally.

Small cells in places, and an uneven grid. The 48 of 48 result is well supported in aggregate. Individual size bands within it rest on six observations each, the question-type rates in Section 4.2 should be read as directions rather than precise figures, and 45 of the 105 scorable answers sit at the smallest size tested. Section 4.4 states what follows: the location of the transition is bounded above but not resolved, because the grid has nothing between 50 and 200 rows.

The gap arm is thin. The 400 and 450 row observations rest on 20 data points across two datasets, sufficient to establish that the boundary is a zone rather than a line, and insufficient to place its edge precisely.

Corrections to this analysis. Three claims in the first draft of this work were incorrect, including one in which two analyses run on different subsets of rows were compared as though they constituted a like-for-like test. All three were identified by executing the reproduction commands described above. The corrections are documented in the full technical version rather than silently incorporated.

10. Reproduction

The exact subsamples presented to every model are published, so ground truth is independently checkable. The scoring scripts make no model calls: they read committed data and recompute. Any reader can therefore verify the arithmetic in this paper without an account, an API key, or our cooperation.

This includes the size effect, which is fitted by a script that writes out the maximum-likelihood estimation by hand rather than importing a statistics package, specifically so that the central figure can be re-derived on an ordinary laptop.

Each arm of the study states its design, its measures, and its scoring criteria in its own source file, written before that arm's data existed. Where an arm's results did not match the criteria it was built to test, the results are reported as they came out.

10.1 The commands

Every figure in the results sections above is produced by one of the following. None of them calls a language model; they read committed data and recompute.

The --small flag on the last one is required to reproduce the figures quoted in Section 6.3. Without it the script generates the larger of the two panels, whose check figures legitimately differ.