Stated uncertainty

Every one of the five slices sees at least one of resubstitution, held-out and the Cover–Hart bracket fail its nominal 0.95 at some sample size, and none of the failures is visible from inside the estimator that carries it: each interval is computed the same way whether it is right or wrong, and only the closed form says which. Which of the two point estimators breaks is slice-dependent, and the ordering inverts. At the knife slice resubstitution covers 0.00–0.03 while held-out covers 0.93–0.97; at the widest floor slice resubstitution starts at 0.00 and recovers only at the largest n while held-out falls 0.90, 0.50, 0.25, 0.15. Neither of them is simply the good one, so a practitioner who validated a pipeline on any single slice would carry the wrong conclusion to the next. Coverage alone does not settle it either: both brackets cover at 0.90 or better at every slice and every n — the divergence one under a feed order that does not group the labels; what the grouped order does to it is the last failure below — and buy that by containing the no-evidence value along with the truth. And at the knife slice a bracket's coverage measures a cap rather than an estimator, since its upper end is held at 1/2, which no binary Bayes error can exceed, and the truth there is 1/2.

Scope. Computed on the five slices at n = 500, 2000, 8000, 32000, forty trials each, one fixed seed, 0-1 score, uniform prior. Nominal coverage 0.95 throughout. Two of the five slices are interior rather than floor slices, and the four find them no harder: what the window moves is the no-evidence baseline, not what an estimator does. A positive control — a two-fiber Bernoulli problem with R* = 0.15, far from 1/2 — passes through the same sampler and the same estimator code before any slice result is read. Toy scale.

verifiers: explore_ceiling_ruler.py, explore_ruler_divergence.py

Two of the failures are estimator mechanisms rather than small-sample accidents — one permanent in n, one transient — and neither is a fact about these rings. At the knife slice resubstitution's deficit is the mean absolute deviation of a fair binomial, ½√(2M/πn), while its own half-bar is the binomial 95% one, 1.96√(1/4n): both shrink as 1/√n, so their ratio √(2M/π)/1.96 is free of n — 1.5766 derived for the size, and −1.48, −1.55, −1.56, −1.59 printed across the four sample sizes, the sign recording that the estimate sits below the truth. The point estimate is consistent, its bias running −0.0644 to −0.0087; the interval is not, and coverage stays at 0.00–0.03 at every sample size, so more data never repairs it. At the widest floor slice held-out fails in the opposite direction, its coverage falling as n grows through the series above — and only there, since at the narrow floor slice the same estimator's coverage climbs with n. It scores a fitted rule whose excess over the optimum decays with the probability that some fiber's empirical majority flips, while its reported precision decays as 1/√n. A tightening interval reads as improving evidence, and here it is not. This one is the transient: the falling lasts only while the slice's 105 near-half fibers' majorities stay unresolved, and a 400-trial re-run sees the coverage turn back up at the largest n (the coordinate that decides returns to it).

Both reappear with no ring anywhere. The same estimators through the same code, on M bare Bernoulli fibers, reproduce the ratio at −1.52 to −1.62 and the falling series at 0.88, 0.68, 0.23, 0.25. A dial the ring cannot turn — the distance δ of η from 1/2 — locates the interval failure in the band 2δ√(n/M) ≲ 1: at n = 8000 a distance of δ = 1/64 from 1/2 already raises coverage from 0.00 to 0.85, and 1/32 restores it outright. So what a designed family contributes is not the failure but the placement: one of its floor slices sits at η = 1/2 exactly, inside a band about that wide — the others at 4/7 and 6/11, close to 1/2 and off the band at the larger n — with the truth a rational number. Calibrated position on a thin edge, and not a cause.

Scope. The rule is the ring-free reappearance: derived by hand off the ring and confirmed at the slices' own M, η and n. The two failure mechanisms are observations — closed forms derived asymptotically and confirmed at four sample sizes. The falling series is stated as a direction and not as rates: it survives all three tie-breaks of the fitted rule's arbitrary empty-fiber choice, the two harsher ones falling further, while two random streams of one setting differ by 0.12, which is the scale of forty-trial coverage noise. The band's location is measured rather than derived: the crossing sits nearer 1/2 than √(M/n). 0-1 score, uniform prior. Toy scale.

verifiers: explore_ceiling_ruler.py, explore_ruler_boundary.py

The two bracket estimators are not two bounds. Once n ≫ M every test point has an exact residue match, so the 1-nearest-neighbour rule returns a random same-fiber training label and its error tends to Σ_r P(r)·2η_r(1−η_r) — 1/2, 24/49 and 60/121 on the slices run, with measured gaps at most 0.0007 at the largest n. The Henze–Penrose divergence the spanning-tree estimator targets has the exact value E[(2η−1)²] here, and (1 − that)/2 is the same quantity again, so the two are one bracket read at two statistics. A second estimator family therefore buys a different way of estimating the bound and never a second opinion on it. Its equality case is this family's own window: the lower endpoint's shortfall is the Jensen gap of |2η_r−1|, which vanishes exactly when that is constant across fibers — when t is divisible by M, or on the middle run of thresholds a cell with odd cofactor carries, where every fiber splits its points as evenly as an odd count can. Off the window the endpoint is strictly conservative, by at most 0.0178 across the thirty thresholds run.

That one bound is also too weak to certify this family's own lift, and that too is a fact about the bound rather than about either estimator reading it. Read exactly, with no sampling anywhere, the bracket excludes the no-evidence value at 2 of the N = 105 cell profile's 14 in-window thresholds and contains it at the other 12; add the sampling half-bar at n = 8000 and it excludes it at none. The family's own exact lift there is 7/105, and the sampled divergence bracket is still 0.0838 wide at the largest n. A bracket can be correct at every slice and unable to certify that the evidence is worth anything, with the emptiness located in the bound rather than in the sample.

Scope. The identity of the two brackets and the tightness set are derived by hand and confirmed in exact rational arithmetic on all 35 (slice, threshold) pairs the file touches, and both are the rule the badge states; the certification counts are observations, computed on the 30-threshold profile of the N = 105 cell at n = 8000, forty trials. Its high coverage has two independent sources and neither is accuracy: the truth sits at the exact bracket's lower endpoint on all five slices, so containing it costs nothing at the top, and the stated bar is binomial where a tree's edges are dependent, which overstates the statistic's real dispersion by 1.75 to 2.75 times at every slice and sample size. Two estimator families, 0-1 score, uniform prior. Toy scale.

verifiers: explore_ceiling_ruler.py, explore_ruler_divergence.py

A spanning-tree estimator needs a distance, and the only one the evidence offers — how many shown residues two points differ in — puts every pair of points in the same fiber at distance zero. So any spanning tree of a fiber's points is a genuine minimum spanning tree, and the statistic is not a function of the sample. Fed the rows in class order — what concatenating two label arrays hands a deterministic routine — each fiber contributes exactly one edge joining opposite labels, so the statistic is a function of the fiber count alone: the cross-label fraction reads 0.0005 at n = 32,000 at all four slices with M = 15, with across-trial spread 0.0000, identically where the truth is 3/7 and where it is 1/2, and the bracket it maps to — [0.0001, 0.0007] — covers at 0.00. It converges on the assertion that the labels are perfectly separable. Shuffling the rows, which changes no datum, moves the same statistic to 0.4897.

Those two are not the extremes. A fiber's zero-weight edges admit a spanning tree of the complete bipartite graph between its two label classes whenever both are present, and every edge of that tree joins opposite labels, so across genuine minimum spanning trees of one sample the reported bracket runs from [0, 0] to [1/2, 1/2]. The identified set — the values the data leaves possible once the estimator's arbitrary choices are allowed to range — is therefore the whole interval a binary Bayes error can occupy. That is not a perturbed answer but the absence of one, and it is a question of whether the estimator is defined on data of this shape rather than of how well it does. What the repair needs is not randomness but a tree whose shape does not consult the labels; the labels are then exchangeable across its edges and the expected cross-label count is the population quantity whatever the tree, so a deterministic order is fine and class order is not. An exact Euclidean minimum spanning tree on a jittered embedding, built by geometry and blind to labels, agrees with the shuffle — 233.1 same-fiber cross edges against 239.8 at n = 500.

Scope. Computed on the five slices at four sample sizes, forty trials, with the mechanism and the identified set derived rather than measured; the label-blind control is a single configuration at n = 500, its agreement inside a conservative Poisson bound. Categorical evidence is what forces the non-identification: the residues carry no order and no size, so every within-fiber distance is zero and the tree has nothing to minimize. 0-1 score, uniform prior. Toy scale.

verifier: explore_ruler_divergence.py

What fails is the statement rather than the estimator. A stated interval is computed from the estimator's own model, so it can price only error that model has a coordinate for; error outside the model is uncounted and reaches the comparison against truth without ever entering the arithmetic. Two properties follow, both measured against exactly known truth — each world’s own Bayes error, a rational number — across six arms differing only in what their statement knows. Each arm runs on a designed world: fifteen equally likely fibers carrying stated rates, clear where no rate is nearer than 1/10 to 1/2, knife where every rate is 1/2 exactly, and mild where three rates sit just past 1/2 and the other twelve well away from it.

The first is an asymmetry. Call a statement a bound when its uncounted term has a known sign: fit the majority rule on half the draw, score it on the other half, state the one-sided upper confidence limit, and the slack carried is the rule's excess over the optimum, which cannot be negative. The bound arm never came out overconfident beyond trial noise, on either world or at any sample size: it covers 0.963–0.988 against its own one-sided nominal 0.975, every reading inside the Monte-Carlo band, at all eight — the clear world and the knife, four sample sizes each — while on that knife world a fit statement — a two-sided interval computed from the fitted model's own noise — covers 0.015–0.035. What is forced is that one direction and not its mirror. A fit is merely the only kind that can fail downward, and nothing makes it: the knife fit's own half-bar runs 3.10 to 3.26 times σ, the true spread of its estimate across trials — too wide, not too narrow. The usual corollary is false too. That bound excluded the no-evidence value in every trial on the clear world, so conservatism is forced by the sign and emptiness is not: what the two brackets above show is their own slack.

The second decides whether more data cures it, and it is one ratio — the uncounted term b against the half-bar h the statement does count. Three regimes, all three from the same plug-in code. Where the model contains the truth, b/h falls 0.032 to 0.002 and coverage sits at nominal: 0.955, 0.945, 0.932, 0.953. At the knife the two shrink at one rate, the ratio holds at 1.573–1.595, and coverage is flat at 0.035, 0.015, 0.025, 0.035 — sample size irrelevant. And where the model cannot hold the truth — the mild world's fifteen fibers pooled into three groups of five, each holding four of the twelve rates well off 1/2 and one of the three just past it — the ratio grows 0.259 to 1.781 while coverage falls 0.915, 0.875, 0.555, 0.045, the bias sitting from the second sample size on at the exact pooling gap 7/750 — the distance from the pooled model's own limit to the truth. Nothing there degrades except the claim. The crossing has a location: coverage passes 1/2 where the uncounted term equals the stated half-bar, which holds wherever that half-bar outruns σ, as it does at every fit arm here — by a factor of 1.9 to 2.1, and of 3.1 to 3.3 at the knife.

The two are one fact. Across six arms and both statement kinds — seven arm-and-world pairs, since the bound runs on two worlds, at four sample sizes each — measured coverage is what the pair of ratios b/σ and h/σ predicts and nothing else enters — largest deviation 0.0133, mean 0.0049 over all twenty-eight. Mechanisms that read as distinct act only by moving those two numbers, and the direction a statement fails in is set by which region of that plane its construction can reach.

Scope. Observation throughout, at 400 trials each and n = 500 to 32,000 over M = 15 uniform fibers with exact rational targets. Each arm's regime follows from its construction; its trajectory does not. The two-ratio formula follows from approximate normality of the estimate, and what is measured is that the approximation holds at every one. A control is read before any of it: the knife arm reproduces the constant derived above, √(2M/π)/1.96 = 1.5766, at 1.573–1.595 — a differently written estimator on a differently built world landing on the same closed form. The third regime is read on the small pooling gap because a large one leaves no trajectory to see: the same pooling on groups that straddle 1/2 gives a gap twenty times wider and coverage 0.000 at every n. Two overconfidence mechanisms are instantiated — a bias omitted at the same rate as the noise that is counted, and a model class that cannot contain the truth — plus the bound kind. Two others are run below and land on the same plane: a precision decaying faster than the fitted rule's excess over the optimum, and an atomic truth smoothed toward 1/2 by a designed weight. Every arm's bias has a known cause, so nothing is said about whether an uncounted term is detectable from inside. 0-1 score, uniform prior. Toy scale.

verifier: explore_uncounted_term.py

That asymmetry — a bound's uncounted term carrying a known sign — meets its sharpest case in an error that is not a displacement at all. A published deletion audit — scored elsewhere against exact posteriors — estimates a mixing weight π by matching one sample's variance against the variance a two-class mixture predicts, minimizing over π in [0, 1], the parameter box. Where the two classes have equal variance that objective cannot tell π from 1 − π, and no sample size separates them. Write γ for the gap between the two class variances. The second solution then sits at 1 − π + γ/4, exact at every γ and matched to 1.1·10⁻¹⁶, so unequal variances never destroy it — it stops competing only when it leaves the box at γ = 4π. Identification is restored by the box and not by the data, and that is a property of the objective rather than a reading of any sample: the resampling distribution's weight on the second solution falls 0.54, 0.42, 0.34, 0.26 at the smallest audit size as γ grows and that solution walks to the box edge, and is 0.00 once it is strictly outside.

Read the identified set here as the values the population itself leaves possible — the same notion as above with the estimator's arbitrary choices replaced by the population's own ambiguity. At γ = 0 it is the two points 1/5 and 4/5. A bound does not have to choose between them: the error it cannot count enters with a known sign and can only widen the statement, so a pair of one-sided bounds closes on that set's hull rather than on a point. The bracket runs [0.004, 0.997], [0.057, 0.946], [0.154, 0.845] across three audit sizes, and its excess beyond [0.2, 0.8] falls 0.196, 0.143, 0.046 at the low end and 0.197, 0.146, 0.045 at the high — the two agreeing to within 0.003 though one arm is computed off the smallest solution and the other off the largest. Three sizes measure the approach; the limit is the identified set by construction and not by measurement. No single arm carries this. Each is one-sided, so whichever end of the box the truth sits nearest is the arm reporting trivially, and only the pair reads the ambiguity. A statement can be tight against a truth the data cannot pin and stay honest: what cannot be tightened is the one direction the second solution sits in.

That is also what makes the failure legible. The bound's slack contracts to 0.051 where the parameter is identified and refuses to fall below 0.645 where it is not, still two thirds of the box at the largest audit size, while on the same resamples the fit band narrows from 0.809 to 0.671 and its coverage does not rise: 0.855, 0.915, 0.820 against its own nominal 0.90. A tightening interval whose coverage stays put is invisible to that statement and not to the data. Changing the statement kind, and nothing else about the computation, converts it into a width that will not close.

Scope. Observation, 24 settings: a Gaussian sweep in γ at π = 1/5 plus two settings of the world that audit was scored on, at three audit sizes, one seed stream, 200 replicates per setting and 200 resamples per replicate — the audited tool's own counts. The second solution's formula and its exit at γ = 4π are a property of the objective, derived and then matched in floating point. A control is read before any of it: on a fresh seed stream the fit arm reproduces the recorded non-identified setting, in both the weight it puts on the second solution and its band coverage. The bound arm is one-sided at 0.95 against the fit band's two-sided 0.90, so the two nominals are stated rather than compared. Its nominal did fail at six settings, four for the upper arm and two for the lower, and the shortfall is anticorrelated with the term the law is about: on the sweep it sits entirely at the identified settings, 0.875–0.965, while every setting where the objective keeps both minima runs 0.955–1.000, which puts the residue in the resampling quantile's own calibration and not in the sign. No setting breaks both arms — at none of the 24 is the pair wrong together. Three repairs that would place this specimen on the two-ratio plane all fail: what the plane assumes is a point plus or minus a half-bar with jointly near-Gaussian parts, and a quantile band off a resampling distribution with two peaks is not that shape. Toy scale.

verifier: explore_nonidentified_statement.py

An auditor holds the other side of that plane: an interval has been computed and there is no known R* to score it against. One of the two coordinates is free. σ is the spread of the estimate across repetitions, and resampling the one sample in hand supplies an estimate of it with no reference to any truth, so h/σ is computable from inside. It is the wrong half. On the mild world at the four sample sizes, the per-fiber fit and the pooled fit divide h by a resampled σ and get 2.21 against 2.01, 2.09 against 1.96, 2.03 against 1.96 and 2.00 against 1.97 — indistinguishable — while their coverages run 0.920 against 0.945, 0.930 against 0.828, 0.955 against 0.562 and 0.920 against 0.065.

Two statistics can be built for the other coordinate, each reading b out of the sample it came from. The bootstrap term is the estimator's mean shift across resamples of the sample itself; the drift term is the estimate on the whole sample minus the estimate on half of it, which reads b(n) − b(n/2). Neither reaches R*, and each loses it differently: the drift differences it away, while the bootstrap measures the estimator's bias against its own estimand — the number its arithmetic returns on the whole population — which for a model that cannot hold the truth is that model's limit and not R*. The resampling itself is from the raw sample, fiber by fiber and label by label, with no pooling in it: what drops the gap is the estimand, never what is resampled. Split b into a part that decays and the gap between the model's limit and R*: the bootstrap reports the first and drops the second, the drift differences the second away, and one kernel is reached from two directions. What the kernel rests on is the estimand gap — the distance from the estimator's own estimand at the sample size in hand to R* — and not on anything having converged. One face of it is the model class, the set of distributions the estimator's own arithmetic can express: where the class cannot hold the truth, the gap is the constant part of b, and nothing computed inside the class can point at an R* the class cannot express. Call one estimator on one world at one sample size a case. Measured at the five cases where b has settled onto such a gap, the true |b|/h is 4.33, 8.84, 17.71, 35.50 and 1.78 against inside readings of at most 0.15 — the dangerous regime, since h shrinks like 1/√n and a constant b is the plainest way to send b/h to infinity. But convergence is not what hides the term. Fit every fiber, then pull each fitted rate toward 1/2 with a designed shrinkage weight decaying as n^−1/4 — a class that holds the truth and a bias that still moves at every n, yet one whose ratio diverges, the weight decaying slower than the half-bar: |b|/h runs 1.22, 1.86, 2.67, 3.75 across the four sample sizes while coverage runs 0.255, 0.015, 0.000, 0.000, more data strictly worsening the statement. That is the corner the settled cases and the knife leave between them: divergent and still moving. The pull is the same function of n on the sample and on any resample of it, so it cancels exactly in the bootstrap's difference: that term reads 0.110 to 0.001 of h as the true ratio climbs — blind by an identity rather than by convergence, a deterministic pull sitting in its kernel while still moving. Only an instrument that reruns the estimator at another size sees it.

A goodness-of-fit test escapes the kernel the one way the kernel allows, by fitting a richer class, and pays for the escape by detecting the model rather than the term. Give it a world whose three groups each have all five of their fibers on one side of 1/2 — the condition is on every fiber and not on the group's mean, which a group of 0.1, 0.1, 0.1, 0.1, 0.9 already breaks. The pooled model is then wrong about every fiber and exactly right about R*, since the smaller of a group's mean and its complement is the mean of the fiber-wise smaller ones. The pooling gap is exactly zero, coverage is nominal at 0.935, 0.955, 0.940, 0.955, and the within-group Pearson statistic — standardized so that a correct pooled model would give mean 0 and unit spread, putting a reading of 3 already well out — climbs 8.0, 33.8, 136.2, 548.5 in n: loudly right about the model and silent about what the model costs.

Where the term has settled the verdict is no, and the failure is anti-monotone. A detector is a per-trial statistic with a threshold. At thresholds holding false alarms to 0.10 over the cases that cover, the bootstrap term catches 0.432 and the drift term 0.196 of the 4,000 trials of the ten cases that do not. The fit test exists only where there is a richer model to fit, so it is read over the six of those ten that pool — 2,400 trials, a pool with none of the knife in it — and catches 0.167 there. Three readings, then, and not one comparison. The bootstrap term is not weakly informative but pointed the wrong way: case by case it catches 1.000 at each of the four knife cases, and then 0.302, 0.022, 0.000, 0.000 as the true |b|/h rises 4.33 → 35.50. Power falls as the damage grows, severity being bought by the very convergence that empties the kernel. Its 0.432 is the four knife cases whole plus a residue: 130 detections across the 2,400 non-knife trials, every one at the two smallest sample sizes of a single arm, where b had not yet settled onto its gap — 0.1897 at the smallest sample size and 0.1934 at the largest, against a gap of exactly 29/150. On the corner the same family splits the other way: the drift's detection rises 0.028, 0.113, 0.193, 0.362 as |b|/h climbs 1.22 to 3.75 — power rising with the damage — and an auditor who runs the audit at two sizes outright and subtracts, 500 nested inside 32,000, detects 0.522, the strongest reading on the corner. So the direction a stated uncertainty fails in is knowable from inside exactly where the uncounted term is still moving — the divergent regime included, on its moving boundary — and not once the term has settled.

The fit test's own repair is to read the same two fits in the target's coordinates rather than the model's: the pooled estimate of R* minus the per-fiber estimate — nonnegative on every sample, its mean the pooling gap itself, the exact quantity the kernel hides. That statistic catches 0.979 of the failing converged-gap cases' trials at false alarms held to 0.10 and stays silent, 0.033 and under, on the zero-cost world above, whose good trials push the raw statistic's threshold past 500 and collapse its detection to the 0.167 read earlier. What it prices is the term's size and never its lethality: on a case still covering at 0.935, carrying the real gap 7/750, it alarms at 0.468 — wrong as a verdict on coverage, correct at every size about what the model costs where R* reads it, h alone deciding when that cost turns fatal. An early warning rather than a verdict. What stays uncounted after both instruments is error outside every class the auditor can nest: the kernel narrows, it does not empty. And the two new mechanisms land on the two-ratio plane beside the rest — 32 cases, worst deviation 0.023 — the shrinkage a permanent resident of the corner, the held-out failure at the widest floor slice above a temporary one: its term decays slower than 1/√n only while its 105 near-half fibers' majorities stay unresolved, and re-run at 400 trials its coverage turned back up inside the measured range of sample sizes: 0.863, 0.495, 0.152, 0.278.

Scope. Observation throughout except the kernel and the shrinkage cancellation, which are properties of the construction, the first measured at five fits whose b has settled onto a nonzero gap. Two rigs, one seed stream each. The first: 400 trials at each of n = 500 to 32,000 over M = 15 uniform fibers with R* an exact rational, 150 resamples per trial, seven arms across four worlds; seven arms at four sample sizes is 28 cases, of which the detector rates are read over 27: 17 covering at 0.90 or above and 10 below 0.80, with the remaining one at 0.828 excluded as neither. The second rig reruns the fit arms and adds the shrinkage, its fast-decay control and the held-out specimen — eight arms, 32 cases. A control is read before either: the first rig's knife arm reproduces √(2M/π)/1.96 = 1.5766 at 1.573–1.613 and its truth-containing arm covers 0.938–0.953 against a nominal 0.95; the second rig's plain and fast-shrinkage arms cover 0.945–0.970 and its pooling gaps print exactly 29/150, 7/750 and 0. Every detector's threshold is set to hold false alarms to 0.10 over the covering cases' trials, the zero-gap world's among them, so 0.167 and 0.979 are readings of detectors of the term and never of misspecification, which the raw statistic detects perfectly. The two-size reading is one nested draw per trial. 0-1 score, uniform prior. Toy scale.

verifiers: explore_uncounted_inside.py, explore_uncounted_corner.py