Skip to content

Language audit: "survives a null model" overclaims what a null model does #4

Description

@bshepp

The site's tagline reads:

Render an integer sequence, then test whether the structure you see survives a null model.

This overclaims, in two separate ways, and the site's own worked examples already contradict it.

1. A null model destroys one specific, identifiable property

Each surrogate is a hypothesis, not a neutral baseline:

surrogate destroys preserves
permutation the ordering, entirely the multiset of values
difference the ordering of the steps the multiset of steps
matched-random everything but the summary trend and spread

So a permutation null answers exactly one question: could this picture have come from these same values in some other order? Rejecting it tells you the ordering matters. It does not tell you the structure is "real", meaningful, or characteristic of the sequence.

And for a large class of sequences the ordering is the definition, so the test is close to vacuous: a(n) = n fails its permutation null resoundingly and the result is worthless. The rainbow-rings entry says this in as many words. The headline and that entry cannot both be right.

2. "The structure you see" is not what gets tested

The eye sees a pentagram; the measurement is residue alternation. Those are different objects, and the a000464-pentagram entry is entirely about the gap between them - the walk closing is real, the five-pointedness is the protractor. The tagline flattens exactly the distinction the best entry on the site exists to draw.

Scope

Not just the tagline. Sweep for the same overclaim in:

  • src/ui/landing.ts - tagline, lede, hero caption
  • src/ui/about.ts - the question, the method section
  • src/examples/entries.ts - captions and bodies; check each verdict word
  • VERDICT_LABEL in src/ui/landing.ts - "Survives the null" carries the whole ambiguity in three words
  • README.md, docs/*.md
  • src/nullmodel/surrogates.ts - SURROGATE_EXPLAIN strings
  • the verdict type doc in src/examples/types.ts - real / artifact are themselves the loose framing

What to replace it with

Say what is actually measured. A null model asks whether a named statistic is reproducible by a named scrambling. Wording should name both, or at minimum stop implying a general verdict on reality. The existing split verdict and the open verdict are already honest; real and artifact are the two that need work.

Raised by the site's author, who is right.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions