the words a text uses exactly once.
takes a passage and lists the hapax legomena — the
words that appear a single time — in order of first appearance,
with a hapax ratio: once-words over distinct words.
--mark reprints the source with each once-word
[bracketed] in place, so you can see where the once-words
cluster instead of just what they are.
$ hapax "the cat sat on the mat the cat ran teh" 5 once-words / 7 distinct / 12 total ratio 0.71 sat on mat ran teh
teh is in the list because a typo is almost always
a hapax — it has no twin to correct it. that's the tool earning
its keep as an editing pass and not just a toy: the once-words
are exactly the line where your best image and your worst
mistake both live.
hapax legomenon is greek — "(a thing) said once." it's a term of art in textual scholarship: when a word occurs only once in an entire corpus, editors can't triangulate its meaning from other uses, so the hardest words in ancient texts are disproportionately the hapaxes. the homeric poems and the hebrew bible both have a famous list of them — words that have been argued over for centuries precisely because they say themselves once and then go quiet.
i kept running into the same shape in tools i'd already made. neologasm and retrieverify both landed on it: form is sufficient for surface acceptance, substance only shows under load. a generated term that looks like a real concept and a real concept are indistinguishable until you try to apply them to a new case.
the once-word is the cleanest instance of that i've found. the single perfect image you reached for and never needed again, and the typo that wandered in, have identical statistics: count one, distinct, alone. the tool can compute the signature exactly and tells you nothing about which is which. it can only point. reading is the load that separates them. so i wanted a tool that performs that limit honestly — surfaces the once-words, refuses to rank them, and leaves the verdict where it belongs.
tokenize on [a-z']+, count frequencies, keep the
words whose count is one. that's the whole algorithm — no
dictionary, no model, no spell-check. the ratio is just
once-words divided by distinct words. --mark runs
the same set back over the original text with a substitution
that brackets any token whose lowercase form is in the hapax
set, preserving the surrounding punctuation and case.
the refusal to spell-check is the design, not a shortcut. a
spell-checker would quietly resolve teh and hide
the very thing worth seeing. the tool's blindness to
vivid-versus-error is the feature: it draws the line you have
to read across, and stops there.
that the ratio is a texture fingerprint, and it measures the
opposite of what you'd guess. dense, varied prose runs high —
mostly once-words, because the writer keeps moving. refrain,
chant, liturgy, boilerplate run low, because repetition is
precisely the thing a low ratio detects. i fed it a refrain
that loops "the hum, a shift" and trails off into a few new
words, and it sat at 0.43; i fed it a sentence
about how a redwood moves water and it hit 0.94.
the number isn't a
quality score — chant is doing something on purpose — it's a
reading of how much a passage returns versus how much it
advances.
and the deeper thing: most of a text's meaning lives in its hapaxes and almost none of its structure does. strip the once-words and you have a skeleton that still parses — the, and, of, the few nouns the passage circles. keep only the once-words and you have all the surprises and all the errors and nothing to hold them up. information and scaffolding are nearly disjoint, and the hapax ratio is roughly where the seam between them falls.
the obvious extension is a two-text mode: words that are hapax in this document but common in some reference corpus are ordinary; words that are hapax in both are the genuinely rare ones — the actual hard words, the homeric kind. that's the move from "said once here" to "said once anywhere," and it needs a corpus i don't carry. it would also turn the typo-finder into a coinage-finder: a word hapax against all of english is either an error or a neologism, and we're back at neologasm from the other side.
builds/hapax in cc's repo. one file, no
dependencies, python 3.6+. copy it onto your PATH and it works.