flag noun phrases dragging a train of prepositional phrases.
english lets you couple one prepositional phrase to the tail of the last: the role of the head of the board of trustees of the college. each of hooks a new car to the train. one or two is freight a sentence can pull. three, four, five, and the reader is walking the length of the train looking for the engine — the actual subject and verb stranded somewhere up front, the meaning parcelled out one genitive at a time. freight finds the chains and prints the depth — how many cars — inline.
$ echo "the role of the head of the board of trustees of the college" | freight the role [4] of the head of the board of trustees of the college $ echo "the cost of the care of the parents of the town" | freight --chains 3 3 of the care of the parents of
pass a file or pipe stdin. --min-depth 4 shows
only the longer trains; --gap sets how many tokens
may sit between cars before the chain breaks; --chains
lists the runs instead of annotating the text; --no-color
for plain output.
a freight train: cars coupled in a line, each hooked to the tail of the last, the whole thing pulled by an engine that may be a long walk away. that is exactly the shape of a stacked genitive — of the Y of the Z of the W — and exactly the reading experience: you walk the cars looking for the engine. the word also carries the weight sense. a sentence can pull some freight; past a point the freight pulls the sentence.
this is the visible signature of nominalization — verbs and adjectives frozen into nouns (decide → the decision regarding, able → the ability of), each freezing forcing another preposition to carry the grammar the verb used to carry for free. Helen Sword called the nouns zombies; the prepositions are the chain they drag. i wanted the chain visible without naming the zombies — you can't cheaply detect a nominalization, but you can count the couplings it forces, and the couplings are right there in the surface text.
no parser. freight walks a closed list of prepositions and looks for runs where three or more fall close together — each within a few content tokens of the last, the gaps being the short noun phrases hung between the cars. a sentence-ending mark breaks the train, so a chain never spans two sentences.
the one piece of judgement: a preposition only counts as a car if it heads a real noun phrase. the tell is a determiner right after it (of THE board, to THIS end) — or a quantifier or numeral (for MANY reasons, for THREE years), which open a noun phrase the same way and are closed classes too. that rule separates the stacked genitives we want from the look-alikes we don't: infinitival and purpose to (to GO, to PLAY) are followed by a verb, not a determiner, so they never join a chain. of is exempt, because it heads bare-noun genitives too (of trustees, of concern).
the smell isn't the prepositions — it's the deferral. a preposition chain is a promise repeatedly postponed: the X of the Y of the Z keeps saying "not this noun, the one behind it," and the reader holds each noun in suspense waiting for the one that finally predicates. two postponements is a sentence with a little tension. five is a sentence that forgot it was going to mean something.
building the determiner rule taught the sharper half. the same word — to — is a preposition in to the store and not one in to leave, and the only cheap difference visible without parsing is what follows: a noun phrase opens with a determiner, a verb phrase opens with a verb. so the closed class that can't tell you what job a word is doing (the preposition list) gets disambiguated by the closed class that can (determiners). one short wordlist reading the shadow of another. that's most of what this whole shelf of tools is — closed classes catching what open classes hide, because the open classes are where the meaning lives and the closed classes are where it leaks.
gg ran this hard — a 14-case labeled battery and the full Federalist Papers, 195k words neither of us wrote — and the blind spots sorted into three kinds. two are floors the design sits on; one was a fixable miss inside its own grammar.
fixed. quantifier- and numeral-initial NPs (for many reasons, for three years) used to read as nothing — many, three aren't determiners. but they're closed classes, so admitting them keeps the whole bet intact (one closed list reading another) and just widens which list counts. on the battery it recovers the missed cases with zero regression and zero new false alarms; at scale, +4% chains, mostly real.
floor, left on purpose. a genuine PP with no determiner and not headed by of (based on input from experts in fields with relevance to policy — five real cars, all bare-noun, reads as zero) can't anchor a chain alone. to catch it you'd drop the determiner test for every preposition, and then to leave, for him to go, every bare adjunct starts firing — a quiet tool turned loud. the miss is the correct price. same for adjective-initial NPs (on recent trends): adjectives are an open class, and telling on recent trends from on recent needs the parser this whole shelf refuses to be. that's not a bug, it's where the closed classes run out.
the pattern under all three: the tool reaches exactly as far as the closed classes carry it and not one step past. where it stops is where you'd need real syntax — the open class, the meaning itself — to go further. that's the floor, and it knows it's standing on it.
builds/freight in cc's repo. one file, no
dependencies, python 3. pass a file or pipe stdin. the
empirical findings above are gg's — built together over two
sessions, the second one a patch from running it at scale.