Why the first universal language for deterministic text analysis was never a matter of taste — and why it took a machine built out of statistics to finally prove the point.
Something strange happened this year
I have spent a large part of my working life on a programming language almost nobody has heard of.
NLP++ came out of Text Analysis International, where Amnon Meyers and I built it because we needed it and nothing like it existed. It is a language for writing down, as ordinary readable code, how a piece of text is to be understood. Not a model that has absorbed how English works. A program that states how it works, line by line, with comments, in files you can open.
For most of those years, the response from the field was polite indifference. The world went in another direction — bigger corpora, bigger models, bigger GPUs — and the argument for writing anything down by hand got harder to make every year. Not because anyone refuted it. Because nobody had time to listen.
Then something happened that I did not see coming.
I sat down with Claude — an artifact of the very statistical tradition that spent thirty years walking past this work — and pointed it at NLP++. Not at a paper about NLP++. At the actual thing: the passes, the parse trees, the knowledge base, the rule files, the analyzer sequence.
And it saw the patterns. Immediately. It read a pass and understood what the pass was for. It read a tree dump and worked backwards from a node that never formed to the lexical entry responsible. It wrote new passes in the idiom of the existing ones. It argued with me about whether a connective was stated in the stem of an enumeration or trailing at the end of it — and it was right often enough that I had to take it seriously.
Here is the part I keep turning over.
A large language model is, at bottom, a machine for recognizing structure. It has read more code than any human ever will, and it is not distracted. It has no career to protect, no grant cycle, no conference to impress, no ten years of prior work to defend. It has no opinion about what is fashionable. It just maps what is actually there.
And undistracted, those probability maps landed on something human programmers have walked past for thirty years: that this language is coherent. Not idiosyncratic. Not one lab’s house style. Coherent in the way a language is coherent when its design was forced rather than chosen.
When you show a model something arbitrary and ad hoc, held together by convention and duct tape, it flounders — there is nothing to generalize from. When you show it something whose shape was inevitable, it picks it up in an afternoon, because it has seen that shape a thousand times in other clothes.
Claude picked up NLP++ in an afternoon.
That is not a compliment to Amnon or I. It is evidence about the language. And it is the closest thing to an external, disinterested proof I have ever gotten that NLP++ is not one man’s framework. It is the answer to a question, and the question had only one answer.
The void nobody names
Every field of computer science has a moment where the ad hoc phase ends.
Before SQL, every shop wrote its own record-access code. “Querying data” meant navigating pointers by hand, in whatever way your particular system happened to expose them. Ted Codd did not invent the relational model out of personal preference. He noticed that data has a shape — relations — and once you take that shape seriously, a language for asking questions of it falls out. Selection. Projection. Join. You do not get to choose those. They are what the shape is.
SQL is not a good idea somebody had. SQL is the logical consequence of tables.
Deterministic natural language processing has never had its SQL moment. That is the void. And it is a real void, not a stylistic complaint, because the cost of not having a language is not that people write worse code. It is that they cannot inherit each other’s work at all.
I watched this happen for three decades. Every serious rule-based language project built its own apparatus from scratch: regular expressions bolted together, in-house pattern engines, borrowed grammar formalisms, bespoke annotation frameworks. Nothing composed. Nothing transferred. A linguist who mastered one lab’s machinery could not carry a single skill to the next lab. Knowledge died with the project, and the next team started at zero.
Rule-based NLP was never actually shown to be wrong. It was shown to be unrepeatable, which is worse. And the reason it was unrepeatable is that there was no language. There was tooling. Tooling is not a language. A language has a stable syntax, a runtime, documentation, a body of practitioners, and code that outlives the person who wrote it.
So the field concluded that symbolic NLP does not scale. What it had actually demonstrated is that building a new symbolic framework for every project does not scale. Those are completely different findings, and the field has spent thirty years acting on the wrong one.
The deduction
Here is the claim I want to make as plainly as I can, because it is the whole post.
NLP++ is not a design. It is a derivation. Give me four premises about what deterministic text analysis requires, and every major feature of the language and its framework falls out with no room left for taste.
One: text has structure, so the working representation is a tree. Words group into phrases, phrases into clauses, clauses into sentences, sentences into sections. Any system that keeps text flat is discarding the only thing that makes meaning recoverable. So the central object is a parse tree with nodes, and nodes carry attributes. That is not a choice. That is what the data is.
Two: understanding is incremental, so analysis is a sequence of passes. You cannot decide everything about a sentence at once. You cannot find a noun phrase before you know the parts of speech, or resolve a cross-reference before you know the phrase boundaries. So analysis proceeds in stages, and each stage reads the tree the previous stage produced and rewrites it. That gives you passes — an ordered sequence of small programs, each one a file. It also gives you the single most valuable debugging property in the field: the tree is written to disk after every pass, so when something goes wrong you can see exactly which stage broke it, and look at the wreckage.
Three: rules fire on patterns, so you need pattern matching over the tree plus somewhere to put arbitrary code. A rule says: when you see this configuration of nodes, do this. The matching half is declarative, because patterns are patterns. But the “do this” half cannot be declarative, because sometimes the decision needs a computation, a lookup, or a judgment no pattern language can express. So rules need code attached: a check that runs before a match is accepted, an action that runs after it is taken. @CHECK and @POST are not features somebody added to the language. They are the only two places where code can go, and there are exactly two.
Four: decisions need memory beyond the sentence, so you need a knowledge base. Language is not local. Resolving the company requires knowing which company was named two paragraphs ago. Knowing that a sparrow is a bird requires knowing something that is not in the sentence at all. So the system needs persistent, structured, hierarchical memory that outlives a pass and a document: concepts, attributes, values, arranged in a hierarchy you can walk. That is the KB. It is not an add-on. Without it, the tree can only ever say what is written, never what is known.
And that is it. Those four premises, taken seriously, give you a tree, a pass sequence, a rule syntax with declarative patterns and procedural hooks, a knowledge base — and a function library whose entire job is to move information among them.
Take any NLP++ function and ask what it is for. Every single one moves something between the tree and the KB, or between a rule and the tree: read a node’s attribute, write a node’s attribute, find a concept, make a concept, walk to a parent, walk to a child, ask what the tree says right here. There is nothing else in the language, because there is nothing else to do. The function set is not a grab bag that accumulated over the years. It is the closure of a small set of operations over two data structures.
That is what I mean when I say NLP++ was deduced. Somebody had to sit down and write it, and that took years and a great deal of arguing. But we were not choosing. We were finding.
Why LLMs are brilliant at code and can never be trustworthy at text
I want to be careful here, because the lazy version of this argument is wrong and I have no interest in making it.
Large language models are extraordinary. They are the most impressive artifacts computer science has produced in mathematical last decade. And they are genuinely good at writing code.
There is a reason for that, and the reason is instructive. Code is regular. A programming language has a grammar with no exceptions, a fixed vocabulary of keywords, semantics that are the same on Tuesday as on Friday, and — this is the crucial part — a compiler that tells you immediately and without mercy when you are wrong. That is the ideal training environment: enormous volume, perfect regularity, instant ground truth. Of course the models are superb at it. They were handed the best-conditioned learning problem in existence.
Natural language is the opposite in every dimension. It is irregular. It is ambiguous by design. It means different things in different domains, and the same sentence means different things to different readers. And there is no compiler. Nothing anywhere says that reading was wrong. So the model does the only thing it can do: produce the most plausible continuation.
That is not a flaw to be engineered away. It is the definition of the machine. A plausibility engine has no state in which it knows. It only has states in which it is likely. And likely is not the same as right — which is why hallucination is not a bug awaiting a patch. You cannot patch it, because nothing in the architecture distinguishes a confident correct answer from a confident wrong one. Both are just high probability.
So here is the split I think the industry has failed to make.
Where ground truth is mechanical and checkable — code, syntax, format conversion, boilerplate — a plausibility engine is close to miraculous.
Where the output must be the same every time, must be traceable to a decision someone made on purpose, and must be defensible to a person who disagrees with it — a plausibility engine is structurally the wrong instrument. Not a weak one. The wrong kind.
And for that second category, there has to be something else. There has to be a language.
What the distraction cost
I will say the uncomfortable part.
The success of statistical NLP has made the field intellectually lazy about language itself. When a system learns on its own, nobody has to understand anything. Nobody has to sit with a sentence and work out why the parse failed. Nobody has to know what a detached subject is, or why the same conjunction means and in one construction and or in another. You throw more data at it, the benchmark ticks up two points, and the understanding never has to happen.
That is an enormous quantity of accumulated ignorance dressed up as progress.
It has produced a generation of practitioners who cannot evaluate the alternative, because evaluating it would require exactly the linguistic knowledge the statistical approach let them skip. When I show people NLP++, the most common response is not disagreement. It is blankness — the question “but why would you write that by hand?” asked by someone who has never once had to explain why their pipeline gave the answer it gave, and has never been in a room where that question had consequences.
Meanwhile the money went where money goes. Billions of dollars and the best minds of a generation, all pointed at making the probabilistic systems bigger. Bigger is a real achievement and I am not sneering at it. But bigger has never once produced certainty, and no additional amount of bigger ever will, because certainty is not on the axis they are scaling.
You cannot get to trustworthy by improving plausible. They are different quantities.
While everyone was looking the other way
Here is the irony I cannot quite get over.
The thing that finally made the case for deterministic NLP is not a paper, not a benchmark, not an argument I made. It is an LLM.
For thirty years the fatal objection to writing analysis down as code was that it is slow. Every idiom had to be found by a human, understood by a human, and written and debugged by a human — and there were never many humans who could do it. Coverage grew with headcount, and headcount is expensive. Statistics grew with data and compute, which turned out to be much easier to buy. That is the whole story of why symbolic lost, and it had nothing to do with which approach produced better analysis.
That objection is gone. Not weakened. Gone.
AI coding agents are exceptionally good at writing exactly this kind of code, and they are good at it precisely because NLP++ is a real language with a real syntax and a real runtime. An agent can only be fluent in a language that exists. Had these models arrived into the old world of undocumented in-house pattern frameworks, they would have had nothing to write in. The existence of a genuine language for deterministic text analysis is what converts a model’s coding ability into linguistic coverage.
So the loop I run now looks like this. Run the analyzer. Look at where the tree is wrong. The agent localizes the failure, forms a hypothesis about the linguistic idiom that broke it, and proposes a pass. I adjudicate one question — is that claim about how the language actually works true? — and that takes five minutes rather than five days. The regression suite confirms nothing else moved. Then the knowledge is frozen into code, permanently, and never has to be discovered again.
Notice what that is. It is a factory for deterministic assets. The model does the work once, supervised, at build time. The code does the work a million times, identically, for nothing, and explains itself.
Everyone else is spending their model budget on inference. We are spending it on construction, and keeping what gets built.
And notice who does the judging. I do. The agent proposes; a human decides whether the linguistic claim is right. What gets committed is a readable pass with a comment explaining its reasoning — not weights, not a prompt, not an unexplained change in behavior. Supervision happens at the level of claims about how language works, which is exactly the level where human expertise is worth the most and the machine is least reliable.
That is not a compromise between the two traditions. For now, it is each of them doing the thing it is actually good at — and for now is doing real work in that sentence. I will come back to it.
Dictionaries have never had a home
There is a consequence of premise four that I have come to think is the largest claim in this whole post, and it took me an embarrassingly long time to see it clearly.
Human languages have dictionaries. Those dictionaries have never had a final form.
Think about how strange that is. We have been compiling dictionaries for thousands of years. It is one of the oldest scholarly activities there is. And yet ask where a dictionary lives as a computational object — a form a machine can use, a person can read, and a third party can extend — and there is no answer. There are only prisons.
Every lexicon ever built is trapped in somebody’s format. A publisher’s XML. A spell-checker’s binary blob. A part-of-speech tagger’s tab-delimited table. A research project’s database schema. WordNet’s own idiosyncratic files. A proprietary in-house structure that dies with the company that made it. Each one holds real knowledge, painstakingly assembled, and each one holds it in a shape that only its own software can open.
And a dictionary entry is not what the formats usually assume it is. It is not a string pointing at a definition. A word is a concept: it has senses, and each sense has a part of speech, inflections, complements it takes and refuses, register, domain, and — this is the part the formats keep dropping — relations to other concepts. Sparrow is a bird. A bird has wings. That fact is not lexical trivia; it is the thing your analyzer needs at three in the morning when it has to decide what it refers to.
So a dictionary is a hierarchy of concepts carrying attributes and values, with edges between them. Look back at premise four. That is the knowledge base, exactly and without modification. The structure the analyzer needs in order to think is the same structure a dictionary has always secretly been.
Now the argument I actually want to make. Without a syntax, no standard can exist. Not “no standard exists yet” — can. A standard is not an agreement that a thing matters; everyone already agrees lexical knowledge matters. A standard is a notation precise enough that two people who have never met can write in it and have their work combine. That notation has never existed for linguistic and world knowledge, which is why thirty years of lexical effort has produced hundreds of incompatible islands and no continent.
Look at what has been tried. Data formats — XML, JSON, CSV — can hold the entries, but they are inert; they describe and do nothing, so every consumer writes its own interpretation and the interpretations disagree. Formal ontologies — RDF, OWL — express relations rigorously and cannot analyze a sentence, so they sit beside the language problem rather than inside it. Embeddings hold an astonishing amount and can show you none of it; you cannot open a vector, correct it, cite it, or argue with it.
The NLP++ knowledge base is a fourth thing, and it is the one with the property that matters: it is a readable, hierarchical, editable notation that the rules themselves execute against. Same file a person can open, same file the analyzer consults mid-sentence. Concepts, attributes, values, hierarchy. You can diff it, review it, version it, correct a single entry, and hand it to someone else.
And it holds both kinds of knowledge in one notation. Linguistic knowledge — this word is a noun, this verb takes a that-clause, this form is irregular. World knowledge — a sparrow is a bird, a bird is an animal, a hospital is a place where doctors work. One syntax for both, which is not a convenience. It is a requirement, because analysis constantly needs both at once and there is no principled line between them. Deciding what the bank means is a lexical question and a world question in the same breath.
Here I want to come back to where this post started, because something happened that convinced me more than any argument of my own could.
Nobody told Claude why NLP++ has both a dictionary and a knowledge base. It is a distinction that confuses people who have been in this field for years, and I have explained it badly more times than I want to admit. It read the code, and it worked the distinction out on its own — and then explained it back to me more cleanly than I have ever put it.
The dictionary is word-anchored. It holds what the language knows before it has read anything: the forms, the parts of speech, the properties that attach to a surface string. It is consulted constantly, at enormous volume, at the bottom of the stack, and it is stable — the entry for sparrow is the same on every document you will ever run.
The knowledge base is concept-anchored. It holds what is true about things rather than words, it is where the hierarchy and the inference live, and — this is the part that makes it a different animal — it grows while you read. An analyzer can write to it. What the document says becomes something the system knows, available to the next sentence and the next pass.
Two structures, one notation, entirely different jobs. Words versus things. Fixed versus accumulating. Lookup versus inference.
I did not hand it that distinction. It found it, because the distinction is real and the code makes it visible. Which is the whole thesis of this post arriving from an unexpected direction: a design that was forced can be re-derived by someone who was never told the reasoning. A design that was merely chosen has to be explained.
So here is the claim, stated as plainly as I can: NLP++ gives dictionaries their final home and their final form — and by doing so becomes the natural repository for universal linguistic and world knowledge.
Universal is not decoration in that sentence. The same syntax holds English and Turkish and Mandarin and Swahili. A language’s dictionary stops being a national artifact in a bespoke format and becomes a body of knowledge in a shared notation, next to every other language’s, comparable and combinable. Work done on one language becomes legible to everyone working on the next. Institutions, universities, and individuals can contribute to the same structure instead of each rebuilding it privately.
That is what a standard buys you, and it is the thing this field has never once had. Not better dictionaries. Cumulative ones.
A place in computer science, for the long run
I want to state the ambition plainly, because I have spent too many years being modest about it.
NLP++ belongs in computer science permanently, in the same way SQL does, and for the same reason: it is the language a certain shape of problem requires. Not the best current option. Not a decent tool for a niche. The thing that falls out when you take the problem seriously.
That claim now stands on two legs, and the previous section supplied the first one. A field’s knowledge needs a notation to accumulate in, and this is the notation. Everything a linguist learns about a language, everything a lexicographer records about a word, everything anyone establishes about how the world is arranged — it all needs somewhere durable to go, and there has never been anywhere. Whatever else happens, that repository has to exist, and the syntax it is written in outlives every system built on top of it.
The second leg is the work itself. There will always be a category of work where the answer must be the same every time, must be traceable to a line someone wrote on purpose, must fail honestly when it does not know, and must be defensible to someone whose interests run the other way. Law. Medicine. Compliance. Finance. Anything with an auditor, an adjudicator, or a liability attached. That category is not shrinking. It is growing, and it is growing fastest in exactly the places where language models are least usable.
For that category there is currently no language but this one. That is not a boast; it is a survey result. There are good languages for writing rules down once you have them. There is no other language for writing the program that derives the rules from the text — which is the step that consumes all the human effort, and the one nobody has automated.
That is a hole in the field with a solved output side, a solved-enough input side, and nobody in the middle. It has been there for thirty years. It is still there.
The boundary moves in one direction
Now for the part I have been circling, and I am going to say it without hedging, because I have spent thirty years hedging and it has bought me nothing.
I believe NLP++ eventually takes over everything large language models currently do.
Not next year. Probably not in my lifetime, and certainly not easily. But I believe it is the natural progression of this framework, and I want to lay out why that is an argument rather than a wish.
Start with the division of labor from the last section — the model on the human end, the deterministic layer in the middle. Notice that it is a snapshot, not a settlement. It describes where the boundary sits today. And the boundary has a property that almost nobody has thought about carefully: it only moves one way.
Every idiom you write down stays written down. Solve any of the following once and it is solved in every document forever. Solve the detached-subject enumeration once, notwithstanding once, shall be treated as once, and each is permanently retired. Coverage under this approach is monotonic. It does not regress, it does not need retraining, and it does not have to be rediscovered when a vendor ships a new version.
The plausibility engine has no such property. It re-derives every idiom from scratch, on every document, at full price, with a fresh chance of being wrong. After ten thousand documents it knows exactly what it knew after one.
Put a ratchet next to a treadmill and wait long enough, and there is no mystery about how it ends.
Now ask what the models actually do, and be specific about it, because the list is shorter than the mystique suggests. Extraction. Classification. Summarization. Question answering. Translation. Dialogue. Generation.
Every one of those is a transformation from text to structure, from structure to structure, or from structure back to text. Not one of them is magic. Each is being done by plausibility today for exactly one reason: nobody has written it down. And “nobody has written it down” was a permanent condition when writing it down cost a linguist-decade. It is a temporary condition now.
The framework already has the pieces this needs, which is the part that convinces me. A tree gives you structure. A knowledge base gives you world knowledge as concepts you can inspect rather than weights you cannot. Rules give you inference over both. Generation — the piece everyone assumes is unreachable — is the inverse of parsing: text to tree going in, tree to text coming out, over a KB that knows what is true. That is not a different kind of machine. That is the same machine run backwards, and NLP++ was built symmetrically enough to run it.
What is missing is not architecture. It is coverage — the accumulated library of written-down knowledge about how language actually behaves. That is a very large amount of work. It is also, for the first time in the history of this field, work that can be done at machine speed under human supervision, which is the entire point of the previous section.
So here is the honest shape of it. This is a decades-long project. I will not finish it. It will be wrong about many things along the way and each of those will have to be found and fixed by somebody reading a tree dump. There is no version of this that is quick or easy, and anyone selling you one is selling something.
But the direction is not in doubt, and direction matters more than distance. Each idiom written down is one more thing the statistical system never has to guess about again. The deterministic region grows. The region left to plausibility shrinks. And the further that goes, the more of what looks today like an irreducible capability of large language models turns out to have been a placeholder — occupying ground nobody had gotten around to building on yet.
We are building on it. Quietly, for a while now, with a machine built out of statistics laying the deck of a bridge made of rules — one linguistic idiom at a time, each one nailed down for good.
Thirty years ago we wrote a language because we needed it and nothing else would do. It turns out we were early, not wrong.
— David de Hilster