A computer never sees a word. It sees a number standing in for a word, and a number carries no meaning at all. The whole adventure of natural language processing is the slow, clever business of smuggling meaning back into those numbers — first by counting which words keep each other company, then by learning dense vectors that place "king" near "queen," then by teaching networks to predict the next word, and finally by tagging every word with its grammatical role so a machine can act on what you said. This is that story, told as one continuous arc: counts to embeddings to neural nets to pretrained models, with the real math and the intuition that makes it click.
Type the word bank into a program and ask it what that means. The honest answer is that the program has no idea, and it never will from the letters alone. b-a-n-k is five symbols. They are the same five symbols whether you meant the place that holds your money or the muddy edge of a river, and they share nothing — not a single bit — with sofa or couch, two words that mean almost exactly the same thing. Meaning is simply not sitting inside the string, waiting to be read off. That is the uncomfortable truth every NLP system has to start from, and everything in this article is, in one way or another, a trick for coping with it.
So here is the question that organizes the whole field: if meaning is not in the word itself, where is it, and how do we get a machine to find it? The answers turn out to form a beautiful staircase. We will climb it one step at a time.
The core problem: meaning is not in the string
Picture how the simplest possible language program represents a vocabulary. You make a list of every word you know — a, aardvark, abacus, …, zulu — and you give each word the number of its position in the list. Now "dog" might be word number 4914 and "cat" word number 7157. This is exactly how classic text-classification and counting methods treat words: as indices, bare integers pointing into a vocabulary.
It is clean, it is fast, and it is meaning-blind in a way that should bother you. The number 4914 is not closer to 7157 than it is to 2 or to 40,000. Dogs and cats are both small furry mammals that live in our homes, yet their indices are as unrelated as two random lottery tickets. If you one-hot encode them instead — a giant vector that is all zeros except a single 1 in the word's slot — the situation is no better. Every word vector is exactly the same distance from every other word vector. "King" and "queen" are orthogonal. "Good" and "great" are orthogonal. The representation has thrown away the one thing we actually care about: which words mean similar things.
Old logic classes used to make this problem feel even more hopeless with a famous little joke. Define the meaning of "dog" as the symbol DOG, with the rule that everything that is a dog is a mammal. Fine. But what is the meaning of DOG? Just another symbol. As Barbara Partee quipped: the meaning of life is … LIFE. Defining a word's meaning by handing it a new symbol explains nothing. We have only relabeled the mystery.
The escape from this trap is one of the most productive ideas in the history of linguistics, and it is almost suspiciously simple. It is called the distributional hypothesis, and the linguist J.R. Firth gave it its slogan: "You shall know a word by the company it keeps."
The idea is this. Suppose you have never seen the word tezgüino. I show you a few sentences: "A bottle of tezgüino is on the table." "Everybody likes tezgüino." "Tezgüino makes you drunk." "We make tezgüino out of corn." You now have a very confident guess: tezgüino is some kind of alcoholic, corn-based drink. You learned that purely from the company the word kept — the contexts it showed up in — without anyone ever defining it. Two words that appear in very similar contexts (whose neighboring words overlap a lot) tend to mean similar things. Oculist and eye-doctor both show up near "eye" and "examined," and lo, they are synonyms. The amount of meaning difference between two words corresponds, roughly, to the amount of difference in their environments.
That reframing is the hinge. Once meaning becomes a point in space, the whole toolbox of geometry opens up: distances, angles, directions, clusters. A picture of words stops being a list of strings and becomes a landscape you can navigate. Everything that follows is operations in that space.
Words as vectors: counting the company they keep
If meaning lives in context, the most literal thing we can do is just count contexts. There are two classic ways to set up that count, and they differ only in what we treat as a "context."
The term-document matrix
The first idea comes from information retrieval. Take a collection of documents — say four of Shakespeare's plays — and build a big table. Each row is a word from the vocabulary; each column is a document. The cell at row w, column d holds the number of times word w appears in document d. Now each word is described by a row: a vector whose entries say how often it occurred in each document. Words that appear in the same plays get similar rows. "Battle" and "soldier" might both be frequent in the histories and rare in the comedies, so their vectors point in similar directions. As a bonus, each document is also a vector now (a column), which is exactly how a search engine decides which documents match your query: it represents the query as a vector and finds documents whose vectors are nearby.
The word-context (co-occurrence) matrix
For word meaning specifically, documents are too coarse. A whole play is a clumsy notion of "context." So we shrink the context down to a small window of nearby words — say, the four words to the left and four to the right of each occurrence. Now we build a word-context matrix (also called a term-term or co-occurrence matrix). It is square-ish: rows are target words, columns are context words, and the cell counts how many times the context word appeared within the window of the target word across the whole corpus.
Fig 1 — A word becomes the vector of counts of the words around it. Words used alike (apricot, peach) point in nearly the same direction; the small angle between them is their similarity.
The vector for a word is now its row in this matrix: a long list of co-occurrence counts. "Apricot" and "pineapple" will have high counts in the columns for "juice," "sugar," "delicious" and "ripe," and low counts everywhere else — so their vectors will look alike. "Digital" and "information" will share a different cluster of columns. The geometry has captured the gist.
There is a catch baked into this representation, and it shapes everything we do next. These vectors are sparse and long. The vocabulary of a real corpus might be 50,000 words, so each vector has 50,000 dimensions, and the overwhelming majority of those entries are zero, because any given word only ever co-occurs with a small slice of the vocabulary. Long, sparse, mostly-zero vectors are wasteful to store and, as we will see, they also miss similarities that a denser representation would catch. Hold that thought — it is the motivation for embeddings later.
Weighting counts: not all co-occurrences are equal
Raw counts have a sneaky problem: the most frequent words are the least informative. The word "the" co-occurs with everything. So does "of," "a," and "it." If you trust raw counts, these function words dominate every vector and drown out the signal. The fact that "apricot" appears near "the" tells you nothing; the fact that it appears near "ripe" tells you a lot. We need to downweight the words that are common everywhere and upweight the words that are distinctive. There are two classic recipes.
TF-IDF: rare words carry more news
In the term-document world, the weapon of choice is TF-IDF: term frequency times inverse document frequency. The intuition is that a word matters in a document if it appears often in that document (high term frequency) but rarely across all documents (high inverse document frequency, meaning it is distinctive rather than universal).
Here \( tf(w,d) \) is how often word \( w \) shows up in document \( d \) (often squashed with a log, since the difference between 1 and 2 occurrences matters more than between 100 and 101). The term \( N \) is the total number of documents, and \( df(w) \) is the document frequency — the number of documents that contain \( w \) at all. When a word appears in every document, \( df(w) = N \), so \( \log(N/df) = \log 1 = 0 \) and the word is zeroed out entirely. "The" appears everywhere, its IDF is zero, and it stops polluting the vectors. A word like "Romeo," which appears in one play and nowhere else, gets a large IDF and shines. That asymmetry — common words muted, distinctive words amplified — is the whole point.
PPMI: do these two words like each other more than chance?
In the word-context world, the better tool is pointwise mutual information. It asks a sharper question than "how often do w and c co-occur?" It asks: do they co-occur more often than we would expect if they were independent?
Read the fraction carefully. The numerator \( P(w,c) \) is how often the two actually appear together. The denominator \( P(w)\,P(c) \) is how often they would appear together by pure coincidence if each word were sprinkled into the corpus at random. If the two like each other — if they show up together more than chance — the ratio is greater than 1 and the log is positive. If they avoid each other, the ratio is below 1 and the log is negative. This automatically handles the "the" problem: "the" is so common that it co-occurs with everything at roughly the chance rate, so its PMI with any word is near zero.
In practice we use positive PMI (PPMI): we clamp all the negative values to zero. The reason is humility about data. A negative PMI claims "these two words actively avoid each other," but to be confident of that you would need to have seen both words a great many times and confirmed they rarely meet. With a finite corpus, negative values are mostly noise — unreliable estimates of rare non-events — so we throw them away and keep only the trustworthy positive associations.
Measuring similarity: the angle, not the length
We keep saying two words are "similar" if their vectors are "alike." Time to make that precise. The obvious first guess is the dot product: multiply the two vectors entry by entry and sum. \( a \cdot b = \sum_i a_i b_i \). It is large when the two vectors have big values in the same positions, which is exactly the overlap we want. So why not use it directly?
Because the dot product is biased by length. A vector's length (its magnitude) grows with how often a word occurs — a very frequent word racks up big counts everywhere, so it has a long vector, so its dot product with anything tends to be large just from sheer size, not from genuine similarity. If we ranked words by raw dot product, frequent words would look "similar" to everything, which is precisely the bias we were trying to escape.
The fix is to divide the length out and keep only the direction. That is exactly what cosine similarity does: it is the dot product normalized by both vectors' magnitudes, which is the cosine of the angle between them.
Because we are dividing by the lengths, a word's overall frequency cancels out. What remains is purely the pattern of contexts. Two vectors pointing in the same direction have an angle of 0, a cosine of 1 — maximally similar. Two vectors at right angles share no contexts, cosine 0 — unrelated. (For count and TF-IDF vectors, entries are non-negative, so the cosine lives between 0 and 1; for embeddings with negative entries it can dip to −1 for opposites.) The angle, not the length, is the meaning. Look back at Fig 1: "apricot" and "peach" sit at a tiny angle apart even if one is far more frequent than the other, because frequency stretches the arrow but does not rotate it.
Dense embeddings: short vectors that generalize
The count-based vectors work, but remember their flaw: they are long (one dimension per vocabulary word) and sparse (almost all zeros). Worse, they have a subtle blind spot. In a sparse vector, the dimension for "car" and the dimension for "automobile" are different, unrelated columns. A word that co-occurs with "car" gets a count in one slot; a word that co-occurs with "automobile" gets a count in a totally different slot. The representation has no way to know those two slots mean nearly the same thing, so it can fail to notice that two words are similar simply because one hangs around "car" and the other around "automobile." The sparsity itself hides synonymy.
What we want instead are dense embeddings: short vectors, maybe 100 to 300 dimensions, where every entry is a real number doing real work. These are the "embeddings" that every modern NLP system runs on. The name comes from the geometry: each word is embedded as a point in a continuous space, and we have folded the meaning down from 50,000 sparse dimensions into a few hundred dense ones.
Why bother? Three reasons, and the third is the one that matters most.
- Fewer parameters. A 300-dimensional feature vector is far easier for a downstream classifier to learn from than a 50,000-dimensional one — fewer weights to tune, less overfitting.
- They capture synonymy. Because the dimensions are learned rather than tied to specific words, "car" and "automobile" can end up with nearly identical vectors. The synonymy that sparse vectors hid in separate columns is now visible as nearness in space.
- They generalize to unseen words. This is the killer feature. Suppose you train a sentiment classifier and one informative feature is "the previous word was terrible." With word identities, that feature only fires on the exact string "terrible." But with embeddings, the feature becomes "the previous word had vector [35, 22, 17, …]." At test time you encounter "awful," whose vector is [34, 21, 14, …] — almost the same. The classifier has never seen "awful" in training, yet the similar vector lets it transfer everything it learned about "terrible." Meaning generalizes through geometry.
Word2Vec: learn meaning by playing a prediction game
So how do we get these dense vectors? The breakthrough, from Mikolov and colleagues, was to stop counting and start predicting. The trick is delightfully sneaky. We set up a fake task — one we do not actually care about — and the vectors we need fall out as a side effect of getting good at it.
The fake task: given a word, predict the words that appear around it. This is the skip-gram model. Run a sliding window through the corpus. At each position, take the center word (say "apricot") and try to predict each of its neighbors ("a," "glass," "of," "juice"). The genius is that the labels are free — the corpus itself tells us what the real neighbors were, so no human annotation is needed. This is self-supervision: the data supervises itself.
Fig 2 — Skip-gram: the center word's vector is nudged, over millions of windows, to predict the words that surround it. The vector that learns to do this well is the embedding.
Mechanically, each word gets two vectors: a "target" vector \( v_w \) for when it is the center word, and a "context" vector \( u_c \) for when it is a neighbor. The model scores how likely context word \( c \) is, given center word \( w \), with a dot product passed through a softmax that turns scores into probabilities:
The dot product \( u_c \cdot v_w \) is big when the two vectors align, so training pushes the vectors of words that genuinely co-occur to point the same way. The denominator sums over the entire vocabulary \( V \) — every possible context word — to normalize the scores into a proper probability distribution. We learn the vectors by nudging them, example after example, to make the real neighbors more probable and everything else less probable. After a few passes over a big corpus, the \( v_w \) vectors are the embeddings we keep; the prediction task itself is discarded like scaffolding.
There is also a mirror-image version called CBOW (continuous bag of words), which flips the task around: instead of predicting context from the center word, it predicts the center word from the average of its context vectors. Skip-gram tends to do better on rare words; CBOW is a touch faster. Same philosophy, opposite direction.
The negative-sampling trick
Look again at that softmax denominator: it sums over every word in the vocabulary. With 50,000 words, computing it — and its gradient — for every single training window is murderously expensive. We would be doing tens of thousands of operations just to update one example, millions of times over. The full softmax is a non-starter at scale.
The fix that made Word2Vec practical is negative sampling. Instead of asking "out of all 50,000 words, which is the right neighbor?", we ask a much cheaper yes/no question: "is this specific pair (center word, candidate word) a real co-occurrence or a fake one?" For each true (center, neighbor) pair we pull from the corpus, we also conjure up a handful — maybe 5 to 20 — of negative pairs by picking random words that did not appear in the window. Training then just nudges the model to say "yes, real" to the true pair and "no, fake" to the random ones, using a cheap sigmoid on each pair rather than a giant normalized softmax. We have replaced one impossible question over the whole vocabulary with a few trivial binary questions. The embeddings come out essentially as good, at a tiny fraction of the cost.
GloVe and the famous analogy trick
A close cousin, GloVe (from Pennington, Socher and Manning), reaches similar embeddings from the other side: rather than scanning local windows one at a time, it builds the global co-occurrence counts first and then factorizes them, fitting vectors whose dot products match the (log) co-occurrence statistics. Counting and predicting, it turns out, are two faces of the same coin.
What makes these embeddings feel almost magical is that the geometry encodes relationships, not just similarity. Directions in the space turn out to be meaningful. The classic demonstration: take the vector for "king," subtract "man," add "woman," and the nearest word to the result is "queen."
The vector that points from "man" to "woman" is, roughly, the same vector that points from "king" to "queen" — a consistent "gender" direction baked into the space by the data. Similarly there are directions that capture singular-to-plural, country-to-capital, and comparative-to-superlative. Nobody designed these axes; they emerged from the prediction game. That is the moment most people fall in love with embeddings.
Language modelling: scoring sequences of words
Embeddings give meaning to individual words. But language is words in sequence, and a huge amount of NLP rests on a different question: how probable is this sequence of words? A model that can assign a probability to a sentence — or, equivalently, predict the next word given the words so far — is a language model, and it is one of the most useful objects in the field.
Why is "which sentence is more probable" so valuable? Because it silently powers a dozen applications. A machine translation system produces several candidate English sentences and picks the one a language model judges most natural: \( P(\text{high winds tonight}) > P(\text{large winds tonight}) \). A spell-checker prefers "about fifteen minutes from now" over "about fifteen minuets from now." Speech recognition, hearing an ambiguous acoustic blur, chooses \( P(\text{I saw a van}) \gg P(\text{eyes awe of an}) \). Your phone's autocomplete is a language model guessing the next word. In every case the model is ranking sequences by plausibility.
The chain rule and the combinatorial wall
The probability of a whole sentence decomposes exactly, with no approximation, via the chain rule of probability. The joint probability of a sequence of words is the product of each word's probability given everything before it:
So \( P(\text{its water is so transparent}) \) equals \( P(\text{its}) \times P(\text{water} \mid \text{its}) \times P(\text{is} \mid \text{its water}) \times \cdots \). This is exactly right, but it is also useless as written. To estimate \( P(\text{transparent} \mid \text{its water is so}) \) by counting, we would need to have seen the exact phrase "its water is so" many times in our data and tallied what followed. Most long histories appear zero or one times in any corpus, no matter how big. The number of possible histories explodes combinatorially. We need to approximate.
The Markov approximation: forget the distant past
The crucial simplifying assumption, due to Andrei Markov, is that the recent past is good enough. Instead of conditioning on the entire history, condition only on the last few words. An n-gram model keeps a window of \( n-1 \) previous words. A bigram model (\( n = 2 \)) keeps just one:
So a bigram model approximates "its water is so transparent" as \( P(\text{water} \mid \text{its}) \times P(\text{is} \mid \text{water}) \times P(\text{so} \mid \text{is}) \times \cdots \). A trigram keeps the last two words, and so on. This is obviously a lie — language has long-distance dependencies ("the computers which I had just put into the machine room on the fifth floor are crashing," where "are" agrees with "computers" eight words back) — but it is a productive lie, because short contexts actually do appear often enough to estimate.
And estimating them is now easy. With a short context, we can just count. The maximum-likelihood estimate of a bigram probability is the number of times the pair occurred divided by the number of times the first word occurred:
If "want" appeared 927 times and "want to" appeared 608 of those, then \( P(\text{to} \mid \text{want}) = 608/927 \approx 0.66 \). Count the pairs, count the singles, divide. That is a working language model.
Why we live in log space
One practical wrinkle. A sentence probability is a product of many numbers, each smaller than 1. Multiply twenty or thirty probabilities of, say, 0.01 each and you get a number so tiny it underflows to zero in floating-point arithmetic — the computer literally cannot represent it. The standard fix is to work with log probabilities and add instead of multiply, since \( \log(ab) = \log a + \log b \). Sums of logs stay in a comfortable numeric range, and because log is monotonic, the sentence with the highest log-probability is still the most probable sentence. Adding logs also happens to be faster than multiplying. Essentially every real language model computes in log space.
The zero-probability problem and smoothing
The maximum-likelihood count estimate has a catastrophic failure mode, and it is worth feeling the full weight of it. Suppose the pair "want Chinese" never appeared in your training corpus — not because it is impossible, but because your data was finite and you just never happened to see it. Then \( C(\text{want Chinese}) = 0 \), so \( P(\text{Chinese} \mid \text{want}) = 0 \). And because the sentence probability is a product, a single zero anywhere annihilates the whole thing: the model declares an entirely reasonable sentence to have probability zero, impossible, forbidden. One unseen pair and the model is certain the sentence can never occur. That is absurd, and it is fatal.
The family of fixes is called smoothing, and the metaphor is exactly right: we shave a little probability mass off the events we did see and spread it over the events we did not, so nothing is ever flatly impossible.
Add-one (Laplace) smoothing
The simplest move: pretend you saw every possible n-gram one extra time before you started counting. Add 1 to every count. To keep the probabilities summing to 1, you also add the vocabulary size \( V \) to the denominator (one extra phantom count for each possible next word):
Now nothing is zero — even an unseen pair gets a small \( 1/(C(w_{n-1})+V) \). Add-one is crude (it steals a surprising amount of mass from the common events and hands it to the vast sea of unseen ones), but it captures the essential idea and it never breaks. More refined variants add a fraction \( k < 1 \) instead of a whole 1.
Backoff and interpolation: ask a shorter question
There is a smarter intuition. If you have never seen the trigram "want Chinese food," maybe you have seen the bigram "Chinese food." When the long context fails you, fall back to a shorter, better-attested one. That is backoff: use the trigram estimate if you have enough data for it; otherwise back off to the bigram; otherwise to the unigram. You consult the most specific evidence you can trust, and retreat to vaguer evidence only when the specific evidence runs dry.
Interpolation is the gentler, usually-better sibling. Rather than switching abruptly from trigram to bigram, it always blends all the orders together with weights that sum to one — a pinch of trigram, a pinch of bigram, a pinch of unigram — so the estimate is a weighted mixture of contexts of every length. The weights are tuned on held-out data. Where backoff is a fallback staircase, interpolation is a smooth blend; both encode the same wisdom that shorter contexts are a safety net for longer ones.
Perplexity: how surprised is the model?
How do we tell whether one language model is better than another? We measure how well it predicts a held-out test set it never trained on. A good model assigns high probability to real, natural text. The standard metric is perplexity, which is the inverse probability of the test set, normalized by the number of words:
The minus sign and the inverse flip the logic so that lower is better: a model that assigns high probability to the test text gets low perplexity. There is a lovely intuitive reading. Perplexity is the weighted average branching factor — roughly, how many words the model thinks could plausibly come next at each step, on average. A perplexity of 100 means the model is, on average, as confused as if it had to choose uniformly among 100 equally likely next words. A perplexity of 20 means it has narrowed the field to about 20. So a low perplexity says "the model is rarely surprised by real text; it had a good guess about what came next," and a high perplexity says "the model is constantly caught off guard." Better language models have lower perplexity, full stop.
The neural turn: from counting to learning
N-gram models hit a ceiling. They are built from discrete counts, so they suffer the sparsity we keep running into, and they can only ever see a few words back. To break through, the field turned to neural networks — and the right way to understand them is to build one up from a single cell.
The neuron: a weighted sum with attitude
A single artificial neuron does something almost embarrassingly simple. It takes a vector of inputs \( x \), multiplies each input by a learned weight, adds them all up, tosses in a learned bias term \( b \), and then passes the result through a nonlinear function. The weighted sum is \( z = \sum_i w_i x_i + b \), or compactly \( z = w \cdot x + b \). The bias lets the neuron shift its threshold; the weights say how much each input matters.
Fig 3 — A single unit: scale each input by a weight, sum, add a bias, then squash through a nonlinear function \( f \). Stack thousands of these and you have a network.
Why the nonlinearity? Without it, a neuron is just a linear function, and stacking linear functions only ever gives you another linear function — no matter how many layers, you could collapse the whole thing into a single line. The nonlinear activation function \( f \) is what lets networks bend and fold the input space into something expressive. Three are ubiquitous. The sigmoid squashes any number into \( (0,1) \), an S-curve handy for probabilities. The tanh does the same but into \( (-1,1) \), centered at zero. And the ReLU (rectified linear unit) simply returns \( \max(0, z) \) — zero for negatives, identity for positives — which is cheap, avoids some training headaches, and is the modern default.
The XOR problem: why one layer is not enough
Here is the historical jolt that proved we need depth. A single neuron with no nonlinearity — a perceptron — computes \( w_1 x_1 + w_2 x_2 + b \) and fires if that is positive. Geometrically, \( w_1 x_1 + w_2 x_2 + b = 0 \) is the equation of a straight line, a decision boundary that splits the plane into a "fire" side and a "don't fire" side. A single perceptron can only carve the input space with one straight cut.
That is enough for AND and OR — you can separate their true and false cases with one line. But XOR (exclusive or: true when exactly one input is true) cannot be split by any single straight line. Its true cases sit on opposite diagonal corners, and no line puts both trues on one side and both falses on the other. Minsky and Papert pointed this out in 1969 and the field briefly despaired. The resolution: stack a hidden layer of neurons between input and output. The hidden layer transforms the inputs into a new space where the problem becomes linearly separable, and then the output neuron draws its line there. Multiple layers with nonlinear activations can represent any function. Depth is not a luxury; for XOR-like problems it is the whole point.
Feedforward nets on text, and the neural language model
To run a feedforward network on text, we feed it word embeddings — those dense vectors from earlier — instead of raw word identities. The network takes the vectors for a window of words, passes them through one or more hidden layers, and produces an output: a sentiment label, a topic, or a probability distribution over the next word. Because the inputs are dense embeddings, the network automatically generalizes across similar words, inheriting all the benefits we discussed.
A neural language model is exactly this idea pointed at the next-word task. Where an n-gram model stored a giant sparse table of counts, the neural version takes the embeddings of the previous few words, runs them through hidden layers, and outputs a probability distribution over the whole vocabulary for what comes next. It never stores explicit counts; it learns weights that compute the probabilities. This fixes the n-gram model's two big weaknesses at once: sparsity (similar contexts share statistical strength through their embeddings) and brittleness (an unseen-but-similar history still gets a sensible prediction). This is the conceptual seed that grows into recurrent networks, then transformers, then the large language models running today.
How a network learns, in one paragraph
Training is conceptually a feedback loop. Define a loss function that measures how wrong the network's output is — for a language model, how much probability it wasted on the wrong next word. Then ask: which way should I nudge each weight to make the loss a little smaller? That direction is the negative gradient of the loss with respect to the weights, and computing it efficiently across all the layers is what backpropagation does — it applies the chain rule of calculus to push the error signal backward from the output through every layer. We then take a small step downhill (gradient descent), repeat over millions of examples, and the weights gradually settle into values that predict well. Loss, gradient, step, repeat — that is the entire engine, and the same loop trains everything from a two-layer net to a frontier model.
Part-of-speech tagging: giving every word a job
Now let us point all of this machinery at a concrete, classic task that shows off the whole pipeline: part-of-speech tagging. The goal is to label each word in a sentence with its grammatical category — noun, verb, adjective, determiner, preposition, and so on. "The koala put the keys on the table" becomes "The/DET koala/N put/V the/DET keys/N on/P the/DET table/N."
This sounds like a dry grammar-school exercise, but it quietly underpins a lot of useful systems. A voice assistant needs to know which word is the action (verb) and which is the target (noun) to do what you asked. A parser needs parts of speech before it can build sentence structure. Consider an agentic assistant hearing "Set the timer for the oven to 10 minutes." If it correctly tags "set" as the verb-action, "timer" as the noun-object, and "for the oven" as a prepositional phrase giving context, it confidently calls the timer API. If it fumbles the structure, it might think you want to set the oven — turn on a heating element — which is genuinely dangerous. POS tagging makes downstream actions more deterministic and safer by pinning down who-does-what before anything irreversible happens. It also feeds search-query understanding ("play cricket rules" versus "cricket playstation game"), information extraction, grammar checking, and machine translation, where word order depends on grammatical role.
Ambiguity: the reason it's hard
If every word had exactly one part of speech, tagging would be a dictionary lookup. The difficulty is ambiguity: the same word can be different parts of speech in different contexts. "Book" is a noun in "read a book" but a verb in "book a flight." "Back" can be a noun (your back), a verb (back the car), an adjective (the back door), or an adverb (go back). To tag correctly you cannot look at a word in isolation; you must use its context — the words and tags around it. That is precisely what makes it a sequence problem rather than a per-word problem.
The Hidden Markov Model
The classic statistical solution is the Hidden Markov Model. The name captures the picture: the parts of speech are "hidden" states we cannot observe directly; all we see are the words they emitted. Tagging is the job of inferring the hidden tag sequence from the visible word sequence. An HMM is built from two kinds of probabilities, both estimable by counting in a tagged corpus.
- Transition probabilities — how likely one tag is to follow another, \( P(\text{tag}_i \mid \text{tag}_{i-1}) \). For example, a determiner is very often followed by a noun, so \( P(\text{N} \mid \text{DET}) \) is high; a determiner is almost never followed by a verb. This is a tag-level bigram model, capturing the grammar's local regularities.
- Emission probabilities — how likely a given tag is to produce a given word, \( P(\text{word} \mid \text{tag}) \). The tag VB emits "run," "eat," "go" with some probabilities; the tag NN emits "dog," "table," "idea." This captures each word's lexical preferences.
The probability the HMM assigns to a tag sequence paired with the observed words is the product of all the transitions and all the emissions. To tag a sentence we want the single tag sequence that maximizes this product — the most probable explanation of the words we saw.
Viterbi: finding the best path without trying them all
Here is the combinatorial trap again. If there are \( T \) possible tags and \( N \) words, there are \( T^N \) possible tag sequences — astronomically many for any real sentence. We cannot score them all. The Viterbi algorithm finds the single best one efficiently using dynamic programming, and the intuition is worth savoring.
Lay the problem out as a grid — a trellis — with one column per word and one row per possible tag. Each cell (tag \( t \), word \( i \)) will store one number: the score of the best possible tag-path that ends in tag \( t \) at word \( i \). The key realization is that to compute the best path reaching a cell, you do not need to know the whole history — you only need the best scores from the previous column. The best way to arrive at "N for word 3" is: look at every cell in column 2, take its best-path score, multiply by the transition into N and the emission of word 3 as an N, and keep the maximum. You also remember which previous cell won, with a back-pointer.
Fig 4 — The Viterbi trellis. Each column is a word, each row a candidate tag. Dynamic programming keeps only the best path into each cell; the highlighted route is the highest-scoring tag sequence overall.
So we sweep left to right, filling one column at a time, each cell reusing the answers already computed in the column before it. By the last word, the highest-scoring cell tells us the best final tag, and we follow the back-pointers leftward to recover the whole winning sequence. We have found the best of \( T^N \) paths while doing work proportional to only \( N \cdot T^2 \) — the exponential collapses to something linear in sentence length. That reuse of overlapping subproblems is the essence of dynamic programming, and Viterbi is its most beautiful NLP appearance.
MEMMs and the limits of looking one way
HMMs have a known weakness: their emission model makes it awkward to throw in rich features about a word (its suffix, capitalization, the word two positions over). Maximum-entropy Markov models (MEMMs) address this by directly modeling \( P(\text{tag} \mid \text{word and context features}) \) with a flexible log-linear classifier, letting you pour in as many features as you like. But many of these models still march strictly left to right, so when they tag a word they have only seen the left context. Often the disambiguating clue sits to the right. Without true bidirectionality, the model can commit to a tag before the evidence that would have corrected it arrives. That limitation is exactly what the neural sequence taggers were built to fix.
Neural sequence tagging: Bi-LSTM-CRF and beyond
The modern statistical baseline for tagging is the Bi-LSTM-CRF, and its name is a two-part recipe that answers two separate needs: a feature extractor that reads context in both directions, and a decoder that enforces valid tag sequences.
An LSTM is a recurrent network that reads a sentence one word at a time, carrying a memory of what it has seen. Its gated design (input, forget, and output gates) lets it hold onto information across long spans, solving the "vanishing gradient" problem that crippled plain recurrent nets on long sentences. The Bi means bidirectional: we run one LSTM left to right and another right to left, then combine them, so the representation of each word is informed by the entire sentence on both sides. This is the cure for the MEMM's one-eyed problem. Consider "I walked to the river bank." The correct tag and sense of "bank" depend on "river" to its left, but in other sentences the disambiguating word sits to the right ("the bank approved my loan"). Only a model that sees both directions reliably gets these right.
On top of the Bi-LSTM sits a CRF (conditional random field) layer, and it earns its keep by thinking about the tag sequence as a whole. A naive tagger picks the single most probable tag for each word independently — and that can produce sequences that are locally tempting but globally illegal, like a determiner directly followed by a verb. The CRF layer learns the compatibility between adjacent tags and scores the entire path, then finds the highest-scoring valid sequence (using Viterbi again, fittingly). It bakes in soft grammatical constraints — "a verb rarely follows a determiner" — so the output is a coherent sequence, not a string of locally greedy guesses. Bi-LSTM gives you context from both directions; CRF makes sure the final answer hangs together. Together they hit around 97–98% on standard benchmarks.
The bridge to transformers and LLMs
Recurrent models read a sentence sequentially, which is slow and still strains over very long distances. The transformer replaced recurrence with self-attention, a mechanism that lets every word look directly at every other word in the sentence at once, regardless of distance, and weigh how relevant each one is. When tagging "He ran to catch the ball," the model can link "He" to "ran" and "ball" in a single step, in parallel, with no information having to travel word by word. Stacking these attention layers gives the deep, fully bidirectional context that powers models like BERT.
The dominant recipe today is pretrain then fine-tune. First, a model like BERT is pretrained on a colossal amount of text with a self-supervised objective — predicting masked-out words — soaking up grammar, word senses, and a great deal of world knowledge with no labels at all (the distributional hypothesis, again, operating at enormous scale). Then, to tag parts of speech, you bolt a tiny classification layer onto the pretrained model's contextual output vectors and fine-tune the whole thing for a few epochs on a tagged corpus. Because each token's vector already encodes deep context, the final layer's job is easy, and accuracy climbs to 98.5–99.5%. At the far end of the spectrum, you can even skip explicit tagging and just prompt a large language model to label the parts of speech directly — flexible, but slower, costlier, and less deterministic, which is exactly why explicit statistical and neural taggers still earn their place in production systems that need speed, transparency, and safety.
One story, told four ways
Step back and look at the staircase we climbed, because it is really one idea elaborated again and again. We began stuck: a word is a string, and a string holds no meaning a machine can use. The escape was the distributional hypothesis — meaning lives in the company a word keeps. We cashed that out first as counts: co-occurrence matrices, reweighted by TF-IDF and PPMI so distinctiveness beat raw frequency, compared by the angle between vectors rather than their length. Counts were long and sparse and a little blind, so we compressed them into dense embeddings, learned by a prediction game (Word2Vec, GloVe) that placed synonyms side by side and even encoded analogies as directions in space. We aimed the same predict-the-context instinct at whole sequences and got language models — the chain rule, the Markov shortcut, smoothing to forbid the impossible-zero, perplexity to keep score. Then we swapped sparse counts for learned vectors and stacked nonlinear layers, and the n-gram table became a neural language model that generalizes across similar contexts. Finally we watched the whole toolkit converge on a single task — tagging every word with its role — and saw it evolve from HMMs decoded by Viterbi, to Bi-LSTM-CRFs that read both directions and enforce valid sequences, to pretrained transformers that learned the language first and specialized second.
Counts, embeddings, neural nets, pretrained models: it looks like four different fields, but it is one continuous thread. Every step kept the same goal — turn the company a word keeps into geometry a machine can compute on — and just found a richer, more general way to do it. Understand that thread and the modern models stop looking like magic. They look like the same good idea, scaled up and sharpened, learning to read the way we taught it to: by paying attention to context.
References & extra reads
- Dan Jurafsky & James H. Martin, Speech and Language Processing (3rd ed. draft, "SLP3") — the canonical free textbook; chapters on vector semantics, n-gram and neural language models, and POS tagging map directly onto this article.
- Tomas Mikolov et al., "Efficient Estimation of Word Representations in Vector Space" and "Distributed Representations of Words and Phrases and their Compositionality," 2013 — the Word2Vec papers, including skip-gram with negative sampling.
- Jeffrey Pennington, Richard Socher & Christopher Manning, "GloVe: Global Vectors for Word Representation," EMNLP 2014.
- Stanford CS224n, Natural Language Processing with Deep Learning — notes and assignments covering embeddings, language models, and sequence tagging in depth.
- J.R. Firth, "A Synopsis of Linguistic Theory," 1957 — origin of "you shall know a word by the company it keeps."
- Jacob Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," 2019 — the pretrain-then-fine-tune recipe for tagging and beyond.