SLP3 Ch3 — N-gram Language Models — NLP Book Notes

// the one-minute version

Chapter 3 explains n-gram language models as a practical model-building toolkit: represent language numerically, estimate useful quantities from text, make simplifying assumptions explicit, and evaluate on held-out data rather than on memory of the training corpus.

Treat this page as a long plain-English companion to Jurafsky and Martin, Speech and Language Processing, 3rd edition, Chapter 3. It expands the why behind each formula, the engineering choices hidden behind clean notation, and the gotchas that usually cause wrong answers or fragile implementations.

representationturn language into numberstokens, counts, vectors, states

estimationlearn from datacounts, objectives, gradients

evaluationtest honestlyheld-out data, metrics, error analysis

01 Probability of sequences

Probability of sequences is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Probability of sequences sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

Read the math as compressed prose. A conditional probability says what information is known. A \(\frac{\text{count}}{\text{normalizer}}\) says which events are being compared. A \(\sum\) accumulates alternatives, evidence, or loss terms. A \(\log\) changes products into stable additive scores. This habit prevents symbol shock: instead of staring at a formula, translate every part into the modeling decision it represents.

For exams and projects, connect probability of sequences to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

02 Chain rule exact factorization

Chain rule exact factorization is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Chain rule exact factorization sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect chain rule exact factorization to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

03 Markov assumption and bigrams/trigrams

formula\[P(w_1^N)=\prod_{i=1}^{N}P(w_i\mid w_1^{i-1}),\quad P_{\text{MLE}}(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)}{C(w_{i-1})},\quad \log P(W)=\sum_i\log P(w_i\mid h_i).\]

Markov assumption and bigrams/trigrams is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Markov assumption and bigrams/trigrams sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect Markov assumption and bigrams/trigrams to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

04 Maximum likelihood from counts

think of it likeThe model is a map, not the territory. It leaves out detail so that routes can be planned, compared, and explained.

Maximum likelihood from counts is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Maximum likelihood from counts sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect maximum likelihood from counts to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

05 Sentence boundary tokens and unknown words

Sentence boundary tokens and unknown words is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Sentence boundary tokens and unknown words sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect sentence boundary tokens and unknown words to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

RepresentationWhat object is used: tokens, counts, vectors, states, or scores?

EstimationWhich corpus evidence changes the numbers?

FailureWhere do sparsity, ambiguity, bias, or domain shift appear?

06 Log probabilities and numerical stability

watch outNever compare scores unless tokenization, vocabulary, train-test split, and evaluation metric are the same.

Log probabilities and numerical stability is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Log probabilities and numerical stability sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect log probabilities and numerical stability to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

07 Generating text from a model

Generating text from a model is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Generating text from a model sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect generating text from a model to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

08 Training development test and extrinsic evaluation

The chapter turns language evidence into a score or structured decision.

Training development test and extrinsic evaluation is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Training development test and extrinsic evaluation sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect training development test and extrinsic evaluation to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

09 Intrinsic evaluation with perplexity

Intrinsic evaluation with perplexity is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Intrinsic evaluation with perplexity sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect intrinsic evaluation with perplexity to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

RepresentationWhat object is used: tokens, counts, vectors, states, or scores?

EstimationWhich corpus evidence changes the numbers?

FailureWhere do sparsity, ambiguity, bias, or domain shift appear?

10 Zero counts and sparsity

the catchClassroom formulas look clean because preprocessing, smoothing, optimization, and error analysis have been hidden. Real NLP lives in those details.

Zero counts and sparsity is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Zero counts and sparsity sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect zero counts and sparsity to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

11 Laplace and add-k smoothing

A second diagram for the same chapter: evidence is weighted, normalized, and interpreted.

Laplace and add k smoothing is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Laplace and add k smoothing sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect Laplace and add k smoothing to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

12 Backoff and interpolation

Backoff and interpolation is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Backoff and interpolation sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect backoff and interpolation to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

13 Kneser-ney continuation intuition

Kneser ney continuation intuition is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Kneser ney continuation intuition sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect Kneser Ney continuation intuition to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

14 How n-grams prepare neural language models

How n grams prepare neural language models is one of the working ideas in N-gram Language Models. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. How n grams prepare neural language models sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect how n grams prepare neural language models to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

15 References and extra reads

The primary reference is Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition online draft, Chapter 3. These notes follow the chapter vocabulary while expanding the intuition, assumptions, equations, examples, caveats, and revision strategy.

Extra reads after SLP3: information retrieval notes for vector spaces, smoothing tutorials for language models, neural network optimization introductions, and sequence labeling resources on HMMs, CRFs, and Bi-LSTM-CRF decoders.

FAQ Quick questions students usually ask

Should I memorize every formula?

Memorize the core shape, but focus on what is conditioned, normalized, summed, maximized, or logged.

Why does preprocessing matter?

Because changing tokens, vocabulary, casing, or unknown-word handling changes the event space the model sees.

How should two models be compared?

Use the same data split, preprocessing, metric, and error analysis protocol.

How does this connect to modern NLP?

The architecture may change, but probability, representation, context, evaluation, and bias remain central.

!! Exam traps, implementation gotchas, and debugging smells

common catches & gotchas

Do not confuse the model with language itself — N-gram Language Models is an approximation.
Held-out evaluation matters — tuning on the test set contaminates it.
Rare and unknown events matter — sparse evidence causes many failures.
Domain shift is real — text genres do not share one distribution.
Metrics hide errors — inspect examples.
Math is compressed prose — read every \(\sum\), \(\log\), and \(\frac{}{}\) aloud.

++ Takeaways and chapter cheatsheet

Start from the task and the representation.
Name the assumption that makes the computation tractable.
Know which quantities are learned and which are chosen.
Use log probabilities, normalization, and held-out data carefully.
Connect formulas to sparsity, ambiguity, overfitting, bias, and domain mismatch.

// chapter cheatsheetconcept quick-ref

core formulas

fraction\(\frac{\text{count}}{\text{normalizer}}\) turns evidence into probabilities or weights.

sum\(\sum\) accumulates losses, evidence, alternatives, or normalizers.

log\(\log\) turns products into stable additive scores.

revision loop

defineState the input, output, and modeled quantity.

estimateExplain what the corpus contributes.

evaluateUse held-out data and inspect errors.

SLP3 Chapter 3 — N-gram Language Models.