SLP3 Ch6 — Vector Semantics and Embeddings — NLP Book Notes

// the one-minute version

Chapter 6 explains vector semantics and embeddings as a practical model-building toolkit: represent language numerically, estimate useful quantities from text, make simplifying assumptions explicit, and evaluate on held-out data rather than on memory of the training corpus.

Treat this page as a long plain-English companion to Jurafsky and Martin, Speech and Language Processing, 3rd edition, Chapter 6. It expands the why behind each formula, the engineering choices hidden behind clean notation, and the gotchas that usually cause wrong answers or fragile implementations.

representationturn language into numberstokens, counts, vectors, states

estimationlearn from datacounts, objectives, gradients

evaluationtest honestlyheld-out data, metrics, error analysis

01 Lexical semantics and word relations

Lexical semantics and word relations is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Lexical semantics and word relations sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

Read the math as compressed prose. A conditional probability says what information is known. A \(\frac{\text{count}}{\text{normalizer}}\) says which events are being compared. A \(\sum\) accumulates alternatives, evidence, or loss terms. A \(\log\) changes products into stable additive scores. This habit prevents symbol shock: instead of staring at a formula, translate every part into the modeling decision it represents.

For exams and projects, connect lexical semantics and word relations to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

02 Distributional hypothesis

Distributional hypothesis is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Distributional hypothesis sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect distributional hypothesis to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

03 Term-document matrices

formula\[\operatorname{tfidf}(t,d)=\operatorname{tf}(t,d)\log\frac{N}{\operatorname{df}(t)},\quad \operatorname{PPMI}(w,c)=\max\left(\log\frac{P(w,c)}{P(w)P(c)},0\right),\quad \cos(\theta)=\frac{\mathbf{x}\cdot\mathbf{y}}{\|\mathbf{x}\|\|\mathbf{y}\|}.\]

Term document matrices is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Term document matrices sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect term document matrices to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

04 Tf-idf weighting

think of it likeThe model is a map, not the territory. It leaves out detail so that routes can be planned, compared, and explained.

Tf idf weighting is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Tf idf weighting sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect TF IDF weighting to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

05 Word-context matrices

Word context matrices is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Word context matrices sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect word context matrices to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

RepresentationWhat object is used: tokens, counts, vectors, states, or scores?

EstimationWhich corpus evidence changes the numbers?

FailureWhere do sparsity, ambiguity, bias, or domain shift appear?

06 Ppmi and association strength

watch outNever compare scores unless tokenization, vocabulary, train-test split, and evaluation metric are the same.

Ppmi and association strength is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Ppmi and association strength sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect PPMI and association strength to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

07 Cosine similarity

Cosine similarity is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Cosine similarity sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect cosine similarity to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

08 Dense versus sparse vectors

The chapter turns language evidence into a score or structured decision.

Dense versus sparse vectors is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Dense versus sparse vectors sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect dense versus sparse vectors to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

09 Word2vec skip-gram with negative sampling

Word2vec skip gram with negative sampling is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Word2vec skip gram with negative sampling sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect Word2Vec skip gram with negative sampling to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

RepresentationWhat object is used: tokens, counts, vectors, states, or scores?

EstimationWhich corpus evidence changes the numbers?

FailureWhere do sparsity, ambiguity, bias, or domain shift appear?

10 Cbow objective

the catchClassroom formulas look clean because preprocessing, smoothing, optimization, and error analysis have been hidden. Real NLP lives in those details.

Cbow objective is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Cbow objective sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect CBOW objective to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

11 Glove and global co-occurrence ratios

A second diagram for the same chapter: evidence is weighted, normalized, and interpreted.

Glove and global co occurrence ratios is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Glove and global co occurrence ratios sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect GloVe and global co occurrence ratios to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

12 Properties analogies and vector offsets

Properties analogies and vector offsets is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Properties analogies and vector offsets sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect properties analogies and vector offsets to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

13 Evaluating embeddings

Evaluating embeddings is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Evaluating embeddings sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect evaluating embeddings to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

14 Bias and domain shift

Bias and domain shift is one of the working ideas in Vector Semantics and Embeddings. In plain English, it tells us what evidence the model is allowed to use and what it must predict. The important move is not the notation by itself; the important move is turning a messy language phenomenon into an object that can be counted, weighted, optimized, decoded, or compared. Once that object exists, the rest of the chapter asks how reliable it is, how it fails, and how to evaluate it honestly.

The reason this matters is the recurring NLP trade-off between rich context and reliable statistics. If the representation keeps every detail, the model sees too many rare cases and cannot generalize. If it throws away too much, it becomes stable but shallow. Bias and domain shift sits somewhere between those extremes. It keeps enough structure to be useful, but it imposes a simplifying assumption so the computation can finish on real corpora rather than imaginary perfect data.

For exams and projects, connect bias and domain shift to errors. Ask what happens with unseen words, rare contexts, ambiguous examples, noisy labels, biased text, and domain shift. Ask whether the evaluation is intrinsic, like perplexity or similarity, or extrinsic, like task accuracy. Ask which choices are learned from data and which were manually chosen. These questions turn the chapter from a list of definitions into a debugging checklist.

15 References and extra reads

The primary reference is Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition online draft, Chapter 6. These notes follow the chapter vocabulary while expanding the intuition, assumptions, equations, examples, caveats, and revision strategy.

Extra reads after SLP3: information retrieval notes for vector spaces, smoothing tutorials for language models, neural network optimization introductions, and sequence labeling resources on HMMs, CRFs, and Bi-LSTM-CRF decoders.

FAQ Quick questions students usually ask

Should I memorize every formula?

Memorize the core shape, but focus on what is conditioned, normalized, summed, maximized, or logged.

Why does preprocessing matter?

Because changing tokens, vocabulary, casing, or unknown-word handling changes the event space the model sees.

How should two models be compared?

Use the same data split, preprocessing, metric, and error analysis protocol.

How does this connect to modern NLP?

The architecture may change, but probability, representation, context, evaluation, and bias remain central.

!! Exam traps, implementation gotchas, and debugging smells

common catches & gotchas

Do not confuse the model with language itself — Vector Semantics and Embeddings is an approximation.
Held-out evaluation matters — tuning on the test set contaminates it.
Rare and unknown events matter — sparse evidence causes many failures.
Domain shift is real — text genres do not share one distribution.
Metrics hide errors — inspect examples.
Math is compressed prose — read every \(\sum\), \(\log\), and \(\frac{}{}\) aloud.

++ Takeaways and chapter cheatsheet

Start from the task and the representation.
Name the assumption that makes the computation tractable.
Know which quantities are learned and which are chosen.
Use log probabilities, normalization, and held-out data carefully.
Connect formulas to sparsity, ambiguity, overfitting, bias, and domain mismatch.

// chapter cheatsheetconcept quick-ref

core formulas

fraction\(\frac{\text{count}}{\text{normalizer}}\) turns evidence into probabilities or weights.

sum\(\sum\) accumulates losses, evidence, alternatives, or normalizers.

log\(\log\) turns products into stable additive scores.

revision loop

defineState the input, output, and modeled quantity.

estimateExplain what the corpus contributes.

evaluateUse held-out data and inspect errors.

SLP3 Chapter 6 — Vector Semantics and Embeddings.