Every equation needed for mid-sem NLP revision, grouped by concept. Each card gives the display equation and defines the symbols immediately after it in plain words. Use this as the formula-first companion to the cheatsheet.
Symbol key
| Symbol | Meaning |
|---|---|
| \(w_i\) | the word at position \(i\) in a sequence |
| \(t_i\) | the tag at position \(i\), such as a POS tag |
| \(h\) | a history or context, often previous words or tags |
| \(V\) | the vocabulary; \(|V|\) is the number of word types |
| \(C(\cdot)\) | a count from data |
| \(x,w,b,z,a\) | input vector, weight vector, bias, pre-activation and activation |
Similarity & weighting
\(a,b\) are vectors with \(d\) dimensions. \(a_i,b_i\) are values in dimension \(i\). A larger dot product means stronger shared weighted evidence, but it also grows with vector length.
\(\theta\) is the angle between vectors. \(\|a\|\) and \(\|b\|\) are vector lengths. Cosine compares direction, so it is useful when document length should not dominate similarity.
\(t\) is a term. \(d\) is a document. \(C(t,d)\) is the count of the term in the document. Log TF keeps repeated words useful without letting them grow linearly forever.
\(D\) is the document collection. \(N=|D|\) is number of documents. \(\mathrm{df}(t)\) counts documents containing term \(t\). IDF is high for rare informative words and low for common words.
\(\mathrm{tf}(t,d)\) measures importance inside one document. \(\mathrm{idf}(t)\) measures rarity across documents. The product rewards terms frequent here but not everywhere.
\(w\) is a target word. \(c\) is a context. \(C(w,c)\) is their co-occurrence count. \(N\) is the total co-occurrence count used for normalization.
\(P(w,c)\) is observed joint probability. \(P(w)P(c)\) is chance co-occurrence under independence. PMI is positive when association is stronger than chance; PPMI clips negative unreliable values to zero.
Word2Vec & embeddings
\(w_i\) is the input center word. \(w_o\) is an observed outside/context word. \(v_{w_i}\) is the input embedding. \(v'_w\) is an output embedding. \(|V|\) is vocabulary size. The denominator normalizes over all words.
\(T\) is number of tokens. \(m\) is window radius. \(w_t\) is center word. \(w_{t+j}\) is a neighbor. The model maximizes log probability of true context words.
\(v_i\) is the center-word vector. \(v'_o\) is the true outside-word vector. \(n_k\) is a sampled noise word. \(K\) is number of negative samples. The first term rewards real pairs; the second punishes random pairs.
\(x\) is a score such as a dot product. The sigmoid maps any real score to a number between zero and one, interpretable as binary probability.
\(\bar{v}\) is the averaged context embedding. \(m\) is window radius. \(U\) is an output weight matrix. CBOW predicts the center word from surrounding words.
\(X_{ij}\) is word-context co-occurrence count. \(w_i\) and \(\tilde{w}_j\) are word and context vectors. \(b_i,\tilde{b}_j\) are biases. \(f\) weights counts so rare and extremely frequent pairs do not dominate.
Language models
\(w_1,\ldots,w_n\) are words in order. The equation is exact: each word is predicted from all previous words.
\(w_1^n\) means the whole sequence. Bigram models keep only the previous word as history, trading linguistic depth for count reliability.
\(n\) is n-gram order. \(w_{i-n+1}^{i-1}\) is the previous \(n-1\) words. The Markov assumption makes estimation possible.
\(C(w_{i-1},w_i)\) counts the bigram. \(C(w_{i-1})\) counts the history word. MLE uses observed relative frequency.
\(C(w_{i-n+1}^{i})\) counts the full n-gram ending at \(w_i\). \(C(w_{i-n+1}^{i-1})\) counts the context. Divide full event count by context count.
\(h_i\) is the chosen history for word \(w_i\). Logs turn products into sums and prevent numerical underflow.
\(h\) is history. \(w\) is a candidate next word. \(|V|\) is vocabulary size. Adding one gives nonzero probability to every word but can over-smooth.
\(k\) is a small positive constant. Smaller \(k\) redistributes less probability mass than add-one smoothing.
\(P_3,P_2,P_1\) are trigram, bigram and unigram models. \(\lambda_1,\lambda_2,\lambda_3\) are mixture weights with \(\lambda_1+\lambda_2+\lambda_3=1\). Every order contributes.
\(P_{\mathrm{cont}}(w)\) is based on how many distinct histories precede \(w\). It prefers words that complete many contexts, not only words with high raw frequency.
\(W\) is the test sequence. \(N\) is number of predicted tokens. Lower perplexity means the model assigns higher average probability to the observed text.
\(H(W)\) is average negative log probability in bits. Perplexity is two raised to that average uncertainty.
Neural equations
\(x\) is input vector. \(w\) is weight vector. \(b\) is bias. \(z\) is pre-activation. \(a\) is output activation. \(f\) is a non-linear activation function.
\(z\) is a scalar score. Sigmoid maps to zero-one, tanh maps to minus-one-one, and ReLU keeps positive values while zeroing negatives.
\(h^{(l)}\) is hidden representation at layer \(l\). \(W^{(l)}\) is weight matrix. \(b^{(l)}\) is bias vector. \(f\) is applied elementwise.
\(s_k\) is score for class \(k\). \(K\) is number of classes. Softmax converts scores into probabilities that sum to one.
\(y_k\) is the true one-hot label value. \(\hat{y}_k\) is predicted probability. The loss is small when the true class gets high probability.
\(\theta\) is any trainable parameter. \(\eta\) is learning rate. \(\nabla_{\theta}L\) is the gradient of loss with respect to the parameter. Move opposite the gradient to reduce loss.
Sequence labeling
\(\hat{t}\) is the chosen tag. \(w\) is word. \(P(w\mid t)\) is emission likelihood. \(P(t)\) is tag prior. This is a local simplification, not full sequence decoding.
\(w_1^n\) is the word sequence. \(t_1^n\) is the tag sequence. \(P(w_i\mid t_i)\) is emission probability. \(P(t_i\mid t_{i-1})\) is transition probability.
\(a_{ij}\) is probability of moving from tag \(t_i\) to tag \(t_j\). Counts are estimated from tagged data.
\(b_j(o)\) is probability that tag \(t_j\) emits observation word \(o\). \(C(t_j,o)\) counts tag-word pairs.
\(\alpha_t(j)\) is probability of observing first \(t\) words and ending in state \(j\). \(o_t\) is current observation. \(a_{ij}\) is transition. \(b_j(o_t)\) is emission.
\(v_t(j)\) is best path score ending in tag/state \(j\) at time \(t\). \(i\) ranges over previous states. Store the maximizing \(i\) as a backpointer.
\(x\) is the observed sentence. \(f_k\) are feature functions. \(\lambda_k\) are learned weights. MEMMs use rich features but normalize locally.
\(x\) is input sentence. \(y\) is a label path. \(A_{y_{i-1},y_i}\) is transition score between labels. \(P_{i,y_i}\) is emission score for label \(y_i\) at position \(i\).
\(\mathcal{Y}(x)\) is the set of possible label sequences for input \(x\). The denominator normalizes over complete paths, giving globally normalized sequence probabilities.