Embedding Geometry and Analogies

An embedding assigns each word $w$ in a vocabulary $V$ a vector $\vec{w} \in \mathbb{R}^d$, typically with $d$ between 50 and 300. Training via Negative Sampling and Training Word2Vec or the co-occurrence factorization of GloVe arranges these vectors so that words appearing in similar contexts end up close together. "Geometry" here means the study of that arrangement: which directions correspond to which meanings, how distance relates to similarity, and whether relationships between word pairs are systematically encoded as consistent vector offsets. The primary similarity measure is cosine similarity, not Euclidean distance:

Cosine similarity measures the angle between two vectors, ignoring their magnitude. This matters because word frequency strongly affects vector length during training — frequent words tend to have larger norms — but frequency is not the semantic property we want to measure. Two vectors pointing in nearly the same direction are treated as similar regardless of how long they are, which normalizes away this frequency artifact. The analogy task asks: given a relationship exemplified by the pair $(a, b)$, and a third word $c$, find $d$ such that $(c, d)$ exhibits the same relationship. Formally, $a : b :: c : d$, e.g. man : king :: woman : queen. The vector-offset method, sometimes called 3CosAdd, proposes:

The intuition: if the relationship "male → royal-female-equivalent" is encoded as a roughly consistent offset vector across many word pairs, then adding that offset to a new word's vector should land near the analogous target. This only works because embedding training does not merely cluster synonyms — it also linearizes certain relational structure, so that "capital-of," "plural-of," and "gender-swap" each correspond, approximately, to a single translation vector usable across many word pairs.

Step 1 — Cosine similarity for nearest neighbors. Given a query vector, rank all vocabulary vectors by cosine similarity to find the most similar words. This underlies "find synonyms" and retrieval-style applications.

Step 2 — Vector offset for analogies. Compute $\vec{b} - \vec{a} + \vec{c}$ and search for the nearest neighbor, excluding the three input words themselves (a critical, easily-missed step — without it, the algorithm frequently just returns $c$).

Step 3 — 3CosMul, a refinement. The additive method has a known failure mode: one large term in the sum can dominate and swamp the others. Levy and Goldberg's multiplicative variant instead computes

using a small $\varepsilon$ (e.g. 0.001) to avoid division by zero. This balances the three similarity terms multiplicatively rather than additively, empirically improving analogy accuracy.

Step 4 — Bias measurement. Define a bias direction, e.g. $\vec{g} = \vec{he} - \vec{she}$ (or an averaged version over several such pairs via PCA). Project any word vector onto $\vec{g}$ to get a scalar bias score; words like nurse or engineer will show measurable projections along this axis even though gender is not part of their dictionary meaning.

Step 5 — Debiasing. Neutralize non-gendered words by removing their component along $\vec{g}$, and equalize gendered pairs (like grandmother/grandfather) so they are equidistant from neutral words, preserving the useful part of the relationship while erasing the harmful leakage. ```text function NEAREST(query_vec, V, k, exclude={}): scores = [] for w in V: if w in exclude: continue s = cosine(query_vec, vec[w]) scores.append((s, w)) sort scores descending by s return top k entries function ANALOGY_3COSADD(a, b, c, V):

# a : b :: c : ? target = vec[b] - vec[a] + vec[c] return NEAREST(target, V, k=1, exclude={a, b, c}) function ANALOGY_3COSMUL(a, b, c, V, eps=0.001): best_word, best_score = None, -infinity for d in V: if d in {a, b, c}: continue num = cosine(vec[d], vec[b]) * cosine(vec[d], vec[c]) den = cosine(vec[d], vec[a]) + eps

# shift cosines to [0,2] first in practice, to keep them positive score = num / den if score > best_score: best_score, best_word = score, d return best_word function BIAS_SCORE(w, bias_direction): return dot(vec[w], bias_direction) / norm(bias_direction) function DEBIAS_NEUTRALIZE(w, bias_direction, neutral_set): if w not in neutral_set: return vec[w] proj = (dot(vec[w], bias_direction) / dot(bias_direction, bias_direction)) * bias_direction return normalize(vec[w] - proj) ``` The exclusion set in `NEAREST` is not a minor detail — omitting it is the single most common bug when reimplementing analogy solvers, since $c$ itself is almost always the closest vector to $\vec{b} - \vec{a} + \vec{c}$ when $a$ and $b$ are similar in magnitude.

Time complexity. Computing cosine similarity between two $d$-dimensional vectors is $O(d)$. A brute-force nearest-neighbor search over vocabulary size $|V|$ costs $O(|V| \cdot d)$ per query. For $|V| \approx 400{,}000$ (typical pretrained vocabularies) and $d = 300$, this is roughly $1.2 \times 10^8$ floating-point operations per analogy query — feasible but non-trivial at scale. In production, approximate nearest neighbor structures (e.g. HNSW graphs, locality-sensitive hashing, or inverted files with product quantization) reduce this to roughly $O(\log |V| \cdot d)$ or better in practice, trading a small accuracy loss for large speedups. 3CosMul has the same asymptotic cost as 3CosAdd, since it computes the same three cosine similarities per candidate, only combining them differently.

Space complexity. Storing all vectors requires $O(|V| \cdot d)$ floats — around 480 MB for $|V| = 400{,}000$, $d = 300$ in 32-bit floats. Precomputing vector norms once ($O(|V|)$ extra space) lets cosine similarity be computed as a single dot product plus lookups rather than recomputing norms per query.

Correctness of the offset method — why it should work at all. This is not a proven theorem but a consequence of the training objective's structure, and it is worth being precise about the argument rather than just asserting the result. In skip-gram with negative sampling, the training objective pushes $\vec{w} \cdot \vec{c}$ high for observed (word, context) pairs and low for sampled negatives. Analyses of word2vec (Levy & Goldberg, 2014) show this is implicitly factorizing a shifted pointwise mutual information (PMI) matrix: the dot product $\vec{w} \cdot \vec{c}$ approximates $\text{PMI}(w, c) - \log k$, where $k$ is the number of negative samples. Now consider a relation like "capital of." If $\text{PMI}(\textit{Paris}, \textit{France})$-like statistics behave consistently across many country–capital pairs — because the corpus co-occurrence patterns of "France, capital, Paris" resemble those of "Japan, capital, Tokyo" in aggregate — then the linear structure of the PMI matrix (and hence the factorized embeddings) will encode this consistency as an approximately constant offset vector. The vector offset $\vec{b} - \vec{a}$ is only meaningful to the extent that the relation's log-probability structure really is translation-invariant across pairs in the corpus. When it is (frequent, well-attested, symmetric relations like capital-of or plural-of), the offset method works well. When the relation is rare, asymmetric, or lexically inconsistent (irregular plurals, obscure professions), the linear approximation breaks down and accuracy drops sharply — this is an honest limitation, not a bug to be patched.

Why cosine, not Euclidean distance. Word frequency correlates with vector norm because frequent words receive many more gradient updates during training. If we used Euclidean distance $\|\vec{u} - \vec{v}\|$, a rare word could appear "far" from a related common word purely due to differing magnitudes, even when perfectly aligned in direction. Cosine similarity divides out this magnitude effect, isolating the semantically meaningful angular component. Formally, for vectors normalized to unit length, $\|\vec{u} - \vec{v}\|^2 = 2 - 2\cos(\vec{u}, \vec{v})$, so ranking by cosine similarity and ranking by Euclidean distance on normalized vectors are equivalent — the distinction only matters when vectors are not pre-normalized, which is the typical raw output of training.

Take pretrained GloVe-style vectors (illustrative values, dimensions truncated to 4 for hand-tracing; real vectors use 50–300 dimensions):

Step 1 — compute the offset.

Step 2 — compare to candidates via cosine similarity. The target vector $(0.6, 0.6, 0.7, 0.2)$ is compared against every vocabulary word except {man, king, woman}. Against queen = $(0.6, 0.6, 0.7, 0.2)$:

(In this illustrative example queen is chosen to exactly equal the target, giving perfect similarity — in real embeddings the match is close, not exact.) Against apple = $(0.1, 0.9, 0.0, 0.3)$:

Step 3 — rank and select. queen scores 1.0, apple scores 0.62. queen wins decisively — this is exactly the intended behavior, and it illustrates why the exclusion set matters: without excluding `woman`, note that $\cos(\vec{t}, \vec{woman})$ would also need checking, and for vectors this close in construction, an unexcluded input can sometimes outscore the true answer.

A harder case, showing the limitation. Try Tokyo : Japan :: Berlin : ? (expected: Germany). If the corpus discusses Tokyo and Japan with different surrounding syntax than Berlin and Germany (e.g. one pair appears mostly in travel contexts, the other in historical/political contexts), the offset $\vec{Japan} - \vec{Tokyo}$ will not equal $\vec{Germany} - \vec{Berlin}$ closely, and the nearest neighbor to the computed target may be a nearby European city or a generic "country" vector rather than Germany specifically. Published analogy benchmarks report accuracy well above 50% on curated syntactic and common semantic categories (word2vec's original paper reports around 60–70% on its combined test set), but accuracy on rarer or looser semantic relations — rather than the tightly curated capital/country and morphological categories that dominate standard benchmarks — is substantially lower, often below 30%. The gap matters: it is easy to over-generalize from a benchmark built almost entirely from the relation types embeddings are best at.

Search and retrieval systems use cosine similarity over embeddings to rank documents, products, and passages, treating query and document embeddings as points in the same geometric space. Recommendation engines position items and users in shared embedding spaces so that "nearby" means "relevant." Machine translation systems have used cross-lingual embedding alignment — literally rotating one language's embedding space onto another's — exploiting the assumption that relational geometry (king-queen, man-woman style offsets) is roughly preserved across languages. Fairness auditing tools built on the Word Embedding Association Test (WEAT) directly measure the geometric bias described here, quantifying how strongly occupation or trait words align with gender or racial identity directions, informing dataset curation and model审计 decisions before deployment. All of this groundwork — static, one-vector-per-word geometry — is also the direct motivation for Contextual Embeddings, where models like BERT and GPT assign a word a different vector depending on its sentence, precisely because a single fixed direction cannot capture how "bank" shifts between "river bank" and "savings bank."

Ask the tutor: How would embedding geometry and the analogy task need to change for a language with rich morphology, where a single "word" can take dozens of inflected forms?