Word Embeddings

Word embeddings are dense, low-dimensional vector representations of words that capture semantic and syntactic relationships, allowing computers to treat language mathematically. They matter enormously in modern AI because virtually every NLP system — from search engines to large language models — depends on some form of learned word representation. The remarkable fact: a simple vector arithmetic operation reveals that king − man + woman ≈ queen, meaning geometry in embedding space encodes human-level conceptual relationships.

The core insight

Words are just symbols to a computer — "cat" and "dog" look no more related than "cat" and "democracy." Word embeddings solve this by mapping every word to a list of numbers (a vector) so that words with similar meanings end up close together in that number space. The model learns these numbers automatically by studying which words tend to appear near each other in billions of sentences. The "aha moment" is realizing that meaning emerges from context — you can tell a lot about a word by the company it keeps.

How It Works

At heart, word embeddings are a lookup table: each word in the vocabulary gets a row in a matrix, and that row is the word's vector. The key question is how to learn good numbers for those rows. The dominant approach — popularized by Word2Vec — trains a tiny neural network on a self-supervised task called predicting context. No human labels are needed; the text itself provides supervision.

The Skip-Gram objective asks: given a center word, predict which words are likely to appear nearby (within a sliding window).

\quad \mathcal{L} = -\sum_{(w,\, c) \in D} \log P(c \mid w)

Explanation: We want to maximize the total log-probability of seeing each observed context word c given its center word w, across all word–context pairs in the training data. Here:

$\mathcal{L}$ is the loss (we minimize its negative, i.e., maximize probability)
$w$ is the center word
$c$ is a context word (neighbor in the window)
$D$ is the dataset of all observed (center, context) pairs

The probability of a context word is computed via a softmax over dot products:

\quad P(c \mid w) = \frac{\exp(\mathbf{v}_c \cdot \mathbf{v}_w)}{\sum_{k=1}^{V} \exp(\mathbf{v}_k \cdot \mathbf{v}_w)}

Explanation: Words whose vectors point in a similar direction (high dot product) get assigned higher probability of being neighbors. Here:

$\mathbf{v}_w$ is the embedding vector for center word $w$
$\mathbf{v}_c$ is the embedding vector for context word $c$
$V$ is the vocabulary size (sum over all possible words in the denominator)

After training, the embedding matrix is extracted and used as a lookup table in downstream tasks.

A minimal Python example — loading and using pre-trained embeddings: ```python

# pip install gensim from gensim.models import KeyedVectors import gensim.downloader as api

# Load a small pre-trained Word2Vec model (~66 MB) model = api.load("word2vec-google-news-300")

# Each word maps to a 300-dimensional vector king_vec = model["king"] print(f"Vector shape: {king_vec.shape}") # (300,)

# Semantic arithmetic: king - man + woman ≈ queen result = model.most_similar( positive=["king", "woman"], negative=["man"], topn=3 ) print(result)

# [('queen', 0.71), ('monarch', 0.64), ('princess', 0.63)]

# Find nearest neighbors neighbors = model.most_similar("python", topn=5) for word, score in neighbors: print(f" {word}: {score:.3f}") ```

Training your own embeddings from scratch with Gensim: ```python from gensim.models import Word2Vec

# Toy corpus — in practice, use millions of sentences sentences = [ ["the", "cat", "sat", "on", "the", "mat"], ["the", "dog", "ran", "in", "the", "park"], ["cats", "and", "dogs", "are", "pets"], ["paris", "is", "the", "capital", "of", "france"], ["london", "is", "the", "capital", "of", "england"], ] model = Word2Vec( sentences=sentences, vector_size=50, # embedding dimensions window=2, # context window size min_count=1, # ignore words appearing fewer times sg=1, # 1 = skip-gram, 0 = CBOW epochs=100 )

# Retrieve a vector vec = model.wv["cat"] print(f"'cat' vector (first 5 dims): {vec[:5]}")

# Save / reload model.wv.save("my_embeddings.kv") ```

Measuring similarity — cosine similarity:

\quad \text{sim}(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}

Explanation: Cosine similarity measures the angle between two vectors — 1 means identical direction (very similar words), 0 means perpendicular (unrelated), −1 means opposite. ```python import numpy as np def cosine_similarity(u, v): return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v)) cat = model.wv["cat"] dog = model.wv["dog"] mat = model.wv["mat"] print(f"cat ↔ dog: {cosine_similarity(cat, dog):.3f}") # high print(f"cat ↔ mat: {cosine_similarity(cat, mat):.3f}") # low ```

researcher · Yoshua Bengio | b. 1964 | Neural Language Models & Word Representations

Bengio is a Canadian computer scientist and Turing Award laureate (2018) who co-founded the deep learning revolution. His landmark 2003 paper A Neural Probabilistic Language Model (with Ducharme, Vincent, and Jauvin) introduced the idea of learning continuous word representations jointly with a language model — the conceptual grandfather of all modern embeddings. Surprising fact: Bengio initially struggled to get the 2003 paper accepted; reviewers found the idea of learning word vectors simultaneously with a language model unnecessarily complicated.

researcher · Tomáš Mikolov | b. 1986 | Word2Vec

Mikolov is a Czech ML researcher who created Word2Vec while at Google in 2013, making high-quality word embeddings practical at massive scale for the first time. His key insight was that you could strip away most of the neural network complexity and train just the embedding lookup efficiently using negative sampling, cutting training time from weeks to hours on billions of words. Notably, he discovered the famous king−man+woman≈queen analogy accidentally while exploring what the geometry of the learned space actually encoded.

Figure

Word2Vec Skip-Gram Architecture

Figure

Embedding Space — Semantic Geometry

Intuition

The Mental Model

Imagine a huge city map where every word is a building. Words that "live together" — appearing in similar sentences — end up built close to each other on this map. "Cat" and "dog" are neighbors because both appear near words like "pet," "furry," "feed," and "vet." "King" and "queen" are neighbors, but they're also both exactly as far from "man" and "woman" as each other — and that direction of difference (the road between the male and female neighborhoods) is the same road you travel between "king" and "queen." The vector arithmetic (king − man + woman = queen) is just asking: start at king, walk the road to the male neighborhood, then walk the same distance in the female direction — and you land at queen. The geometry of the map encodes human concepts automatically, without anyone programming it. The model learned this map purely by reading text and noticing who lives near whom. Think of it this way: if you were blindfolded and could only hear what words were said around you, you'd eventually figure out that "cat" and "dog" must be similar things — because people talk about them the same way. That's exactly what Word2Vec does.

Historical Development

Origins · 1 minhow this was discovered

History

1986

Hinton, McClelland & Rumelhart introduce distributed representations — the idea that concepts should be represented as patterns of activation across many units, not single symbolic nodes. This is the conceptual seed of embeddings.

2001

Bengio, Ducharme, Vincent & Jauvin publish A Neural Probabilistic Language Model, the first paper to jointly learn continuous word vectors and a language model using a neural network.

2008

Collobert & Weston show in A Unified Architecture for NLP that sharing word representations across tasks gives strong performance gains — demonstrating that embeddings transfer across NLP problems.

2011

Mikolov et al. introduce Recurrent Neural Network Language Models (RNNLM), using learned word vectors as input and achieving state-of-the-art language modeling results.

2013

Mikolov et al. at Google publish Word2Vec (two papers: Skip-Gram and CBOW), demonstrating that simple architectures trained on billions of words produce remarkably structured vector spaces with the famous analogy property.

2014

Pennington, Socher & Manning at Stanford publish GloVe (Global Vectors), combining Word2Vec-style local context with global corpus co-occurrence statistics, achieving competitive results with a cleaner mathematical objective.

2016

Facebook Research releases fastText, extending Word2Vec to subword units (character n-grams), allowing embeddings for out-of-vocabulary words and handling morphologically rich languages far better.

2018

Peters et al. introduce ELMo (Embeddings from Language Models) — the first widely-used contextual embeddings, where the same word gets different vectors depending on sentence context (solving the polysemy problem).

2018

–present: BERT, GPT, and subsequent Transformer models make contextual embeddings the standard, with static word embeddings like Word2Vec still widely used for efficiency-constrained applications and as an entry point for learning NLP.

Why It Matters

Real-World Impact

Google SearchWord embeddings (and their descendants) power semantic search — queries like "best way to cook pasta" match pages about "how to boil spaghetti" even with no shared keywords, because their embeddings are geometrically close.

Spam Filters (Gmail)Email classification models use word embeddings as features so that new spam phrasing ("cheap m3ds online") is caught based on semantic proximity to known spam vocabulary, not just exact keyword matches.

Machine Translation (Google Translate, DeepL)Early neural MT systems used word embeddings to initialize encoder and decoder vocabularies. Even modern Transformer-based systems build on the same concept — a learned embedding table is the first layer of every translation model.

Recommendation Systems (Spotify, YouTube)The "Word2Vec trick" is applied to sequences of songs, videos, or products treated as "words" in a "sentence" of user behavior. Items a user consumes together end up with similar vectors, powering "you might also like" recommendations.

Sentiment Analysis (customer review tools)Simple models — logistic regression or SVMs — trained on averaged word embeddings can classify product reviews as positive or negative with ~90% accuracy, far outperforming bag-of-words approaches.

Biomedical NLP (BioWordVec, PubMedBERT)Medical literature is too specialized for general embeddings; domain-specific Word2Vec models trained on PubMed abstract corpora are used to link drug names, symptoms, and genes in clinical decision support systems.

Code Search (GitHub Copilot precursors)Source code tokens (function names, variable names, API calls) are embedded using Word2Vec-style training on large code corpora, enabling search-by-meaning across codebases and powering early code completion tools.

Word Embeddings

How It Works

Intuition

Historical Development

Why It Matters

Prerequisites

What This Leads To

See Also

Where this leads