Open Source LLMs: Llama, Mistral, Phi
Open-source large language models (LLMs) — most notably Meta's Llama, Mistral AI's Mistral, and Microsoft's Phi families — are powerful transformer-based text generation systems whose weights are publicly released, enabling anyone to download, run, fine-tune, and study them. They matter because they democratize access to frontier AI capabilities that were previously locked behind proprietary APIs, allowing researchers, startups, and individuals to build without usage fees or data-privacy concerns. The counterintuitive fact: Microsoft's Phi-2 (2.7B parameters) outperforms many models ten times its size on reasoning benchmarks, proving that carefully curated training data matters far more than raw scale.
The "aha moment" is this: a language model is just a very large function that takes a sequence of tokens and predicts the next token — and if you train that function on enough high-quality text with the right architecture, remarkably general intelligence emerges. Open-source LLMs prove you don't need a billion-dollar data center to use this intelligence; you only need one to train it from scratch. Once the weights exist, running inference is cheap enough to do on a laptop. The real competition between Llama, Mistral, and Phi is therefore not about raw size, but about the training recipe: what data you use, how you fine-tune, and what architectural tricks you apply.
How It Works
Open-source LLMs are all built on the transformer architecture introduced in 2017. The fundamental operation is self-attention: every token in a sequence looks at every other token and decides how much to "attend" to it when producing the next prediction. What distinguishes Llama, Mistral, and Phi from each other — and from the original transformer — is a set of specific architectural choices and training innovations layered on top of this foundation.
Tokenization: Turning Text into Numbers
Before any math happens, text is split into tokens — subword chunks — using a learned vocabulary. Llama and Mistral both use SentencePiece with a ~32,000-token vocabulary. Phi uses a tiktoken-based vocabulary with ~50,000 tokens. ```python
# Install: pip install transformers from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf") text = "Open source LLMs are powerful." tokens = tokenizer.encode(text) print(tokens) # e.g. [1, 4673, 2752, 365, 28755, 29879, ...] print(tokenizer.decode(tokens)) # "Open source LLMs are powerful." ```
The Attention Score
The core computation that makes transformers work is the scaled dot-product attention formula:
Explanation: Each query vector Q "asks a question," each key vector K "offers an answer," and the dot product measures how well they match; dividing by √dₖ prevents the values from growing so large that softmax saturates.
- Q (Query): what the current token is "looking for"
- K (Key): what each token "offers" to be matched against
- V (Value): the actual information each token carries
- dₖ: the dimension of the key vectors (e.g. 128 per head in Llama-7B)
- softmax: converts raw scores into weights that sum to 1
RMS Normalization
Llama replaced the original LayerNorm with RMSNorm, which is faster because it skips computing the mean:
Explanation: This rescales each activation vector to have unit root-mean-square magnitude, stabilizing training without the extra cost of computing a mean shift.
- x: the input activation vector
- n: the number of elements in x
- ε: a tiny constant (e.g. 1e-6) added for numerical stability
- γ: a learned per-dimension scale parameter
Rotary Positional Embedding (RoPE)
Both Llama and Mistral use RoPE instead of learned position embeddings, encoding position by rotating query and key vectors:
Explanation: Multiplying a query at position m by a complex rotation of angle m·θ encodes its position so that when you take the dot product with a key at position n, only the relative gap (m−n) influences the attention score.
- q̃ₘ: the rotated query at sequence position m
- qₘ: the original query vector
- e^{imθ}: a complex rotation (implemented as paired 2D rotations in practice)
- θ: a base frequency (default 10,000), extended in later models for longer contexts
Sliding Window Attention (Mistral's Key Innovation)
Mistral 7B introduced Grouped Query Attention (GQA) and Sliding Window Attention (SWA) to handle long contexts efficiently. With a window of size W, each token only attends to the W most recent tokens: ```python
# Conceptual illustration of sliding window attention import torch def sliding_window_mask(seq_len: int, window: int) -> torch.Tensor: """Returns a boolean mask: True where attention is allowed.""" mask = torch.zeros(seq_len, seq_len, dtype=torch.bool) for i in range(seq_len): start = max(0, i - window + 1) mask[i, start:i+1] = True return mask mask = sliding_window_mask(seq_len=8, window=3) print(mask.int())
# Each row: token can only see itself + 2 previous tokens ```
Running a Model: The Minimal Example
```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_name = "microsoft/phi-2" # ~2.7B params, fits on 8GB GPU tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.float16, # half precision = half the memory device_map="auto", # auto-place layers on GPU/CPU trust_remote_code=True, ) prompt = "Explain transformers in one sentence:" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=60, temperature=0.7, # randomness: 0=greedy, 1=sampling do_sample=True, ) print(tokenizer.decode(output[0], skip_special_tokens=True)) ```
Instruction Tuning and RLHF
A base model only predicts next tokens — it doesn't "follow instructions." To make a chat model, two extra steps are applied: 1. Supervised Fine-Tuning (SFT): continue training on (instruction, ideal-response) pairs 2. Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO): optimize so the model prefers responses humans rated higher ```python
# Simplified SFT data format (the actual format used by Llama-2-chat) sft_example = { "instruction": "Summarize the following article in 3 bullet points.", "input": "...article text...", "output": "• Point 1\n• Point 2\n• Point 3", }
# Libraries: trl (TransformersRL), axolotl, LLaMA-Factory all automate this ```
Vaswani led the Google Brain team that published "Attention Is All You Need" (2017), the paper that eliminated recurrent networks entirely and introduced the pure-attention transformer that all modern LLMs — Llama, Mistral, and Phi included — are built upon. His specific contribution was showing that multi-head self-attention alone, without any convolution or recurrence, could achieve state-of-the-art machine translation while being dramatically faster to train in parallel. Surprisingly, Vaswani left Google shortly after the paper's publication and later co-founded Adept AI before moving to independent research.
Lample is a French researcher who, while at Meta AI (FAIR), co-led the development of the original Llama model family alongside Hugo Touvron, establishing that a carefully trained smaller model could match GPT-3-scale performance. His earlier work on cross-lingual language models and unsupervised machine translation laid the groundwork for the multilingual pretraining recipes used in Llama 2 and beyond. Notably, Lample left Meta in 2023 to co-found Mistral AI — meaning the same researcher is directly responsible for two of the three model families covered in this article.
Intuition
Imagine the model as an incredibly well-read librarian who has absorbed the patterns in trillions of words. When you hand them a partial sentence, they don't look up the "correct" continuation in a database — they use their internalized sense of language patterns to guess what word most plausibly comes next. This is all a language model does, one token at a time, in an autoregressive loop. The transformer's self-attention mechanism is like the librarian's ability to hold the entire sentence in mind simultaneously and consider how every word relates to every other word before making a prediction — unlike older recurrent networks that had to read left-to-right and often "forgot" early context. The difference between Llama, Mistral, and Phi is mostly in the quality and composition of the librarian's reading list: Phi-3 was trained on heavily filtered, textbook-quality synthetic data, which is why a small librarian with focused education outperforms a much larger one who read more but read widely.