Text-to-Image with CLIP and Latent Diffusion

A text-to-image generator must solve two coupled subproblems: representing what an image should depict in a form a numeric model can use, and generating pixels that satisfy that representation. CLIP solves the first problem by training an image encoder and a text encoder jointly with a contrastive objective, so that the embedding of an image and the embedding of a caption describing it are pulled close together in a shared vector space, while embeddings of mismatched pairs are pushed apart. This gives any text prompt a fixed-length vector that is directly comparable to image content — a semantic address the diffusion model can be told to aim for.

Latent diffusion solves the second problem efficiently. Ordinary Diffusion Models for Images learn to reverse a fixed noising process applied directly to pixels: start from pure Gaussian noise and iteratively predict-and-subtract noise until a clean image emerges. Doing this directly on, say, 512×512×3 pixel tensors is expensive, because the denoising network must run many times (tens to thousands of steps) over a large spatial grid. Latent diffusion instead trains a perceptual autoencoder — an encoder $\mathcal{E}$ and decoder $\mathcal{D}$, typically a VAE with an adversarial and perceptual loss (a form of Image-to-Image Translation compression) — that maps images to a much smaller latent grid, e.g. 64×64×4, and back, with the decoder trained to reconstruct pixel-level detail. The entire diffusion process — forward noising and learned reverse denoising — happens in this small latent space; the decoder is invoked only once at the very end to produce final pixels. The bridge between the two ideas is conditioning. The denoising network (a U-Net) is given the CLIP-encoded text embedding at every layer via cross-attention, so at each of the many denoising steps the network's noise prediction is informed by the prompt. To make the model actually listen to this conditioning rather than treat it as a weak hint, the model is trained to also produce unconditional predictions, and at sampling time these two predictions are combined via classifier-free guidance, which extrapolates in the direction the conditioning signal points, deliberately overshooting to sharpen adherence to the prompt at some cost to diversity and realism.

The full pipeline is trained and then run in three stages: train CLIP (once, as a general-purpose model), train the latent autoencoder plus the conditioned diffusion U-Net (once, on a large image-text corpus), and finally sample — the part the user experiences — which iteratively denoises latent noise into a latent image under text guidance, then decodes it to pixels.

Stage 1 — CLIP pretraining. Given a large batch of $N$ (image, caption) pairs, encode each image with a vision encoder (ViT or ResNet) and each caption with a text transformer, project both to a shared $d$-dimensional space, and $\ell_2$-normalize. Build the $N\times N$ cosine-similarity matrix; the diagonal entries are the true pairs. Train with a symmetric contrastive (InfoNCE) loss that treats each row and column as a classification problem: pick out the true match among $N$ candidates.

Stage 2 — latent autoencoder + conditioned diffusion training. Freeze (or lightly fine-tune) the autoencoder trained to compress images to latents with high fidelity reconstruction. Train the diffusion U-Net to predict the noise added to a latent at a random timestep $t$, conditioned on the CLIP text embedding of the corresponding caption, injected via cross-attention layers interleaved with the U-Net's convolutional blocks. During training, the text conditioning is randomly dropped (replaced with a null/empty embedding) some fraction of the time, so the model learns both a conditional and unconditional denoiser — this is what enables classifier-free guidance at sampling time.

Stage 3 — sampling (inference). This is the part users invoke. Encode the prompt with CLIP's text encoder once. Initialize a latent as pure Gaussian noise. Run a fixed number of denoising steps; at each step, query the U-Net twice (conditional and unconditional), combine the two predictions with the guidance scale, and step the latent toward a less noisy state using a diffusion sampler (DDPM, DDIM, or a faster ODE solver). After the final step, decode the latent to pixels with $\mathcal{D}$. ```pseudocode

# ---- Stage 1: CLIP training (schematic, run once) ---- function TRAIN_CLIP(image_batch, text_batch, temperature): I = normalize(ImageEncoder(image_batch)) # N x d T = normalize(TextEncoder(text_batch)) # N x d logits = (I @ transpose(T)) / temperature # N x N similarity matrix labels = [0, 1, ..., N-1] # diagonal = matching pairs loss_i2t = CrossEntropy(logits, labels) # image finds its caption loss_t2i = CrossEntropy(transpose(logits), labels) # caption finds its image return (loss_i2t + loss_t2i) / 2

# ---- Stage 2: latent diffusion training (run once, prompt-conditioned) ---- function TRAIN_LATENT_DIFFUSION(image, caption, encoder E, unet, clip_text_encoder): z0 = E(image) # compress to latent, e.g. 64x64x4 t = uniform_random(1, T_max) eps = sample_gaussian(shape_of(z0)) z_t = sqrt(alpha_bar[t]) z0 + sqrt(1 - alpha_bar[t]) eps # forward noising if random() < p_drop: # classifier-free guidance training trick c = NULL_EMBEDDING else: c = clip_text_encoder(caption) eps_hat = unet(z_t, t, c) # cross-attention conditioning return mean_squared_error(eps_hat, eps)

# ---- Stage 3: sampling (what runs when a user submits a prompt) ---- function GENERATE_IMAGE(prompt, unet, decoder D, clip_text_encoder, steps, guidance_scale w): c = clip_text_encoder(prompt) c_null = NULL_EMBEDDING z = sample_gaussian(latent_shape) # pure noise latent for t in reversed(schedule(steps)): # e.g. 50 DDIM steps eps_cond = unet(z, t, c) eps_uncond = unet(z, t, c_null) eps_guided = eps_uncond + w * (eps_cond - eps_uncond) # classifier-free guidance z = sampler_step(z, eps_guided, t) # e.g. DDIM update rule image = D(z) # single decode at the end return image ``` The cross-attention mechanism inside the U-Net is what actually lets text steer spatial layout. At each resolution level, the U-Net's spatial feature map (queries) attends over the sequence of CLIP text token embeddings (keys and values), so different spatial regions of the latent can attend more strongly to different words — this is how "a red cube on top of a blue sphere" can bind color words to distinct objects, though this binding is imperfect (see pitfalls below).

Why latent diffusion is cheaper. If pixel-space diffusion runs the U-Net on an $H \times W \times 3$ tensor and the self-attention / convolutional cost scales roughly with the number of spatial positions $HW$ (linearly for convolutions, and for attention layers quadratically in the number of tokens, i.e. $O((HW)^2)$ per attention layer), then compressing to a latent of size $\frac{H}{f}\times\frac{W}{f}\times c$ (with downsampling factor $f$, typically 8) reduces the spatial position count by $f^2$ — a 64x reduction in pixel count for $f=8$. Since this factor compounds across every one of the many U-Net forward passes (typically 20–1000 depending on sampler), and since attention cost scales quadratically with token count, the savings are substantial: this is precisely why latent diffusion made high-resolution text-to-image generation tractable on consumer hardware, whereas naïve pixel-space diffusion at the same resolution required far larger compute budgets.

Time complexity of sampling. Let $S$ be the number of sampling steps, and let $C(n)$ be the cost of one U-Net forward pass over a latent with $n$ spatial positions. Total sampling cost is $O(S \cdot C(n))$, doubled for classifier-free guidance because both a conditional and unconditional forward pass are needed per step (some implementations batch these into one call, but the FLOPs are the same). Faster samplers (DDIM, DPM-Solver) reduce $S$ from the thousand-step regime of the original DDPM formulation to 20–50 steps by treating the reverse process as an approximate ODE solve rather than a full stochastic Markov chain, cutting wall-clock time by roughly an order of magnitude with modest quality cost.

Space complexity. Memory is dominated by the U-Net's activations, scaling with $n$ (latent spatial size) and network width; operating on the $f=8$-downsampled latent instead of full pixels cuts activation memory by the same $f^2$ factor for convolutional layers. CLIP encoders are comparatively small and run once per prompt, not once per step, so their cost is negligible against the $S$-times-repeated U-Net cost.

Correctness of CLIP's contrastive objective. The InfoNCE loss used to train CLIP is a lower bound argument: minimizing it is equivalent to maximizing a lower bound on the mutual information between the image and text modalities, because the softmax classification task (identify the true pair among $N$ distractors) is only solvable well if the embedding correctly encodes shared semantic content. As $N$ grows, this bound tightens, which is why CLIP is trained with very large batches — the harder discrimination task forces a richer shared representation.

Correctness of the diffusion reverse process. The forward process $q(z_t \mid z_{t-1})$ is defined as fixed, small Gaussian noise addition; the joint forward chain has a closed-form marginal $q(z_t \mid z_0) = \mathcal{N}(\sqrt{\bar\alpha_t}z_0,\, (1-\bar\alpha_t)I)$, obtained by the reproductive property of Gaussians composed over $t$ steps. The reverse process is trained to approximate $q(z_{t-1}\mid z_t, z_0)$, which is itself Gaussian (a standard Bayesian-Gaussian computation), by matching its mean via a network that predicts the noise $\epsilon$. Because the true reverse conditional is Gaussian whenever the forward step's noise is small, a sufficiently expressive network minimizing the variational bound (equivalent to the simple MSE noise-prediction loss shown in the pseudocode) provably approximates the true posterior in the limit of infinitesimal steps — this is the same convergence argument underlying Diffusion Models for Images generally, and latent diffusion inherits it unchanged because the noising process is identical, just applied in a different, lower-dimensional coordinate system.

Correctness of classifier-free guidance. Guidance computes $\epsilon_{\text{uncond}} + w(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})$. Since $\epsilon$ is (up to a known scaling) the negative gradient of the log-density with respect to the noisy latent, this is a linear combination in score-function space:

which, rearranged, is a sample from a distribution proportional to $p(z_t)\,p(c\mid z_t)^w$ — an implicit classifier term raised to a power $w$. Setting $w>1$ therefore sharpens the influence of the (implicit) classifier $p(c \mid z_t)$ beyond its natural strength, which explains empirically why larger $w$ increases prompt fidelity at the cost of sample diversity and can push samples toward higher-likelihood-under-classifier but lower-likelihood-under-data-distribution regions (oversaturated colors, loss of variety).

Tradeoffs, stated honestly. The autoencoder introduces a lossy compression step: fine detail (small text, specific facial identity, fingers) can be corrupted or blurred in the compress/decompress round trip, independent of how well the diffusion process itself performs. Guidance improves prompt fidelity but is not free — pushed too high it produces artifacts and reduced diversity; pushed too low the model may ignore parts of the prompt. Neither CLIP similarity nor the diffusion loss directly optimizes for compositional correctness (attribute binding, counting, spatial relations), so these remain weak points regardless of how well each subsystem is individually trained.

Consider one denoising step near the end of sampling (small residual noise), for a toy latent of size $2\times2\times1$ (real models use e.g. $64\times64\times4$; the arithmetic is identical, just larger). Suppose the guidance scale is $w = 7.5$, and at timestep $t$ the U-Net returns two noise-prediction latents:

Step 1 — compute the difference (the "conditioning direction"):

Step 2 — apply classifier-free guidance with $w=7.5$:

Note the extrapolation: the guided prediction is not between the conditional and unconditional predictions, it overshoots past the conditional one in the direction the prompt pushes — exactly the score-function argument from the previous section.

Step 3 — DDIM-style update. Suppose $\bar\alpha_t = 0.20$ and $\bar\alpha_{t-1} = 0.35$ (noise schedule values decreasing as $t$ decreases), and the current noisy latent is:

First recover the predicted clean latent estimate:

With $\sqrt{\bar\alpha_t} = \sqrt{0.20}\approx 0.447$ and $\sqrt{1-\bar\alpha_t}=\sqrt{0.80}\approx0.894$: for the top-left entry: $\hat z_0[0,0] = (0.9 - 0.894 \times 0.62)/0.447 = (0.9 - 0.554)/0.447 \approx 0.774$ Then the DDIM step moves to $z_{t-1}$ by re-noising $\hat z_0$ with the new schedule's noise level and the same predicted $\epsilon_{\text{guided}}$:

for the top-left entry: $z_{t-1}[0,0] = \sqrt{0.35}\times 0.774 + \sqrt{0.65}\times 0.62 \approx 0.458 + 0.500 = 0.958$ Repeating this arithmetic entrywise across the full latent grid and across all $S$ scheduled timesteps (e.g. 50 steps from $t=1000$ down to $t=0$ on a strided schedule) produces the final clean latent $z_0$, which is then passed once through the decoder $\mathcal{D}$ to produce the output image. The table below summarizes the shapes and costs at each pipeline stage for a realistic configuration:

The path to this architecture runs through three separate lineages: contrastive representation learning, diffusion-based generative modeling, and perceptual image compression, which were unified into a single practical system only around 2021–2022.

Latent diffusion conditioned on CLIP-style embeddings underlies most consumer and creative-industry text-to-image tools in production today: Stable Diffusion and its many fine-tuned derivatives, Midjourney's rendering pipeline, and (with variations in the exact conditioning and diffusion backbone) DALL·E 2/3 and Google's Imagen family. Beyond pure image generation, the same recipe — a frozen or lightly adapted latent diffusion backbone plus cross-attention conditioning — has been extended to image editing and inpainting (conditioning on a masked image plus text), to ControlNet-style structural conditioning (conditioning on edges, depth maps, or poses in addition to text), and to text-to-video and text-to-3D pipelines that reuse the same denoising machinery over additional temporal or volumetric dimensions. Industrially, the CLIP component is reused far beyond image generation: as a zero-shot image classifier, as a retrieval embedding for text-to-image and image-to-image search, and as an automatic evaluation metric (CLIP score) for judging how well a generated image matches its prompt. The broader architectural pattern — a general-purpose joint embedding model supplying a conditioning signal to a separately trained generative backbone — recurs throughout the emerging space of Vision-Language Models and is a direct stepping stone toward unified Multimodal Models that jointly reason over and generate text, images, audio, and video within a single system.

Ask the tutor: How does the choice of noise schedule (linear vs. cosine) affect the quality and speed of the reverse diffusion process?