DALL-E and Text-to-Image Generation

Text-to-image generation with DALL-E involves three interacting systems working in sequence. Understanding each piece builds a complete picture.

Step 1 — Encoding the Text Prompt

Before anything visual happens, the text prompt must be converted into a form the image model can understand. DALL-E 2 uses CLIP (Contrastive Language–Image Pre-training), a model trained to align text and image representations in a shared embedding space.

Explanation: This cosine similarity score measures how closely a text embedding and an image embedding point in the same direction in the shared semantic space — a score near 1 means the text and image strongly agree.

• $\vec{e}_T$ — the vector embedding of the text prompt, produced by a text encoder (typically a Transformer)
• $\vec{e}_I$ — the vector embedding of an image, produced by an image encoder (typically a Vision Transformer)
• $\cdot$ — the dot product (sum of element-wise products)
• $\|\cdot\|$ — the vector's magnitude (length)

CLIP is trained by pushing matching text-image pairs to have high similarity and mismatched pairs to have low similarity, across hundreds of millions of internet examples.

Step 2 — The Diffusion Process

Modern DALL-E uses a diffusion model to generate images. During training, the model learns to reverse a noise-adding process.

Explanation: This describes how a clean image $x_0$ is progressively corrupted into pure noise $x_t$ at timestep $t$ — the model's job is to learn the reverse journey, from noise back to clean image.

• $x_0$ — the original clean image
• $x_t$ — the noisy image at timestep $t$
• $\bar{\alpha}_t$ — a scheduled coefficient that shrinks from ~1 (little noise) to ~0 (pure noise) as $t$ increases
• $\epsilon$ — random Gaussian noise sampled fresh each time
• $\mathcal{N}(0, I)$ — the standard normal distribution (zero mean, unit variance)

At generation time, the model starts from pure noise and calls a neural network (U-Net architecture) repeatedly, each call removing a little noise while being nudged by the text embedding.

Step 3 — Classifier-Free Guidance

To make the output closely match the text prompt, DALL-E uses a technique called classifier-free guidance that amplifies the text's influence on generation:

Explanation: The guided noise prediction steers the generation away from "no text condition" and toward "with text condition," with the guidance scale $w$ controlling how strongly the prompt dominates.

• $\tilde{\epsilon}_\theta$ — the final guided noise estimate used to update the image
• $\epsilon_\theta(x_t, \emptyset)$ — model's noise prediction with no text prompt (unconditional)
• $\epsilon_\theta(x_t, c)$ — model's noise prediction with text prompt $c$ (conditional)
• $w$ — guidance scale (typically 7–15); higher values give more prompt-faithful but less diverse images
• $c$ — the encoded text condition (CLIP embedding of your prompt)

A Simple Python Illustration

This snippet shows the conceptual loop of diffusion inference using the 🤗 Diffusers library: ```python from diffusers import StableDiffusionPipeline import torch

# Load a pretrained diffusion model (similar architecture to DALL-E 2) pipe = StableDiffusionPipeline.from_pretrained( "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16 ) pipe = pipe.to("cuda") prompt = "a red panda astronaut floating in space, digital art"

# guidance_scale is the 'w' from equation [3]

# num_inference_steps controls how many denoising steps to run image = pipe( prompt, guidance_scale=7.5, # higher → more prompt-faithful num_inference_steps=50, # more steps → higher quality, slower ).images[0] image.save("red_panda_astronaut.png") ``` ```python

# You can also visualize what the noise schedule looks like import numpy as np import matplotlib.pyplot as plt T = 1000 # total timesteps betas = np.linspace(1e-4, 0.02, T) # linear noise schedule alphas = 1 - betas alpha_bar = np.cumprod(alphas) # cumulative product = ᾱ_t plt.figure(figsize=(8, 3)) plt.plot(alpha_bar, color='steelblue') plt.xlabel("Timestep t") plt.ylabel("ᾱ_t (signal retained)") plt.title("How much of the original image survives at each noise level") plt.tight_layout() plt.savefig("noise_schedule.png") ``` The loop in inference: start with random Gaussian noise, run ~50 denoising steps guided by your prompt embedding, and out comes an image.