Interpretability and Adversarial Examples

Interpretability in deep learning means producing a human-understandable explanation for a model's prediction. Feature attribution is the dominant paradigm: assign each input feature (pixel, word, tabular column) a real-valued score indicating how much it contributed to the output. Gradient-based methods compute $\partial f_c(x) / \partial x_i$, the sensitivity of the class-$c$ logit or probability to feature $i$, exploiting the same backpropagation machinery used for training. Perturbation-based methods instead treat the network as a black box, measuring $f_c(x) - f_c(x \setminus i)$, the change in output when feature $i$ is removed or altered, possibly averaged over many subsets (as in Shapley-value methods). An adversarial example is an input $x' = x + \delta$ such that $\delta$ is small under some norm ($\|\delta\|_\infty \le \epsilon$ is standard), $x'$ looks unchanged to a human, yet the model's prediction on $x'$ differs from its prediction on $x$ (untargeted attack) or matches a specific attacker-chosen class (targeted attack). Adversarial examples exist because a deep network's decision boundary, while achieving low error on the data distribution, can pass surprisingly close to individual data points in directions orthogonal to the manifold of "natural" variation. The network is locally almost-linear in many directions (a consequence of piecewise-linear activations like ReLU and near-linear layers), so a well-aligned perturbation of tiny magnitude, when summed over thousands of input dimensions, produces a large change in the pre-activation logits — this is the linear explanation of adversarial vulnerability. The connective tissue between the two topics is the gradient of the loss with respect to the input, $\nabla_x \mathcal{L}(f(x), y)$. Saliency maps visualize this gradient (or a variant of it) to explain a prediction. Adversarial attacks use this same gradient, but instead of visualizing it, they walk along it to change the prediction. This shared object explains why saliency maps can be brittle (a small adversarial-style perturbation that leaves the prediction almost unchanged can drastically alter the saliency map) and why the search for robust interpretability and adversarial robustness are studied together under the umbrella of GAN Training Dynamics and Failure Modes-adjacent research on stable, trustworthy generative and discriminative models.

We separate the mechanism into three pipelines: (1) gradient-based attribution, (2) perturbation-based attribution, and (3) adversarial example generation, followed by (4) the standard defense.

1. Gradient (vanilla) saliency. For input $x \in \mathbb{R}^d$ and target class $c$, compute

a single backward pass. Variants smooth this out:

• SmoothGrad: average the gradient over many noisy copies of $x$ to reduce visual noise.
• Integrated Gradients: integrate the gradient along a straight-line path from a baseline $x_0$ (e.g., all-black image) to $x$, which satisfies a completeness axiom (attributions sum exactly to $f_c(x) - f_c(x_0)$).
• Grad-CAM: use gradients flowing into the last convolutional layer, weighted and summed with the layer's activations, to produce a coarse class-discriminative heatmap.

2. Perturbation-based attribution.

• Occlusion: slide a gray/black patch over the image, recording the drop in $f_c$ at each location; large drops indicate important regions. Cost is one forward pass per patch position, $O(d/k^2)$ forward passes for patch size $k$.
• LIME: sample perturbed versions of $x$ (e.g., randomly zero out superpixels), fit a local, interpretable linear (or sparse linear) model to the black-box outputs on these samples, and read off the linear model's weights as attributions.
• SHAP: define attribution via Shapley values from cooperative game theory — the unique attribution scheme satisfying efficiency, symmetry, dummy, and additivity axioms, approximated because exact computation is $O(2^d)$.

3. Adversarial example generation. Fast Gradient Sign Method (FGSM) — a single step in the direction that locally maximizes the loss:

This is the first-order (linear) approximation to the constrained maximization $\max_{\|\delta\|_\infty \le \epsilon} \mathcal{L}(f(x+\delta), y)$: linearizing $\mathcal{L}$ around $x$, $\mathcal{L}(x+\delta) \approx \mathcal{L}(x) + \delta^\top \nabla_x \mathcal{L}$, and the $\delta$ maximizing $\delta^\top g$ subject to $\|\delta\|_\infty \le \epsilon$ is exactly $\epsilon \cdot \mathrm{sign}(g)$ (Hölder duality between $\ell_\infty$ and $\ell_1$). Projected Gradient Descent (PGD) — the iterative, stronger version, essentially multi-step FGSM with a projection back onto the $\epsilon$-ball after every step:

PGD is widely treated as a near-worst-case first-order attack and is the standard benchmark for "adversarial robustness."

4. Adversarial training (defense). Replace the standard empirical risk minimization objective with a robust (min-max) objective:

The inner maximization is approximated at each training step using PGD (or FGSM for speed), and the outer minimization is a standard gradient step on $\theta$ using the adversarial example in place of the clean one. ```

# ---- Gradient saliency map ---- function GRADIENT_SALIENCY(model, x, target_class): x.requires_grad = True logits = model.forward(x) loss = logits[target_class] loss.backward() # one backprop pass return abs(x.grad) # same shape as x

# ---- Occlusion-based attribution ---- function OCCLUSION_MAP(model, x, patch_size, baseline_value): base_score = model.forward(x)[target_class] heatmap = zeros_like(x) for each top-left position p in stride grid over x: x_occluded = copy(x) x_occluded[patch at p] = baseline_value score = model.forward(x_occluded)[target_class] heatmap[patch at p] += (base_score - score) # accumulate drop return heatmap # normalize by overlap count if strided

# ---- FGSM attack ---- function FGSM(model, x, y, epsilon): x.requires_grad = True loss = cross_entropy(model.forward(x), y) grad = gradient(loss, x) x_adv = x + epsilon * sign(grad) x_adv = clip(x_adv, valid_input_range) # e.g. [0,1] for images return x_adv

# ---- PGD attack ---- function PGD(model, x, y, epsilon, alpha, num_steps): x_adv = x + uniform_noise(-epsilon, epsilon) # random start (recommended) x_adv = clip(x_adv, valid_input_range) for t in 1..num_steps: x_adv.requires_grad = True loss = cross_entropy(model.forward(x_adv), y) grad = gradient(loss, x_adv) x_adv = x_adv + alpha * sign(grad) delta = clip(x_adv - x, -epsilon, epsilon) # project onto L_inf ball x_adv = clip(x + delta, valid_input_range) # project onto valid range return x_adv

# ---- Adversarial training (outer loop) ---- function ADVERSARIAL_TRAIN(model, data_loader, epsilon, alpha, num_steps, epochs): for epoch in 1..epochs: for (x, y) in data_loader: x_adv = PGD(model, x, y, epsilon, alpha, num_steps) # inner max loss = cross_entropy(model.forward(x_adv), y) # outer min update_parameters(model, gradient(loss, model.parameters)) return model ```

Gradient saliency. One forward pass plus one backward pass, i.e., the same $O(\text{cost of one training step})$ as ordinary backpropagation — typically $O(L \cdot w^2)$ time for an $L$-layer network with width $w$ per layer (dominated by matrix multiplications), and $O(L \cdot w)$ auxiliary memory to store activations for the backward pass. This is why gradient saliency is cheap and ubiquitous. Correctness caveat: the raw gradient is only a first-order local approximation; it does not certify that the highlighted pixels are causally necessary, only that infinitesimal perturbation there changes the logit fastest. Saturating activations (e.g., a ReLU firmly in its flat regime, or a softmax near 0/1) can make the gradient vanish for features that are nonetheless important, motivating Integrated Gradients, which fixes this via the fundamental theorem of calculus:

guaranteeing $\sum_i \text{IG}_i(x) = f_c(x) - f_c(x_0)$ exactly (completeness), at the cost of $O(m)$ forward/backward passes for an $m$-step Riemann approximation of the integral.

Occlusion. $O(d/k^2)$ forward passes for a $d$-dimensional input and $k\times k$ patches (or $O(d)$ if occluding one feature at a time), each an $O(\text{one forward pass})$ cost — so total time $O((d/k^2)\cdot \text{fwd-cost})$. No gradient needed, so it works for non-differentiable models too, but it can miss feature interactions: occluding two jointly-necessary pixels separately may show no effect individually.

LIME/SHAP. LIME costs $O(N)$ black-box queries for $N$ sampled perturbations, plus fitting a linear model, $O(N \cdot d')$ for $d'$ interpretable features (e.g., superpixels) — cheap and model-agnostic, but the explanation depends on the sampling distribution and kernel width, hence not guaranteed stable across random seeds. Exact Shapley values require evaluating the model on all $2^d$ feature subsets (an exponential-time coalitional game computation); this is proven to be the unique attribution satisfying efficiency, symmetry, dummy, and additivity, but practically approximated via sampling (KernelSHAP, $O(N)$ samples) or exploiting model structure (TreeSHAP, polynomial time for tree ensembles).

FGSM. $O(1)$ forward and backward pass — as cheap as one training step. It is a provably optimal single linear step under the $\ell_\infty$-ball constraint (by the Hölder-duality argument above), but because real networks are only locally, not globally, linear, one step often undershoots the true worst-case perturbation.

PGD. $O(T)$ forward/backward passes for $T$ steps — $T\times$ the cost of FGSM. PGD approximately solves the inner maximization $\max_{\|\delta\|_\infty\le\epsilon} \mathcal{L}(x+\delta,y)$, a non-convex problem in general (since $\mathcal{L}\circ f$ is non-convex in $x$), so PGD gives no global optimality guarantee, but empirically with random restarts and enough steps it is accepted as a strong proxy for the worst case, and is far harder to "gradient mask" against than FGSM.

Adversarial training. Each training step now costs $O(T)$ times a normal step (for the inner PGD loop), so total training time is $O(T \cdot \text{epochs} \cdot |\mathcal{D}|/B \cdot \text{fwd-bwd cost})$ for batch size $B$ — often 3–10× slower than standard training in practice ($T \in [3,10]$ typically for reasonable defenses via fast variants). Its correctness rests on solving a min-max game: if the inner maximization were solved exactly, the outer minimization would converge (under standard SGD assumptions) to parameters that are locally loss-robust within the $\epsilon$-ball around every training point — but since PGD only approximately solves the inner problem, the resulting robustness is only empirical, not certified, and can be broken by stronger, unseen attacks (a repeated historical pattern — see Pitfalls). Certified defenses (randomized smoothing, interval bound propagation) trade some accuracy for provable guarantees but are outside adversarial training's scope.

The tradeoff. There is a well-documented accuracy–robustness tradeoff: making a classifier robust to $\ell_\infty$-perturbations of radius $\epsilon$ tends to reduce clean accuracy, because the robust optimum forces the decision boundary to keep a margin of at least $\epsilon$ everywhere, which can conflict with fitting the true (non-robust but predictive) features present in the data — some features that are predictive for clean accuracy are, by construction, non-robust (small perturbations flip them), and adversarial training suppresses reliance on them.

Take a small logistic-regression-style linear classifier operating on a 4-pixel toy "image" $x = (x_1,x_2,x_3,x_4)$, with weight vector $w = (2, -1, 3, -4)$ for class "cat" (logit $z = w^\top x$, sigmoid probability). Suppose the true image is $x = (0.9, 0.1, 0.8, 0.2)$, giving

a confident "cat" prediction ($\sigma(3.3) \approx 0.964$).

Saliency: the gradient of $z$ with respect to $x$ is simply $w$ itself (linear model), so $|\nabla_x z| = (2, 1, 3, 4)$ — pixel 4 has the largest-magnitude gradient despite the smallest input value, because its weight is largest in magnitude. This already illustrates a subtlety: raw gradient saliency measures sensitivity, not the pixel's actual contribution to the current logit. The actual per-feature contribution to $z$ is $w_i x_i = (1.8, -0.1, 2.4, -0.8)$, which ranks pixel 1 and 3 as the true drivers of the positive logit — pixel 4 contributes negatively despite having the largest gradient magnitude. This gap between "sensitivity" and "contribution" is exactly why Integrated Gradients multiplies the gradient by $(x_i - x_{0,i})$ rather than reporting the gradient alone.

FGSM attack with $\epsilon = 0.05$, attacking the "cat" logit to push it toward "not cat" (untargeted, so we ascend the loss, i.e., move $x$ in the direction that decreases $z$, which is $-\mathrm{sign}(w)$ scaled by $\epsilon$ for a class where lower $z$ means the loss for label "cat" increases):

New logit:

The logit dropped from $3.3$ to $2.8$ ($\sigma(2.8)\approx 0.943$) — a real but modest change from one FGSM step, changing each pixel by at most $0.05$, imperceptible in an actual image with 8-bit intensities (a shift of about 13/255 gray levels).

PGD, run for $T=3$ steps with step size $\alpha = 0.03$ and the same $\epsilon=0.05$ box, would repeat this signed-gradient step three times, clipping to stay within $\pm 0.05$ of the original $x$ each time; since the model is linear here the gradient direction never changes, so PGD converges immediately to the same corner of the $\ell_\infty$ ball as one large FGSM step — $x^* = (0.85, 0.15, 0.75, 0.25)$ — illustrating that for (locally) linear models FGSM is already optimal, while for deep nonlinear networks the direction changes step to step and PGD typically finds a substantially harder (lower-loss-for-attacker) point than FGSM.

Now suppose adversarial training is applied: the model retrains using $(x', y=\text{cat})$ instead of $(x,y)$ at every step. Because $x'$ has systematically smaller cat-aligned features (the attack pushes toward $z=2.8$), the optimizer must increase reliance on the still-robust, high-weight features ($x_1, x_3$) and can no longer coast on borderline sensitivity to $x_2, x_4$; over many epochs the learned weights shift toward larger $|w_1|, |w_3|$ and smaller $|w_2|, |w_4|$, precisely the "suppress non-robust features" mechanism described in Complexity & correctness.

!A researcher's portrait placeholder representing the adversarial-examples and interpretability research lineage

Interpretability tooling built on saliency and attribution methods is embedded in production ML platforms (e.g., model cards and explanation dashboards in cloud ML services) and is a regulatory expectation in domains like credit scoring and medical diagnosis, where "right to explanation" rules require some human-interpretable justification for automated decisions. Radiology and pathology AI tools use Grad-CAM-style heatmaps so clinicians can sanity-check whether a model is looking at the lesion rather than a scanner artifact. Autonomous vehicle perception stacks and biometric authentication systems are stress-tested against adversarial patches and perturbations before deployment, since a single misclassified stop sign or spoofed face can have safety consequences. Financial fraud-detection and spam-filtering systems face adversaries who actively probe models for blind spots, making adversarial robustness a live security concern, not just an academic curiosity. Adversarial training and its variants are now standard benchmarks (e.g., RobustBench) used by researchers building on GAN Training Dynamics and Failure Modes-related generative defenses, gradients-based attack/defense loops, and general robust classification pipelines across vision, speech, and tabular domains.

Ask the tutor: How does the accuracy–robustness tradeoff change as the perturbation budget $\epsilon$ grows, and is there a principled way to choose $\epsilon$ for a given deployment?