Defenses: ASLR, DEP, and Stack Canaries

A memory-corruption exploit is a chain of assumptions. The attacker assumes they can overflow a buffer to overwrite adjacent memory (the vulnerability itself, discussed under Code Injection Attacks). Then they assume they know where their payload or a useful gadget lives (an addressing assumption). Then they assume the memory holding their payload is executable (an execution assumption). Finally, classic overflows assume the corruption of control data — like a return address — goes unnoticed until it is used (an integrity assumption).

Stack canaries attack the integrity assumption. The compiler places a secret, unpredictable value — the canary — directly below the saved return address on the stack. Any linear buffer overflow that reaches the return address must first overwrite the canary. Before the function returns, generated epilogue code re-reads the canary and compares it to the known-good value; a mismatch triggers immediate program termination rather than a return to attacker-controlled code.

DEP (Data Execution Prevention), also called NX (No-eXecute) or W^X (write XOR execute), attacks the execution assumption. Using a hardware feature — a no-execute bit in the page table entry — the operating system marks stack, heap, and other writable data regions as non-executable. Injected shellcode may still land in memory, but the CPU refuses to fetch instructions from that page, raising a fault instead.

ASLR attacks the addressing assumption. At process load time, the OS randomizes the base addresses of the stack, the heap, shared libraries, and (when the binary is compiled Position-Independent, PIE) the main executable image itself. An attacker who does not know these addresses cannot hardcode a return-to-libc target or a Return-Oriented Programming gadget chain, because the same exploit code will land at a different offset on every run — and almost certainly crash the process instead of hijacking it.

Stack canary mechanism. The compiler inserts prologue and epilogue instrumentation into every function whose stack frame contains a vulnerable buffer (heuristically: local arrays, especially of `char`). At process start, a random canary value is generated once — often derived from `/dev/urandom` — and stored in a fixed location (a thread-local `fs`/`gs` segment slot on Linux/x86, for example). Every protected function copies that master value onto its own stack frame at prologue time and re-checks it at epilogue time. ``` // Compiler-generated instrumentation (conceptual) GLOBAL_CANARY = read_random_word() // set once at process/thread start function f(args...): // ---- prologue ---- frame.canary = GLOBAL_CANARY // copy secret onto stack allocate_locals() // e.g., char buf[64] // ---- function body ---- do_work(args) // vulnerable code may run here // ---- epilogue ---- if frame.canary != GLOBAL_CANARY: call __stack_chk_fail() // abort: log + terminate else: return normally ``` A linear overflow of `buf` writes forward in memory toward the saved frame pointer and return address. Because the canary sits between the buffer and the return address, the overflow must overwrite the canary first — corrupting it — before it can reach the return address, and the epilogue check catches this before control transfer occurs.

DEP/NX mechanism. This is enforced by the memory management unit (MMU), not the compiler. The OS loader sets the NX bit in the page table entry for every page that is writable but should not be executable — the stack, the heap, and `.data`/`.bss` segments. Code pages (`.text`) are marked executable but read-only. ``` // Conceptual page table entry PTE { physical_frame: ... present: 1 writable: W no_execute: X // hardware NX/XD bit } // Loader policy (W^X) for each segment in binary: if segment.is_code: set(writable = 0, no_execute = 0) // R-X else: set(writable = 1, no_execute = 1) // RW-, never executable ``` When the instruction pointer attempts to fetch from a page with `no_execute = 1`, the CPU raises a general protection / page fault before executing a single injected instruction. The OS handler treats this as a fatal security violation and kills the process.

ASLR mechanism. At `exec()` time, the OS kernel chooses random offsets for each major memory region and applies them when constructing the process's virtual address space. ``` // Conceptual process loader logic function load_process(binary): stack_base = STACK_BASE_DEFAULT + random_offset(entropy_bits_stack) heap_base = HEAP_BASE_DEFAULT + random_offset(entropy_bits_heap) lib_base = LIB_MMAP_BASE + random_offset(entropy_bits_mmap) if binary.is_PIE: exe_base = PIE_BASE_DEFAULT + random_offset(entropy_bits_pie) else: exe_base = FIXED_LOAD_ADDRESS // legacy non-PIE: no randomization map_segments(binary, exe_base) map_stack(stack_base) map_heap(heap_base) for each shared_library in binary.dependencies: map_library(shared_library, lib_base + running_offset) return entry_point(exe_base) ``` Randomization typically happens once per process launch (or once per `exec`), not per instruction — so within a single running process, addresses are stable, but they differ across runs and across machines, defeating hardcoded exploit addresses.

Stack canary. Time overhead: O(1) additional instructions per protected function call — one write in the prologue, one compare-and-branch in the epilogue. Empirically this is a small constant-factor slowdown (typically a few percent) since it does not scale with buffer size or program size. Space overhead: O(1) extra word per stack frame. Correctness argument: the canary is placed at a fixed offset directly adjacent to (below, in growth order) the saved return address, chosen by the compiler's stack layout algorithm. A linear, contiguous overflow that starts inside the buffer and grows toward higher addresses must pass through the canary's memory location before reaching the return address's location. Since the canary value is secret (drawn from a high-entropy source, typically ≥32 bits, and containing a null byte to also break naive C-string copy overflows), the probability an attacker overwrites it with the correct value by guessing is at most $2^{-b}$ for a $b$-bit canary, assuming no leak of the canary itself. This is why the invariant "corruption reaching the return address implies corruption of the canary" holds for the overflow class canaries were designed against — but the invariant does not hold for overflows that jump directly to a target (arbitrary write primitives, format string writes, or overflows of a separate stack variable that is not adjacent to the canary).

DEP/NX. Time overhead: effectively zero in the steady state — the check is a single hardware bit test performed by the MMU on every instruction fetch, already part of normal page-table-walk hardware. Space overhead: zero additional memory; NX reuses an existing bit in the page table entry format. Correctness argument: the invariant enforced is "no page is simultaneously writable and executable" (W^X). Because the CPU — not software — checks the NX bit on every fetch, this invariant is enforced even against arbitrary corruption of kernel data structures the attacker cannot directly touch from user space. The guarantee is airtight for the specific attack of injecting fresh instruction bytes and jumping to them. It says nothing about reusing already-executable, already-present code — the entire premise behind return-to-libc and Return-Oriented Programming, which route control flow through legitimate `.text` pages that are (correctly) marked executable.

ASLR. Time overhead: negligible — a handful of extra random-number-generation and offset-arithmetic operations at process load time, not per-instruction. Space overhead: zero extra memory; it changes where things are mapped, not how much. Correctness argument (entropy math): Let a region's base address be randomized over $2^{n}$ possible aligned positions ("n bits of entropy" — on 64-bit Linux, stack entropy is commonly around 28–33 usable bits depending on configuration; on 32-bit systems it can be as low as 8 bits for some regions). If an attacker must guess the correct base to build a working exploit and has no other information, the probability of a single blind guess succeeding is

Crucially, ASLR's real-world strength depends on whether failed guesses are cheap to retry. In a forking network daemon that doesn't re-randomize per connection (a "brute-forceable" service, historically exploited via `fork()`-server ASLR bypass), an attacker can attempt $k$ guesses; if each wrong guess merely crashes and restarts the child (inheriting the same randomized base from the parent), the effective success probability across $k$ attempts is

which is why low-entropy 32-bit ASLR (n ≈ 8–16 bits) is brute-forceable in seconds to minutes, while 64-bit ASLR with n ≈ 28+ bits makes blind brute force impractical — but not impossible in principle, and trivial if an information leak reveals even one address in the target region (since offsets within a region are typically fixed, one leaked pointer defeats that region's ASLR entirely).

Honest tradeoffs. None of these is a proof of security; each is a probabilistic or structural obstacle. Canaries only guard sequential overflows that must cross their memory location — they do nothing against arbitrary-write bugs, heap overflows into function pointers, or non-adjacent stack variable corruption. DEP does nothing against code-reuse attacks that never inject new code. ASLR does nothing once any pointer is leaked, and does nothing against attacks that require no addresses at all (like some data-only attacks). This is why practitioners describe them collectively as raising the bar, not closing the door — the natural segue into why Return-Oriented Programming emerged as a direct response to DEP, and why info leaks became a first-class exploit primitive as a direct response to ASLR.

Consider a vulnerable C function compiled with all three mitigations enabled (`-fstack-protector-all`, NX enabled by the loader, ASLR enabled by the kernel): ```c void handle_request(char *input) { char buf[64]; strcpy(buf, input); // classic unbounded copy — the vulnerability } ``` Trace the attacker's attempt, step by step, against the fully mitigated binary:

Row 7 is the point of the whole exercise: all three defenses were individually sound and individually bypassed one at a time, each bypass requiring an additional capability beyond the original overflow — a canary leak, an already-present-code target, and an address leak, respectively. That escalation, needing a leak plus a code-reuse chain, is precisely the machinery formalized in Return-Oriented Programming.

Every mainstream operating system and compiler toolchain in production today implements all three defenses by default. GCC and Clang enable stack-protector variants (`-fstack-protector-strong` is default on many Linux distributions); MSVC ships `/GS` by default on Windows. The Linux kernel, Windows, macOS, iOS, and Android all implement ASLR for user-space processes and increasingly for kernel space itself (KASLR). Hardware NX/XD support is universal on modern x86-64 and ARM64 CPUs, and OS loaders universally apply W^X policy to process memory maps. Beyond general-purpose OS security, these mitigations are foundational building blocks referenced by security certifications, secure coding standards (CERT C, MISRA), and compliance regimes for embedded and automotive software. Browsers (Chrome, Firefox, Safari) layer additional JIT-hardening and Control-Flow Integrity on top of these baseline defenses precisely because JIT-compiled code creates writable-and-executable regions that undermine plain W^X — a tension that motivates more advanced isolation techniques. Mobile platforms (iOS, Android) combine ASLR/DEP/canaries with code signing and hardware-backed pointer authentication (ARM PAC) as an additional layer. Understanding exactly what these three mitigations do — and precisely what they leave open — is the direct prerequisite for understanding both Return-Oriented Programming, the technique developed specifically to survive DEP, and modern Sandboxing and Isolation architectures, which assume individual mitigations will eventually be bypassed and instead confine the blast radius of a successful exploit.

Ask the tutor: How does a leaked stack or heap pointer get turned into a full working ROP exploit chain in practice?