Sandboxing and Isolation

A sandbox is a restricted execution environment that limits what a piece of code can observe or affect, regardless of what that code's own logic tries to do. Isolation is the general property that one execution context cannot access another's memory, files, or resources except through explicitly permitted channels. The two terms overlap heavily: sandboxing is isolation applied specifically to untrusted or partially trusted code, usually with an explicit policy describing exactly what is allowed (a network socket to one host, read access to one directory, a fixed set of syscalls) and everything else denied by default. The key conceptual move is separating mechanism from policy. The mechanism is whatever the hardware or OS actually enforces — a memory management unit refusing an out-of-range access, a kernel refusing a disallowed system call, a hypervisor trapping a privileged instruction. The policy is the human-authored description of what should be allowed for this particular untrusted component. Good sandbox design keeps the mechanism small and formally checkable (e.g., "the CPU's MMU disallows access to pages not mapped into this process") while the policy can be as fine-grained as needed (e.g., "this process may only call `read`, `write`, `exit`, and may only touch file descriptors it was handed at startup"). Isolation exists at multiple layers, each with a different granularity and trust boundary: 1. Hardware privilege levels (rings, or ARM exception levels) separate kernel from user code. 2. Process isolation gives each process its own virtual address space, enforced by the MMU. 3. OS-level sandboxing restricts a process's view of the system: namespaces virtualize what it can see (its own PID tree, filesystem mounts, network stack), and mechanisms like `seccomp-bpf` and capabilities restrict what it can do even within that view. 4. Language/runtime sandboxes (WebAssembly, JVM bytecode verification, JavaScript engines) enforce memory safety and API restriction inside a single OS process. 5. Hypervisor/VM isolation virtualizes hardware itself, so a compromised guest kernel still cannot touch the host or other guests.

Privilege separation is the architectural technique of decomposing one program into multiple components at different trust levels, connected by a narrow, well-defined interface — so that the component handling untrusted input runs with minimal privilege, and only a small, carefully audited component holds the powerful privileges (root, private keys, raw sockets) it occasionally needs.

The mechanics differ by layer, but they share a common pattern: create a distinct execution context, strip its default privileges down to (ideally) nothing, then explicitly grant back only what is needed. This is least privilege applied structurally rather than just to individual permission checks.

Process-level isolation (the foundation). Each process gets its own virtual address space via the MMU; page tables map that process's virtual addresses to physical frames it owns, and any access outside those mappings faults. This alone stops one buggy or malicious process from reading another's memory directly — but processes still share the kernel, the filesystem, and often inter-process channels (signals, shared memory, sockets), any of which can leak or be abused unless further restricted.

chroot and filesystem confinement. `chroot(path)` changes a process's apparent filesystem root, so paths like `/etc/passwd` resolve inside the jail rather than the real root. This is a weak, historically important primitive — it is not a security boundary against a privileged process, because root inside a chroot can often break out (e.g., via file descriptors opened before the chroot call, or `chroot` + `chdir(".."​)` tricks if not done carefully). It is the ancestor of modern container filesystem isolation but must never be relied on alone.

Namespaces (Linux) virtualize a specific global resource so a process sees its own private instance:

• PID namespace: process sees itself as PID 1, cannot see or signal processes outside.
• Mount namespace: private view of mounted filesystems.
• Network namespace: private network interfaces, routing table, ports.
• UTS namespace: private hostname.
• User namespace: maps container "root" (UID 0) to an unprivileged host UID — a compromise

inside the container is not actually root outside it.

• IPC namespace: private System V IPC / POSIX message queues.

Control groups (cgroups) limit and account for resources (CPU shares, memory, I/O bandwidth, number of processes) rather than visibility. Namespaces answer "what can you see?"; cgroups answer "how much can you use?" — together they are the two pillars containers are built on.

seccomp-bpf restricts which system calls a process may invoke at all, using a small BPF program evaluated on every syscall. A process that only needs `read`, `write`, `mmap`, and `exit` can be locked to exactly that set; any other syscall triggers a kill or an error, even if a memory-safety bug lets an attacker hijack control flow, because the kernel itself refuses the forbidden call before it executes.

Capabilities split the monolithic "root" privilege into ~40 independent bits (`CAP_NET_BIND_SERVICE`, `CAP_SYS_ADMIN`, `CAP_CHOWN`, …), so a process can be granted exactly the narrow slice of root's power it needs — e.g., binding to port 80 — without full root.

Privilege separation restructures an application itself: a small trusted monitor process retains high privilege and performs only a few, simple, auditable operations; a large worker process does all the complex, attack-surface-heavy parsing/networking with privileges dropped to the minimum, communicating with the monitor over a narrow IPC channel with a fixed message format. ```pseudocode // Generic privilege-separated server, modeled on OpenSSH's design function main(): listen_socket = bind_and_listen(PORT) while true: client_conn = accept(listen_socket) (monitor_pipe, worker_pipe) = create_pipe_pair() pid = fork() if pid == 0: // ---- WORKER: untrusted-input-facing, minimally privileged ---- close(monitor_pipe) drop_privileges_to_unprivileged_user() apply_seccomp_filter(ALLOWED_SYSCALLS_FOR_WORKER) chroot_or_mount_namespace(EMPTY_JAIL) worker_loop(client_conn, worker_pipe) exit(0) else: // ---- MONITOR: trusted, retains privilege, does almost nothing ---- close(worker_pipe) monitor_loop(monitor_pipe, pid) function worker_loop(conn, pipe): while true: request = read_from_untrusted(conn) // fully untrusted input if request.needs_privileged_op(): send(pipe, request.privileged_op) // ask monitor to act result = receive(pipe) // small, fixed-format reply respond_to_client(conn, result) else: handle_locally(conn, request) function monitor_loop(pipe, worker_pid): while worker_alive(worker_pid): op = receive(pipe) if not is_in_allowed_op_whitelist(op): // strict validation terminate(worker_pid); break result = perform_privileged_operation(op) // e.g., verify password, open file as root send(pipe, result) ``` The crucial design invariant: the monitor's logic must be small enough to audit by hand, must validate every request against a strict whitelist, and must never simply "forward" arbitrary privileged operations requested by the worker — otherwise the whole separation collapses back into one trust domain.

Sandboxing is a systems technique, not an algorithm, so "complexity" here means runtime overhead and policy-checking cost, and "correctness" means the isolation property actually holds against a defined threat model.

Overhead.

• Process/address-space isolation: essentially free at steady state (already required for

multitasking); the cost is context-switch overhead, $O(1)$ per switch dominated by TLB/cache effects, independent of sandboxing per se.

• Namespaces: near-zero runtime overhead once created — they change which kernel data structure

a syscall resolves against (e.g., which mount table), not how expensive the resolution is.

• cgroups: $O(1)$ accounting overhead per resource-consuming operation (a counter increment/limit

check), amortized into existing scheduler and allocator paths.

• seccomp-bpf: each syscall now runs a short BPF program before dispatch — $O(k)$ per syscall where

$k$ is the number of filter instructions (typically tens), i.e., a small constant added to every syscall; measured overhead is usually well under 5% for realistic filters.

• Full VM/hypervisor isolation: highest overhead, since privileged instructions and I/O are trapped

and emulated; modern hardware virtualization extensions (Intel VT-x/AMD-V) bring this down to single-digit percentages for compute-bound workloads but I/O-heavy workloads can see more.

Correctness argument. The right way to reason about a sandbox is as an invariant on reachable state, not a proof that the code inside behaves. Define the security policy as a set $A$ of allowed actions (syscalls, memory regions, files, network destinations). The mechanism is correct if it maintains the invariant:

checked by the kernel or hypervisor, not by the confined code itself. This is why sandboxing composes well with an attacker who controls the confined program's logic completely (e.g., via a memory-corruption exploit): the invariant is enforced at a layer the attacker's code cannot touch, as long as that enforcement layer has no bugs and the confined process cannot reach it through some side channel. This gives an honest tradeoff spectrum:

Notice the pattern: every isolation boundary reduces "trust the whole program" to "trust a smaller enforcement layer," and the residual risk is exactly the correctness of that layer. This is why container isolation is considered weaker than VM isolation — containers share one kernel, so any kernel bug reachable from a container's allowed syscalls is a potential escape, whereas a VM escape requires a bug in the (smaller, more scrutinized) hypervisor trap surface.

Trace a real, simplified container launch (conceptually matching what `runc`/Docker does) for running an untrusted web app that should only serve HTTP on port 8080 and write to `/data`.

Now suppose the app has a buffer overflow in its HTTP parser, and an attacker sends a crafted request that hijacks control flow and gets shellcode execution.

• ASLR/DEP/canaries (prerequisite defenses) may have already made this exploit hard to land

reliably — but assume it succeeded anyway.

• The attacker's shellcode tries `execve("/bin/sh")` → allowed by seccomp? If `execve` is not in

the allowed list (it typically is not needed post-startup), the kernel kills the process outright.

• The attacker tries to `ptrace` another process to steal secrets → denied by seccomp.
• The attacker tries `mount()` to remount `/` read-write and escape the filesystem jail → denied,

both by missing `CAP_SYS_ADMIN` and by the seccomp filter.

• The attacker tries to read `/etc/shadow` on the host → the mount namespace means that path does

not even exist in the container's view; only the minimal image's files are visible.

• The attacker tries to fork repeatedly to exhaust PIDs and pivot to a denial-of-service → capped

at 128 by the cgroup PID limit. The exploit succeeded at step "hijack control flow," yet is contained: every subsequent action the attacker's code attempts is checked against the invariant $a \in A$ from the correctness argument, and rejected. This is the entire point of defense in depth: sandboxing does not prevent the bug, it prevents the bug from mattering.

Sandboxing grew out of decades of OS research on protection domains, then was repeatedly re-invented at each new abstraction layer — time-sharing systems, virtual machines, mobile app stores, and containers — each time facing the same underlying problem: how much do you trust code you did not write?

• Browsers: Chrome, Firefox, and Edge run page rendering in heavily sandboxed processes

(seccomp-bpf on Linux, job objects on Windows, restricted entitlements on macOS), separate from a privileged browser process — directly mirroring the monitor/worker pattern above.

• Cloud infrastructure: multi-tenant clouds isolate customer workloads using VMs (strong

hypervisor boundary) or lightweight VMs like Firecracker (microVMs) that combine container-like startup speed with hypervisor-grade isolation, precisely because shared-kernel container isolation alone is judged too weak for mutually adversarial tenants.

• Mobile OSes: Android and iOS sandbox every app in its own UID/process with a restricted

filesystem view and mandatory-access-control policy (SELinux on Android), so a malicious app from an app store cannot read another app's data without an explicit, user-granted permission.

• Server software with historically large attack surfaces: OpenSSH's privilege-separated

daemon, `sudo`, and web servers running worker processes as `nobody` all apply the privilege separation pattern to bound the damage of parser and protocol bugs.

• Language-level sandboxes: WebAssembly runtimes and the JVM enforce memory safety and API

restriction inside a single process, letting untrusted plugins run with near-native speed while still being unable to forge pointers or call arbitrary syscalls directly.

• CI/CD and package managers: build systems run untrusted third-party build scripts inside

containers or VMs specifically to prevent a compromised dependency from exfiltrating credentials from the build host — a scenario increasingly targeted by real supply-chain attacks.

Ask the tutor: How would you design a seccomp-bpf allow-list for a specific real service, and what tradeoffs arise between strict security and application compatibility?