VPNs and DDoS Defense

A tunnel is a mechanism for carrying one network protocol's packets inside another protocol's packets, so that intermediate routers see only the outer envelope. Formally, if the original (inner) packet is $P$ with header $H_{inner}$ and payload $D$, a tunnel produces a new packet $P' = H_{outer} \Vert E_k(H_{inner} \Vert D)$, where $E_k$ is an authenticated encryption transform under a shared session key $k$. The outer header carries only enough information (source and destination tunnel endpoints) for ordinary IP routing to deliver $P'$. A VPN (Virtual Private Network) is a tunnel construction plus a key-management protocol that establishes $k$ securely, giving the two properties of confidentiality (an eavesdropper on the path cannot read $D$) and integrity/authenticity (a modified $P'$ is detected and dropped). Two dominant architectures exist: IPsec, which operates at the network layer and defines two sub-protocols — AH (Authentication Header, integrity only) and ESP (Encapsulating Security Payload, confidentiality plus integrity) — negotiated via IKE (Internet Key Exchange); and TLS/UDP-based VPNs such as OpenVPN and WireGuard, which build a virtual network interface (a TUN/TAP device) and tunnel IP packets inside an encrypted transport-layer or datagram-layer channel. A Denial of Service (DoS) attack degrades or denies a service's availability to legitimate users; a Distributed DoS (DDoS) attack does this using many source hosts simultaneously, typically a botnet, to multiply attack volume and to defeat simple source-IP filtering. DDoS attacks are classified by which resource they exhaust:

• Volumetric attacks flood the victim's link capacity itself (e.g., UDP floods, DNS/NTP amplification), measured in bits per second (bps).
• Protocol attacks exploit the state machines of connection-oriented protocols to exhaust finite server resources such as connection tables (e.g., TCP SYN flood, measured in packets per second, pps).
• Application-layer attacks send well-formed but resource-intensive requests (e.g., HTTP GET floods against expensive database queries) that look legitimate at lower layers but exhaust CPU, disk I/O, or application threads.

Defense is not a single mechanism but a pipeline: overprovision and distribute capacity (anycast, scrubbing centers), detect anomalies statistically, and apply the cheapest possible filter that removes attack traffic without dropping legitimate users. Because attackers can spoof source addresses and mimic legitimate traffic distributions, no filter is perfectly precise — the goal is to shift the cost asymmetry back in the defender's favor.

VPN tunnel establishment and per-packet processing

A VPN's lifecycle has two phases: a handshake that authenticates the parties and derives a shared session key via a protocol like TLS and Secure Channels or IKEv2's Diffie–Hellman exchange, and a data-transfer phase that encapsulates every packet under that key. ``` // Phase 1: Handshake (illustrated in WireGuard/IKEv2 style) function ESTABLISH_TUNNEL(local_identity, peer_identity): // 1. Ephemeral Diffie-Hellman exchange (my_priv, my_pub) = DH_GENERATE() send(my_pub, sign(local_identity, my_pub)) (peer_pub, peer_sig) = receive() if not VERIFY(peer_identity, peer_pub, peer_sig): abort("authentication failed") shared_secret = DH_COMPUTE(my_priv, peer_pub) // 2. Key derivation (HKDF or similar KDF) (k_enc_send, k_enc_recv, k_auth) = KDF(shared_secret, context_info) session = { send_key: k_enc_send, recv_key: k_enc_recv, send_counter: 0, recv_window: SLIDING_WINDOW(size = 64) } return session // Phase 2: Per-packet encapsulation (sender side) function TUNNEL_SEND(session, inner_packet): nonce = session.send_counter session.send_counter += 1 ciphertext, tag = AEAD_ENCRYPT(session.send_key, nonce, inner_packet) outer_packet = OUTER_HEADER(src = my_tunnel_ip, dst = peer_tunnel_ip) ++ nonce ++ ciphertext ++ tag physical_send(outer_packet) // Phase 2: Per-packet decapsulation (receiver side) function TUNNEL_RECEIVE(session, outer_packet): (outer_hdr, nonce, ciphertext, tag) = PARSE(outer_packet) if session.recv_window.already_seen(nonce): drop("replay detected") return plaintext = AEAD_DECRYPT(session.recv_key, nonce, ciphertext, tag) if plaintext == DECRYPT_FAILURE: drop("authentication failed") return session.recv_window.mark_seen(nonce) inject_into_local_stack(plaintext) // now routed as if arrived on trusted LAN ``` The key structural fact: the inner packet's own IP header is preserved unmodified inside the ciphertext. This is what makes the tunnel transparent — the corporate LAN's routing, addressing, and firewall rules apply exactly as if the remote host were physically local. AEAD (Authenticated Encryption with Associated Data) provides both secrecy and tamper-detection in one primitive, so a single decryption call rejects forged or modified packets rather than requiring a separate MAC-verify-then-decrypt step (which is exactly the design IPsec's ESP and WireGuard both use, differing mainly in ciphersuite and handshake protocol).

DDoS detection and mitigation pipeline

``` // Continuously running detection loop at a scrubbing center / edge router function DDOS_MONITOR(traffic_stream, baseline_profile): window = SLIDING_TIME_WINDOW(duration = 10s) for packet in traffic_stream: window.add(packet) if window.elapsed(): stats = COMPUTE_STATS(window) // stats include: pps, bps, SYN/ACK ratio, unique-source entropy, // per-destination request rate, protocol distribution anomaly_score = DEVIATION(stats, baseline_profile) if anomaly_score > THRESHOLD_HIGH: ACTIVATE_MITIGATION(window, stats) elif anomaly_score > THRESHOLD_LOW: ESCALATE_MONITORING() baseline_profile = UPDATE_EMA(baseline_profile, stats) // adapt slowly window.reset() function ACTIVATE_MITIGATION(window, stats): if stats.is_volumetric(): REROUTE_VIA_ANYCAST_SCRUBBING(window.destination) RATE_LIMIT(protocol = stats.dominant_protocol, cap = safe_baseline) if stats.is_syn_flood(): ENABLE_SYN_COOKIES(window.destination) if stats.is_application_layer(): CHALLENGE(window.suspect_sources) // CAPTCHA / JS proof-of-work / rate cap BLOCK(confirmed_bot_signatures) // Stateless SYN-flood countermeasure at the TCP layer function HANDLE_SYN(src_ip, src_port, dst_ip, dst_port, client_isn): // Instead of allocating a connection-table entry, encode state in the // sequence number itself, using a keyed cryptographic function t = current_time_slot() // e.g., 64-second granularity cookie = MAC(secret_key, src_ip, src_port, dst_ip, dst_port, t) mod 2^24 server_isn = (t << 24) | cookie // pack timestamp + cookie into 32-bit ISN send_SYNACK(server_isn) // Crucially: NO connection state is allocated yet. function HANDLE_ACK(src_ip, src_port, dst_ip, dst_port, ack_number): server_isn = ack_number - 1 t = server_isn >> 24 if not RECENT(t): drop("expired cookie") return expected_cookie = MAC(secret_key, src_ip, src_port, dst_ip, dst_port, t) mod 2^24 if (server_isn & 0xFFFFFF) != expected_cookie: drop("invalid cookie, likely spoofed ACK") return // Valid! Only now allocate the real connection-control-block. ALLOCATE_TCB(src_ip, src_port, dst_ip, dst_port) ``` The SYN-cookie trick is the crux of protocol-attack defense: it makes the server memoryless during the vulnerable half-open period. A spoofed SYN flood normally works because each SYN forces the server to allocate a Transmission Control Block (TCB) and wait for an ACK that spoofed sources never send, exhausting a finite table. By encoding all necessary state into the SYN-ACK's sequence number itself — verifiable via a MAC keyed on a server secret — the server needs zero storage until a genuine three-way handshake completes.

VPN handshake cost. A Diffie–Hellman-based handshake requires $O(1)$ modular exponentiations per party — typically 1–2 exponentiations of cost $O(\log^3 M)$ bit operations for modulus size $M$ using standard exponentiation algorithms, or a small constant number of elliptic-curve scalar multiplications for ECDH. This is a one-time cost per session (or per rekey interval). Per-packet cost thereafter is $O(n)$ in packet length $n$ for AEAD encryption/decryption (a symmetric-key stream cipher plus authentication tag, e.g., ChaCha20-Poly1305 or AES-GCM), since these primitives process data in a single linear pass. Space overhead per session is $O(1)$: a session key, a send counter, and a fixed-size replay window (commonly 32–64 bits), independent of how many packets have been sent.

Correctness of the tunnel invariant. The key safety property is: no packet is injected into the receiver's local stack unless it was produced by TUNNEL_SEND under the shared key and has not been seen before. This holds because AEAD decryption fails (with overwhelming probability, bounded by the forgery probability of the underlying MAC, typically $2^{-128}$) on any ciphertext not produced under the correct key, and the sliding replay window rejects any nonce already marked seen. The invariant is maintained by induction on packets processed: initially the window is empty (vacuously true), and each successful decrypt-and-accept step marks the nonce, preserving the "no double accept" property for subsequent packets within the window's range. Packets that arrive too far out of order (older than the window) are conservatively dropped — a safety-over-liveness tradeoff: correctness is never violated, at the cost of occasionally discarding legitimate, badly reordered traffic.

SYN-cookie correctness. The claim is that HANDLE_ACK accepts a connection if and only if the ACK is a response to a genuine SYN-ACK the server actually sent (within the freshness window), without the server having stored any state for that half-open connection. This follows from the MAC's unforgeability: an attacker who never saw the server's SYN-ACK cannot compute `expected_cookie` because it depends on a secret key unknown to them, so the probability of an off-path attacker guessing a valid 24-bit cookie is $2^{-24}$ per attempt — negligible against a flood, and further reduced if the source-IP/port fields are checked. The tradeoff is real: the server loses TCP options negotiated in the original SYN (window scaling, SACK, MSS beyond a default), because that information is not stored; well-designed cookie schemes squeeze a few of the most common option combinations into the spare cookie bits, but not all of them.

Detection statistics. A common anomaly-scoring approach computes a distance between an observed feature vector (packet rate, protocol mix, source-IP entropy) and a rolling baseline, e.g., a z-score $z = (x - \mu)/\sigma$ against an exponentially-weighted moving average $\mu$ with decay $\alpha$, updated in $O(1)$ per window. This is efficient — $O(1)$ amortized per statistic per window, $O(k)$ for $k$ tracked features — but not exact: it is a statistical hypothesis test with both false positives (legitimate flash-crowd traffic misclassified as attack) and false negatives (a slow-and-low attack blending under threshold). This is fundamentally unavoidable: distinguishing "many real users suddenly interested" from "many bots simulating interest" is not decidable from traffic shape alone in the worst case, which is why defense-in-depth (rate limiting and CAPTCHA and reputation lists) is standard rather than any single perfect classifier.

Anycast scrubbing complexity. Routing attack traffic to the nearest of $m$ geographically distributed scrubbing centers via BGP anycast divides a volumetric attack of total volume $V$ into up to $m$ shares, so any single center need only absorb roughly $V/m$ in the best case (assuming the botnet's sources are geographically spread proportionally) — this is a capacity argument, not an algorithmic complexity bound, but it is the load-balancing principle that makes multi-terabit attacks survivable: no single link needs $O(V)$ capacity, only $O(V/m)$ per site, at the cost of operating and provisioning $m$ separate scrubbing facilities.

Trace A — WireGuard-style tunnel setup and one data packet.

If an attacker replays outer packet from step 6 later, step 8 finds `nonce = 0` already marked and drops it — the replay-window check enforces the "no double delivery" invariant even though the ciphertext is byte-for-byte identical to a valid one.

Trace B — SYN flood against a web server, with and without SYN cookies. Assume the attacker sends spoofed SYNs at 500,000 packets/second from randomized source IPs, and the server's connection table holds 10,000 half-open entries before defense.

The qualitative lesson: SYN cookies convert an attack against a bounded, stateful resource (the TCB table) into an attack against an effectively unbounded, stateless computation (a MAC evaluation per packet), which the server can perform at line rate far beyond what a connection table could ever hold.

VPN cryptography descends directly from the public-key revolution of the 1970s, while its tunneling mechanics trace to earlier encapsulation ideas in internetworking research. DDoS defense, by contrast, matured reactively: SYN cookies existed as a known idea before major floods forced their widespread deployment, and anycast-based scrubbing architectures emerged largely in response to escalating attack volumes from botnets exploiting insecure consumer and IoT devices. The two fields converge in practice at the network edge, where the same routers and appliances that terminate VPN tunnels for legitimate remote users must also filter and rate-limit everything else.

Enterprise remote access (Cisco AnyConnect, corporate IPsec gateways, and modern WireGuard-based deployments) relies on exactly the tunnel mechanics described here to let employees reach internal resources securely from arbitrary networks. Cloud providers offer VPN gateway services (AWS Site-to-Site VPN, Azure VPN Gateway, Google Cloud VPN) as managed IPsec endpoints connecting on-premises data centers to cloud VPCs. Consumer VPN products build the same tunneling primitive for privacy and geo-access purposes, though their trust model shifts from "your own gateway" to "a third-party operator." On the DDoS side, content delivery networks and DDoS-scrubbing providers (Cloudflare, Akamai, AWS Shield) operate globally distributed anycast networks that absorb multi-terabit floods by design, applying SYN-cookie-like stateless defenses, rate limiting, and challenge-response systems (CAPTCHAs, JavaScript proof-of-work) at the edge before traffic ever reaches an origin server. Modern operating system kernels ship SYN-cookie support natively (Linux's `tcp_syncookies` sysctl), and most enterprise firewalls and load balancers implement connection-rate limiting as a first line of defense, illustrating how both VPNs and DDoS mitigation have moved from research curiosities into default, load-bearing infrastructure.

Ask the tutor: How would you redesign the SYN-cookie scheme to also preserve TCP window-scaling information without adding server-side state?