Computer networks are built upon the foundational abstractions of the OSI 7-Layer model and the TCP/IP stack. However, the original protocols of the Internet (such as IPv4, ARP, DNS, and baseline TCP) were designed decades ago with zero built-in authentication or encryption, under the optimistic assumption that all network nodes were mutually trusted academic institutions.
In this deep guide, we examine the vulnerabilities inherent in core networking protocols, dissect packet analysis using Berkeley Packet Filters (BPF), and evaluate modern wire defenses from Stateful Inspection Firewalls to TLS 1.3.
Anatomy of the TCP 3-Way Handshake and the SYN Flood#
Transmission Control Protocol (TCP) establishes reliable, stateful, bi-directional streams between hosts using a three-way handshake:
Client Server
| |
| -------- SYN (Seq = X) ------------------------------> | [ Allocates TCB in SYN Queue ]
| |
| <------- SYN-ACK (Seq = Y, Ack = X + 1) -------------- | [ State: SYN-RECEIVED ]
| |
| -------- ACK (Seq = X + 1, Ack = Y + 1) -------------> | [ Moves TCB to Accept Queue ]
| | [ State: ESTABLISHED ]The SYN Flood Vulnerability#
During the handshake, when the server receives a SYN packet, it allocates a Transmission Control Block (TCB) memory structure (typically 280 to 1,024 bytes) in the operating system kernel's SYN Queue (backlog) and starts a retransmission timer.
In a SYN Flood Denial of Service attack:
- The attacker broadcasts tens of thousands of
SYNpackets per second from forged, spoofed IP addresses. - The server responds with
SYN-ACKpackets to the spoofed IPs and waits for the finalACK. - The spoofed hosts either discard the unprompted
SYN-ACKor do not exist, so no finalACKever arrives. - The server's SYN Queue fills up to maximum capacity (
net.ipv4.tcp_max_syn_backlog), after which the kernel drops all subsequent incoming legitimate connection requests!
The Mitigation: SYN Cookies (RFC 4987)#
Instead of allocating a TCB in kernel memory upon receiving a SYN, the server encodes the connection state directly into the 32-bit Initial Sequence Number (ISN) of the SYN-ACK packet:
ISN_server = Hash(Client_IP, Server_IP, Client_Port, Server_Port, Timestamp, Secret_Key)The server stores zero state in RAM. When the legitimate client replies with ACK (Ack = ISN_server + 1), the server subtracts 1 from the acknowledgment number, recalculates the cryptographic hash using its secret key, and verifies the connection. Only then does it allocate a socket structure in memory, completely neutralizing SYN flood memory exhaustion attacks!
Low-Level Packet Analysis: Berkeley Packet Filters (BPF)#
Network security analysts inspect packet traffic on the wire using tcpdump and Wireshark. Under the hood, packet capture utilizes Berkeley Packet Filters (BPF), an in-kernel virtual machine architecture that discards unwanted packets before they are copied from kernel space into user space:
# Capture all TCP traffic targeting port 443 with the SYN flag set (initial connection probes)
tcpdump -nn -i eth0 'tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) == 0 and dst port 443'
# Detect potential DNS tunneling: UDP DNS requests with payload size > 200 bytes
tcpdump -nn -i eth0 'udp and port 53 and greater 200'
# Capture plaintext HTTP authentication attempts
tcpdump -nn -A -i eth0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12:2]&0xf0)>>2)) != 0)' | grep -i 'Authorization: Basic'Modern Wire Defense: TLS 1.3 vs. TLS 1.2#
Transport Layer Security (TLS) encrypts application traffic across the transport layer:
+------------------------------------+------------------------------------+
| TLS 1.2 Handshake (2 RTT) | TLS 1.3 Handshake (1 RTT) |
+------------------------------------+------------------------------------+
| ClientHello | ClientHello + KeyShare (ECDHE) |
| ------------> | ------------> |
| ServerHello + Cert + ServerKeyEx | ServerHello + Cert + Finished |
| <------------ | <------------ |
| ClientKeyEx + Finished | Client Finished + App Data |
| ------------> | ------------> |
| App Data Flow Begins | App Data Flow Begins |
+------------------------------------+------------------------------------+Key Security Enhancements in TLS 1.3 (RFC 8446)#
- Zero-RTT Resumption (0-RTT): Clients reconnecting to a previous server can send early encrypted application data in the very first flight.
- Elimination of Broken Cryptography: Completely banned insecure ciphers and primitives:
- Removed RC4, DES, 3DES, MD5, SHA-1.
- Removed static RSA key exchange (enforcing Perfect Forward Secrecy exclusively).
- Removed CBC mode ciphers (eliminating padding oracle attacks).
- Standardized strictly on Authenticated Encryption with Associated Data (AEAD) ciphers: AES-128-GCM, AES-256-GCM, and ChaCha20-Poly1305.
- Encrypted Handshake: TLS 1.3 encrypts server certificate messages on the wire, preventing passive eavesdroppers from identifying which domain or SNI the client is connecting to.
Checkpoint Questions & Practice Exercises#
Question 1: An attacker on a shared local network broadcasts unsolicited ARP replies associating the IP address of the default gateway with their own MAC address. What attack is this, and what switchport security feature prevents it?
- Answer: This is an ARP Spoofing / ARP Cache Poisoning attack (Man-in-the-Middle). The standard enterprise switchport defense is Dynamic ARP Inspection (DAI), which inspects ARP packets against a database of valid IP-to-MAC bindings built by DHCP Snooping.
Question 2: Explain why modern security architectures forbid the use of static RSA key exchange in TLS configurations.
- Answer: Static RSA key exchange does not provide Perfect Forward Secrecy (PFS). If an adversary captures and archives encrypted traffic, and at any point in the future compromises the server's private key, all historical sessions can be decrypted. Ephemeral Diffie-Hellman (ECDHE) must be enforced instead.