Binary Memory Corruption & Buffer Overflow

Binary memory corruption represents the most enduring, devastating, and technically nuanced class of vulnerabilities in systems programming. Operating beneath the type-safety guarantees and managed garbage collection of high-level languages, low-level systems implemented in C, C++, and unsafe assembly interact directly with raw virtual memory addresses, hardware registers, and CPU execution pipelines. When developers introduce subtle flaws in pointer arithmetic, array boundary validation, or memory lifetime management, software ceases to operate as an abstract state machine and becomes an instrument under the deterministic control of an adversary.

To defend mission-critical operating systems, hypervisors, cryptographic runtimes, and carrier-grade network appliances, systems engineers and security architects must possess an exhaustive, assembly-level comprehension of process memory layouts, processor calling conventions, stack-smashing mechanics, heap metadata internals, and modern compiler-assisted and hardware-enforced exploit mitigations. This guide delivers a rigorous, mathematically formal, and architecture-accurate treatise on memory corruption mechanics across the x86-64 architecture.

1. The x86-64 Virtual Memory Architecture & Process Address Space#

On modern 64-bit operating systems (such as Linux and FreeBSD), the Memory Management Unit (MMU) and the OS kernel cooperate to present every user-space process with an isolated, contiguous, 64-bit flat virtual address space.

+---------------------------------------------------------------------------------------+
|                         x86-64 Process Virtual Memory Layout                          |
|                                                                                       |
|   High Memory: 0x7FFFFFFFFFFF                                                         |
|   +-------------------------------------------------------------------------------+   |
|   | Kernel Space (Inaccessible from Ring 3 User Mode)                             |   |
|   +-------------------------------------------------------------------------------+   |
|   | Environment Variables, Command-Line Arguments (argc, argv, envp)              |   |
|   +-------------------------------------------------------------------------------+   |
|   | User Stack (Grows Downward towards Lower Addresses)                            |   |
|   |      |                                                                        |   |
|   |      v [RSP points to Stack Top]                                              |   |
|   |                                                                               |   |
|   |      ^ [Unmapped Memory / Guard Pages / Address Space Gap]                    |   |
|   |      |                                                                        |   |
|   | Memory Mapping Segment (mmap, Shared Libraries like libc.so, Thread Stacks)    |   |
|   +-------------------------------------------------------------------------------+   |
|   |                                                                               |   |
|   | Heap (Grows Upward via brk() / sbrk())                                        |   |
|   |      ^                                                                        |   |
|   +------+------------------------------------------------------------------------+   |
|   | BSS Segment (Uninitialized Static / Global Data, zero-filled by kernel)       |   |
|   +-------------------------------------------------------------------------------+   |
|   | Data Segment (Initialized Static / Global Variables)                          |   |
|   +-------------------------------------------------------------------------------+   |
|   | Text / Code Segment (Executable Instructions, read-only, mapped from binary)   |   |
|   +-------------------------------------------------------------------------------+   |
|   Low Memory:  0x000000000000 (Page Zero: Trap on Null Pointer Dereference)           |
+---------------------------------------------------------------------------------------+

Virtual Address Geometry: Canonical Addresses#

Although x86-64 uses 64-bit pointers, current physical CPU architectures implement either a 48-bit (4-level paging) or 57-bit (5-level paging) virtual address bus:

• In a 48-bit implementation, the virtual address space spans 256 Terabytes (2^{48} bytes).
• The hardware enforces Canonical Addressing: Bits 47 through 63 must be identical (a sign-extension of Bit 47):

- Lower Canonical Half (User Space): 0x0000000000000000 to 0x00007FFFFFFFFFFF (Addresses where Bit 47 is 0).

- Non-Canonical Hole: 0x0000800000000000 to 0xFFFF7FFFFFFFFFFF (Attempting to dereference any address within this gap immediately triggers a hardware General Protection Fault #GP).

- Upper Canonical Half (Kernel Space): 0xFFFF800000000000 to 0xFFFFFFFFFFFFFFFF (Addresses where Bit 47 is 1).

Detailed Process Segments#

1. Text Segment (`.text`):

- Contains the raw machine instructions compiled from source code.

- Marked with permissions r-x (Read, Execute, No-Write). Enforced by the MMU page table NX (No-Execute) and R/W bits to prevent self-modifying code and code injection.

1. Data Segment (`.data`):

- Stores initialized global and static variables (e.g., static int counter = 42;). Marked rw-.

1. BSS Segment (`.bss` - Block Started by Symbol):

- Stores uninitialized global and static variables (e.g., static char buffer[1024];).

- Occupies zero bytes in the on-disk ELF binary image; the kernel automatically maps zero-filled anonymous pages upon process invocation. Marked rw-.

1. The Heap:

- Managed dynamically at runtime via memory allocators (e.g., glibc ptmalloc, jemalloc). Expands upward toward higher memory via the brk() system call or allocates large memory chunks via mmap(). Marked rw-.

1. Memory Mapping Segment (`mmap`):

- Maps shared dynamic libraries (such as libc.so, libssl.so), inter-process shared memory regions, and secondary thread stacks. Typically grows downward in modern Linux memory layouts.

1. The User Stack:

- Manages function execution state, local variables, compiler temporaries, and return addresses.

- Grows downward from high memory toward low memory. Marked rw-.

2. Low-Level Stack Frame Architecture & System V AMD64 ABI#

To understand stack corruption, one must dissect the exact mechanics of function calls at the machine register and memory level. The System V AMD64 ABI dictates the calling convention for Linux, macOS, and BSD systems.

+---------------------------------------------------------------------------------------+
|                             Standard x86-64 Stack Frame                               |
|                                                                                       |
|   Memory Address       Stack Frame Element                                            |
|   High Memory          +---------------------------------------------------------+    |
|        ^               | Function Arguments 7, 8, ... (Pushed in reverse order)  |    |
|        |               +---------------------------------------------------------+    |
|        |               | Saved Return Address (Pushed automatically by CALL)     |    |
|        |               +---------------------------------------------------------+ <-- [Caller Frame Boundary]
|        |               | Saved Base Pointer (RBP pushed in function prologue)    |    |
|        |               +---------------------------------------------------------+ <-- Current RBP (Frame Base)
|        |               | Compiler Stack Canary (fs:0x28, inserted if enabled)    |    |
|        |               +---------------------------------------------------------+    |
|        |               | Local Variable 1                                        |    |
|        |               +---------------------------------------------------------+    |
|        |               | Local Buffer [N Bytes]                                  |    |
|        v               +---------------------------------------------------------+ <-- Current RSP (Stack Top)
|   Low Memory           | Red Zone (128 Bytes below RSP, leaf functions only)     |    |
+---------------------------------------------------------------------------------------+

Hardware Registers & Argument Passing#

Under the System V AMD64 ABI, the first six integer and pointer arguments to a function are passed directly within CPU general-purpose registers rather than pushed onto the stack:

1. RDI: First argument
2. RSI: Second argument
3. RDX: Third argument
4. RCX: Fourth argument
5. R8 : Fifth argument
6. R9 : Sixth argument

Any additional arguments (7th and above) are pushed onto the stack in reverse order prior to the call. Floating-point arguments are passed in vector registers XMM0 through XMM7.

The Lifecycle of a Function Call#

When a caller invokes a function target_function():

Step 1: The `CALL` Instruction

The CPU executes call target_function. Hardware automatically performs two atomic operations:

1. Pushes the address of the next sequential instruction (the Return Address, or saved RIP) onto the stack:

RSP arrow RSP - 8

[RSP] arrow RIP_{next}

1. Sets the Instruction Pointer RIP to the entry point address of target_function.

Step 2: The Function Prologue

Inside target_function, the compiler emits prologue instructions to construct a new stack frame:

push   rbp             ; Save the caller's base pointer
mov    rbp, rsp        ; Set the current frame pointer to the stack top
sub    rsp, 0x40       ; Allocate 64 bytes for local variables (16-byte aligned)

• The Base Pointer (`RBP`): Serves as a fixed reference point throughout the function's execution. Local variables are accessed via negative offsets (e.g., [rbp - 0x10]), while stack-passed arguments are accessed via positive offsets (e.g., [rbp + 0x10]).
• The Stack Pointer (`RSP`): Points dynamically to the current lowest active byte of the stack.
• Stack Alignment Invariant: The ABI mandates that prior to executing any CALL instruction, the stack pointer RSP must be 16-byte aligned (RSP mod 16 = 0). Failure to maintain this alignment causes hardware SSE/AVX vector instructions (e.g., movaps) to crash with a General Protection Fault.

Step 3: The Function Epilogue

When the function finishes, it cleans up its stack frame and transfers control back to the caller:

mov    rsp, rbp        ; Deallocate all local variables by restoring RSP to RBP
pop    rbp             ; Restore the caller's saved base pointer
ret                    ; Pop the saved return address from stack into RIP

Equivalently, compilers frequently emit the single hardware instruction leave, which is microcode-equivalent to mov rsp, rbp followed by pop rbp.

The ret instruction executes:

RIP arrow [RSP]

RSP arrow RSP + 8

The CPU resumes fetching instructions from the address specified by the popped value.

3. The Classic Stack Buffer Overflow: Overwriting Control Flow#

A buffer overflow occurs when an application writes more data into a fixed-size memory buffer than that buffer was dimensioned to hold.

+---------------------------------------------------------------------------------------+
|                         Stack Buffer Overflow Exploitation                            |
|                                                                                       |
|   Memory Layout Before Overflow:                                                      |
|   [ Buffer (64 Bytes) ][ Compiler Padding ][ Saved RBP (8 B) ][ Saved RIP (8 B) ]    |
|   ^                                                                                   |
|   |--- Valid string copy fits safely within 64 bytes.                                 |
|                                                                                       |
|   Memory Layout After Malicious Overflow (e.g., gets(buffer) with 88 Bytes):          |
|   ['A' x 64 Bytes     ]['A' x 8 Bytes      ]['B' x 8 Bytes   ][ TARGET EXPLOIT ADDR ] |
|   ^                                                           ^                       |
|   |--- Buffer fills    |--- Padding smashed |--- Saved RBP    |--- Saved RIP hijacked!|
|                                                  corrupted!   When function executes  |
|                                                               'ret', CPU jumps to:    |
|                                                               TARGET EXPLOIT ADDR!    |
+---------------------------------------------------------------------------------------+

The Vulnerability Pattern#

Consider the classic vulnerable C snippet:

// Vulnerable C function exhibiting unbounded stack write
void process_user_input(int socket_fd) {
    char user_buffer[64];
    // Unsafe: read() does not bound input to sizeof(user_buffer)
    read(socket_fd, user_buffer, 256);
}

In this function:

• The compiler allocates space on the stack for user_buffer starting at [rbp - 0x40].
• The user input starts writing at [rbp - 0x40] and writes consecutively toward higher memory addresses (toward rbp).
• If the socket delivers 88 bytes of payload:

- Bytes 0 through 63 fill user_buffer.

- Bytes 64 through 71 overwrite any alignment padding.

- Bytes 72 through 79 overwrite the Saved `RBP`.

- Bytes 80 through 87 overwrite the Saved `RIP` (Return Address).

• When process_user_input() executes ret, the CPU pops the attacker's 8-byte value into RIP. The CPU immediately begins executing instructions at whatever address the attacker supplied.

The Historical Shellcode Era vs. Modern Execution Constraints#

In early computing (e.g., Aleph One's 1996 paper "Smashing the Stack for Fun and Profit"), the stack was marked both Writable and Executable (rwx). Attackers injected compiled binary machine code (known as Shellcode) directly into the buffer, calculated the stack address of the buffer, and overwrote the saved RIP with that stack address. When the function returned, the CPU directly executed the attacker's shellcode on the stack, typically executing an execve("/bin/sh", NULL, NULL) system call to spawn an interactive root shell.

Today, modern hardware and operating systems implement multi-layered defenses that completely eliminate naive shellcode injection, forcing adversaries into advanced exploitation paradigms.

4. Modern Exploit Mitigations: The Defense-in-Depth Matrix#

To eradicate memory corruption exploitation, modern computing platforms introduce four foundational defenses operating across the compiler, operating system kernel, and CPU silicon:

+---------------------------------------------------------------------------------------+
|                               Exploit Mitigation Matrix                               |
|                                                                                       |
|  [ Data Execution Prevention / NX Bit ]                                               |
|      Enforces Harvard-like separation in hardware: Memory can be Writable OR          |
|      Executable, NEVER both (W ^ X). Completely defeats stack/heap shellcode.         |
|                                                                                       |
|  [ Stack Canaries (__stack_chk_fail) ]                                                |
|      Compiler places a random 64-bit secret between local buffers and saved RIP.      |
|      Epilogue verifies secret before executing 'ret'. Aborts process if corrupted.    |
|                                                                                       |
|  [ Address Space Layout Randomization (ASLR / PIE) ]                                  |
|      Kernel randomizes base addresses of Stack, Heap, Libraries, and Executable.      |
|      Renders hardcoded payload addresses completely unpredictable.                    |
|                                                                                       |
|  [ Control Flow Integrity (CFI / Intel CET) ]                                         |
|      Hardware verifies indirect call targets (IBT) and maintains an isolated,         |
|      read-only Shadow Stack (CET SS) validating return addresses on 'ret'.            |
+---------------------------------------------------------------------------------------+

1. Data Execution Prevention (DEP) / The NX (No-Execute) Bit#

• Hardware Mechanism: The AMD64 and Intel 64 MMU page table entry allocates Bit 63 as the Execute-Disable (XD / NX) bit.
• If Bit 63 is set to 1, the CPU hardware memory controller refuses to fetch instructions from that physical memory page. Any attempt to point RIP to a memory address located within that page triggers an immediate page fault exception (#PF with error code 0x11).
• Operating system kernels mark the Stack, Heap, and BSS segments with NX = 1.
• The W ^ X Invariant: Memory pages are either Writable or Executable, but strictly never both simultaneously:

Writable ∩ Executable = ∅

• Defensive Impact: Injected shellcode residing in stack or heap buffers is rendered completely inert. Attempting to execute shellcode on the stack crashes the program immediately.

2. Stack Canaries (StackGuard / `-fstack-protector`)#

Stack canaries are compiler-generated defensive guard values designed to detect linear stack overflows before the function epilogue can execute ret:

; Compiler-Generated Function Prologue with Stack Canary
process_input:
    push   rbp
    mov    rbp, rsp
    sub    rsp, 0x50
    mov    rax, QWORD PTR fs:0x28      ; Load 64-bit canary from Thread Local Storage
    mov    QWORD PTR [rbp-0x8], rax    ; Store canary immediately below Saved RBP
    xor    eax, eax                    ; Clear RAX register to prevent leakage

    ; ... [Function Body Executes] ...

; Compiler-Generated Function Epilogue with Canary Verification
    mov    rax, QWORD PTR [rbp-0x8]    ; Reload canary from stack
    sub    rax, QWORD PTR fs:0x28      ; Compare against master TLS canary
    jne    .stack_check_failed         ; If mismatch, jump to abort routine
    leave
    ret

.stack_check_failed:
    call   __stack_chk_fail            ; Prints "*** stack smashing detected ***", SIGABRT

• Canary Generation: The canary is generated randomly by the kernel during process initialization (via getrandom()) and stored in the Thread Control Block (TCB) accessed via the FS segment register at offset 0x28 (fs:0x28).
• The Terminator Canary: In Linux, the least significant byte of the 64-bit canary is intentionally set to null (0x00). Because standard C string copy functions (such as strcpy, strcat, sprintf, gets) treat 0x00 as the string terminator, any string-based buffer overflow is forced to terminate precisely when it touches the canary, preventing the attacker from writing past it without terminating the copy.

3. Address Space Layout Randomization (ASLR & PIE)#

ASLR ensures that the virtual memory addresses of all memory segments are randomized on every process execution.

• Kernel Entropy: When the Linux kernel executes an ELF binary via execve(), it adds a pseudo-random offset (entropy) to the base addresses:

- Stack Base Address

- Heap Base Address (brk)

- Memory Mapping Base Address (mmap, dynamic libraries like libc.so)

• Position-Independent Executables (PIE / PIC):

- Historically, the executable text segment was mapped to a static base address (e.g., 0x400000).

- With -fPIE (Position Independent Executable), the compiler emits code using relative instruction addressing ([rip + offset]). The kernel randomizes the Text segment base address as well.

• The Security Invariant: An attacker who corrupts a return address cannot jump to a specific function (e.g., system()) because they cannot predict where libc or the application code resides in memory.

4. Hardware-Assisted Control Flow Integrity: Intel CET#

The newest frontier in memory protection is Intel Control-Flow Enforcement Technology (CET) and ARM Branch Target Identification (BTI):

1. Shadow Stack (CET-SS):

- The CPU maintains a secondary, hardware-isolated stack in physical memory inaccessible to standard user-space write instructions.

- When a CALL instruction executes, hardware pushes the return address onto BOTH the normal stack AND the Shadow Stack.

- When RET executes, hardware pops both return addresses and compares them. If a mismatch is detected, the CPU fires a Control Protection Exception (#CP), instantly halting execution before an attacker's corrupted address can load into RIP.

1. Indirect Branch Tracking (IBT):

- Validates forward-edge control flow (indirect calls via function pointers, e.g., call rax).

- Mandates that every legal jump target must begin with a special hardware instruction: ENDBR64 (End Branch 64-bit). If an indirect jump lands on any instruction other than ENDBR64, the CPU faults immediately.

5. Heap Exploitation Foundations: `ptmalloc` Chunk Internals#

While the stack handles local function scopes, dynamic long-lived memory allocations (malloc(), calloc(), realloc(), free()) are serviced by the heap memory allocator. In Linux systems, glibc implements the `ptmalloc` allocator (derived from Doug Lea's dlmalloc).

+---------------------------------------------------------------------------------------+
|                               Allocated ptmalloc Chunk Layout                         |
|                                                                                       |
|   Offset (Bytes)   Chunk Field                                                        |
|   +0               +-------------------------------------------------------------+    |
|                    | Previous Chunk Size (prev_size) - Valid only if PREV_INUSE=0|    |
|   +8               +-------------------------------------------------------------+    |
|                    | Chunk Size (Includes header size) | A | M | P |                  |
|   +16              +-------------------------------------------------------------+    |
|                    | User Data Payload (Returned pointer points HERE!)           |    |
|                    | ...                                                         |    |
|   +Chunk Size      +-------------------------------------------------------------+    |
|                                                                                       |
|                                 Freed ptmalloc Chunk Layout                           |
|                                                                                       |
|   +0               +-------------------------------------------------------------+    |
|                    | Previous Chunk Size (prev_size)                             |    |
|   +8               +-------------------------------------------------------------+    |
|                    | Chunk Size (Includes header size) | A | M | P |                  |
|   +16              +-------------------------------------------------------------+    |
|                    | Forward Pointer (fd)  -> Points to next free chunk in bin   |    |
|   +24              +-------------------------------------------------------------+    |
|                    | Backward Pointer (bk) -> Points to prev free chunk in bin   |    |
|   +32              +-------------------------------------------------------------+    |
|                    | Unused payload space (until next chunk header)              |    |
+---------------------------------------------------------------------------------------+

Chunk Geometry & Alignment#

Every heap allocation in ptmalloc is encapsulated inside a metadata chunk aligned to 16-byte boundaries on x86-64:

1. `prev_size` (8 octets):

- If the physically preceding chunk in memory is FREE, this field stores the exact size of that preceding chunk. This allows the allocator to coalesce adjacent free chunks backwards in O(1) time.

- If the preceding chunk is ALLOCATED, prev_size is unused and acts as user payload space for the previous chunk (space optimization).

1. `size` (8 octets):

- Stores the total size of the current chunk (metadata header + user data + alignment padding).

- Because all chunks are 16-byte aligned, the lowest 3 bits of the size integer are always 000. ptmalloc steals these three unused bits to store boolean control flags:

- `A` (NON_MAIN_ARENA, Bit 2, Value 4): Set to 1 if the chunk belongs to a secondary thread arena; 0 if from the main process arena.

- `M` (IS_MMAPPED, Bit 1, Value 2): Set to 1 if the allocation was obtained directly via mmap() rather than heap brk().

- `P` (PREV_INUSE, Bit 0, Value 1): Set to 1 if the physically preceding chunk is currently allocated; set to 0 if the preceding chunk is free.

Classic Heap Vulnerability Primitives#

1. Heap Buffer Overflow: Writing past the bounds of an allocated heap buffer overwrites the header of the adjacent chunk in memory (corrupting prev_size, size, or the P bit).
2. Use-After-Free (UAF):

- Occurs when an application frees a memory pointer (free(ptr)), but fails to nullify the reference (ptr = NULL).

- The dangling pointer ptr continues to be used by the application.

- If an attacker allocates a new object that reclaims that exact heap chunk, the attacker controls the internal data fields (such as C++ virtual method table pointers, or vptr), transforming a UAF into arbitrary code execution.

1. Double Free:

- Calling free(ptr) twice on the identical pointer without intervening allocation.

- Corrupts the allocator's internal singly-linked or doubly-linked free lists (such as tcache or fastbins), causing the allocator to return identical memory addresses across subsequent distinct malloc() requests.

6. Programmatic Binary Analysis & Stack Dissector in Python#

Security engineering requires automated tooling capable of dissecting binary ELF structures, calculating stack offsets, and detecting missing compiler mitigations. Below is an industrial-grade Python analyzer utilizing the native struct library:

# Programmatic ELF Mitigation & Memory Dissector Engine
import struct
from typing import Dict, Any, List

class ELFSecurityAuditor:
    # ELF Header Identification Constants
    ELF_MAGIC = b'\x7fELF'
    EI_CLASS_64 = 2
    PT_GNU_STACK = 0x6474e551
    PF_X = 0x1

    def __init__(self, binary_path: str):
        self.path = binary_path
        with open(binary_path, 'rb') as f:
            self.data = f.read()

    def audit_mitigations(self) -> Dict[str, Any]:
        # Inspects ELF Program Headers to determine binary-level compiler mitigations
        if not self.data.startswith(self.ELF_MAGIC):
            raise ValueError("Not a valid ELF binary")

        is_64bit = self.data[4] == self.EI_CLASS_64
        if not is_64bit:
            raise NotImplementedError("Only 64-bit ELF binaries supported.")

        # Extract Program Header Table Metadata from ELF Header
        e_phoff = struct.unpack('<Q', self.data[32:40])[0]
        e_phentsize = struct.unpack('<H', self.data[54:56])[0]
        e_phnum = struct.unpack('<H', self.data[56:58])[0]

        nx_enabled = True
        for i in range(e_phnum):
            ph_offset = e_phoff + (i * e_phentsize)
            p_type, p_flags = struct.unpack('<II', self.data[ph_offset:ph_offset+8])

            # Check GNU_STACK Segment permissions
            if p_type == self.PT_GNU_STACK:
                if p_flags & self.PF_X:
                    nx_enabled = False  # Stack is marked executable!

        # Check for Stack Canary Symbol Imports
        has_canary = b'__stack_chk_fail' in self.data

        # Check for Position Independent Executable (PIE) via ELF Type
        e_type = struct.unpack('<H', self.data[16:18])[0]
        is_pie = (e_type == 3)  # ET_DYN indicates PIE shared object executable

        return {
            'NX_DEP': nx_enabled,
            'StackCanary': has_canary,
            'PIE': is_pie,
            'BinaryPath': self.path
        }

class StackFrameModel:
    # Models x86-64 stack frame layout and computes exact overwrite vectors
    def __init__(self, buffer_size: int, canary_present: bool = True):
        self.buffer_size = buffer_size
        self.canary_present = canary_present

        # Compute compiler padding to satisfy 16-byte alignment
        # Base frame allocates: buffer + (canary if present: 8 B)
        allocated_bytes = buffer_size + (8 if canary_present else 0)
        self.padding = (16 - (allocated_bytes % 16)) % 16
        self.total_local_space = allocated_bytes + self.padding

    def compute_offsets(self) -> Dict[str, int]:
        # Returns byte offsets from buffer start to critical control elements
        offset_to_canary = self.buffer_size + self.padding if self.canary_present else -1
        offset_to_rbp = self.total_local_space
        offset_to_rip = offset_to_rbp + 8

        return {
            'buffer_size': self.buffer_size,
            'padding_bytes': self.padding,
            'offset_to_canary': offset_to_canary,
            'offset_to_saved_rbp': offset_to_rbp,
            'offset_to_saved_rip': offset_to_rip
        }

7. Checkpoint Practice Problems & Applied Walkthroughs#

Consolidate your comprehension of binary memory corruption through four comprehensive multi-step mathematical problems.

Problem 1: Stack Frame Geometry & Exact Exploit Offset Calculation#

A reverse engineer analyzes a vulnerable networked binary compiled on an x86-64 Linux system using GCC. Disassembly of the target function reveals the following assembly prologue and stack initialization:

vulnerable_handler:
    push   rbp
    mov    rbp, rsp
    sub    rsp, 0x70                   ; Allocate 112 bytes of stack space
    mov    rax, QWORD PTR fs:0x28      ; Load master canary
    mov    QWORD PTR [rbp-0x8], rax    ; Store canary at [rbp - 8]
    xor    eax, eax
    lea    rax, [rbp-0x60]             ; Target buffer begins at [rbp - 96]
    mov    rdi, rax
    call   network_read_unbounded      ; Vulnerable read function

• Calculations Required:

1. Determine the allocated capacity of the local buffer located at [rbp - 0x60] in decimal bytes.

2. Calculate the exact byte offset from the start of the buffer to the Stack Canary.

3. Calculate the exact byte offset from the start of the buffer required to overwrite the Saved `RBP`.

4. Calculate the exact byte offset required to overwrite the Saved `RIP` (Return Address), and specify the minimum payload length needed to gain control of the instruction pointer.

Step-by-Step Solution:

Step 1: Determine the layout of stack variables relative to RBP.

The assembly instructions explicitly specify base-relative addressing:

• Stack frame base is RBP.
• Stack Canary is placed at: [rbp - 0x08] (Offset -8 from RBP).
• Target buffer starts at: [rbp - 0x60] (Hexadecimal 0x60 = 96 in decimal).
• Total stack allocated: 0x70 bytes = 112 bytes.

The buffer occupies the memory interval from [rbp - 0x60] up to the memory boundary preceding the canary.

• Buffer start address: RBP - 96
• Canary start address: RBP - 8
• Space between buffer start and canary:

Distance to Canary = 96 - 8 = 88 bytes.

Therefore, the compiler allocated a buffer of at most 88 bytes (or a smaller buffer, such as 80 bytes, padded by the compiler to 88 bytes for alignment).

Step 2: Calculate offset to the Stack Canary.

Because memory writes progress upward from lower addresses (RBP - 96) toward higher addresses:

• Bytes 0 through 87 fill the local buffer and any compiler padding.
• Byte 88 represents the first byte of the Stack Canary.

Offset to Canary = 88 bytes.

Step 3: Calculate offset to Saved RBP.

The Stack Canary occupies exactly 8 bytes (from RBP - 8 to RBP - 1):

• Bytes 88 through 95 overwrite the 8-byte Stack Canary.
• Byte 96 is located at [rbp + 0], which stores the Saved `RBP`.

Offset to Saved RBP = 96 bytes.

Step 4: Calculate offset to Saved RIP & Minimum Payload Length.

The Saved RBP occupies 8 bytes (from RBP + 0 to RBP + 7):

• Bytes 96 through 103 overwrite the Saved RBP.
• Byte 104 is located at [rbp + 8], which is the exact address of the Saved `RIP` (Return Address)!

Offset to Saved RIP = 104 bytes.

To completely overwrite the 8-byte return address with a target 64-bit pointer:

Minimum Payload Length = 104 + 8 = 112 bytes.

Payload Structure:

[88 Bytes Buffer Padding] + [8 Bytes Canary (if leaked)] + [8 Bytes Junk RBP] + [8 Bytes Target Exploit Address].

Problem 2: ASLR Entropy Mathematics & Brute-Force Probabilities#

An adversary attempts an automated remote memory corruption exploit against a network service running on a Linux daemon. The binary is compiled without Stack Canaries, but the host operating system enforces Address Space Layout Randomization (ASLR).

• Architecture Evaluation:

- The daemon is a multi-threaded service: If a child thread crashes due to a segmentation fault, the entire process terminates and restarts with a completely new, newly randomized address space.

- On a 32-bit Linux architecture: ASLR provides E_{32} = 16 bits of stack entropy.

- On a 64-bit Linux architecture: ASLR provides E_{64} = 28 bits of stack entropy.

• Calculations Required:

1. Calculate the total number of possible randomized stack base locations N for both 32-bit and 64-bit architectures.

2. Compute the exact probability P of successfully guessing the correct stack return address on a single attempt for both systems.

3. If the attacker can execute 1,000 exploit attempts per second against a 32-bit system, calculate the expected time (in seconds) required to achieve a 50% probability of successful exploitation.

4. Repeat the calculation for the 64-bit system and explain why remote brute-forcing of ASLR is considered mathematically impossible on 64-bit platforms.

Step-by-Step Solution:

Step 1: Calculate total possible randomized locations N.

Formula: N = 2^E

• For 32-bit system (E = 16 bits):

N_{32} = 2^{16} = 65,536 possible base addresses

• For 64-bit system (E = 28 bits):

N_{64} = 2^{28} = 268,435,456 possible base addresses

Step 2: Calculate single-attempt success probability P.

Assuming a single valid landing address:

• For 32-bit:

P_{32} = 1 / N_{32} = 1 / 65,536 ≈ 1.5259 × 10^{-5} (0.00153%)

• For 64-bit:

P_{64} = 1 / N_{64} = 1 / 268,435,456 ≈ 3.7253 × 10^{-9} (0.000000373%)

Step 3: Calculate time to 50% success probability on 32-bit system.

For independent attempts with replacement (since the daemon restarts with a fresh random seed on each crash):

The probability of failing across k attempts is:

P_{fail}(k) = (1 - P)^k

We seek k such that probability of at least one success is 50% (P_{success} = 0.50):

1 - (1 - P)^k = 0.50 ⟹ (1 - P)^k = 0.50

Taking natural logarithms:

k · ln(1 - P) = ln(0.50)

Using the Taylor series approximation ln(1 - P) ≈ -P for small P:

k · (-P) ≈ -0.693147 ⟹ k ≈ (0.693147) / (P) = 0.693147 × N

For 32-bit:

k_{32} ≈ 0.693147 × 65,536 ≈ 45,426 attempts

At a rate of R = 1,000 attempts/sec:

T_{32} = 45,426 / 1,000 ≈ 45.43 seconds

Conclusion: On 32-bit systems, 16 bits of entropy is trivially defeatable in less than one minute of brute-forcing!

Step 4: Calculate time to 50% success probability on 64-bit system:

For 64-bit:

k_{64} ≈ 0.693147 × 268,435,456 ≈ 186,065,190 attempts

At R = 1,000 attempts/sec:

T_{64} = 186,065,190 / 1,000 ≈ 186,065.19 seconds

Converting to hours and days:

T_{64} = 186,065.19 / 3,600 ≈ 51.68 hours ≈ 2.15 days

Why 64-bit ASLR Is Structurally Secure:

In production environments, a server crashing 186 million times triggers automated Intrusion Detection System (IDS) alerts, firewall rate-limits, and service manager throttling (e.g., systemd restarts are capped at 5 crashes per 10 seconds before putting the unit into a failed state).

Furthermore, modern Linux kernels support up to 33 bits of mmap entropy on 64-bit architectures (2^{33} = 8.58 billion states), requiring ~5.95 billion attempts (~68 days at 1,000 attempts/sec), rendering remote brute-force exploitation mathematically non-viable.

Problem 3: Stack Canary Leakage via Format String Vulnerability#

An auditor evaluates an authentication service written in C. The service contains an information disclosure vulnerability via a printf format string:

void authenticate_user(int sock) {
    char username[64];
    read(sock, username, 63);
    username[63] = '\0';
    // Vulnerable: Directly passing user input as format string to printf
    printf(username);
}

• Architecture Specifications:

- Target system is x86-64 Linux.

- The System V AMD64 ABI passes the first 6 integer/pointer arguments in registers: RDI, RSI, RDX, RCX, R8, R9.

- Additional arguments are fetched from the stack starting at [RSP].

- Disassembly reveals that when printf is called:

- RDI holds the pointer to username.

- The local Stack Canary is stored at memory location [RSP + 0x48] relative to the stack pointer at the moment printf begins evaluating arguments.

• Calculations Required:

1. Explain how format specifiers like %p traverse registers and stack frames.

2. Determine how many 8-byte slots correspond to registers before printf begins pulling arguments from the stack.

3. Calculate the exact positional format specifier string %Np (where N is an integer) that the auditor must send to extract the secret 64-bit Stack Canary directly to standard output.

Step-by-Step Solution:

Step 1: Analyze how `printf` argument resolution operates.

The printf(format, ...) function is a variadic C function (va_start, va_arg).

Under the System V AMD64 ABI:

• The first argument (format string pointer) is placed in register RDI.
• When printf parses a format specifier (such as %p or %lx), it assumes subsequent parameters were passed in order:

- Specifier 1: Reads RSI (Register Argument 2)

- Specifier 2: Reads RDX (Register Argument 3)

- Specifier 3: Reads RCX (Register Argument 4)

- Specifier 4: Reads R8 (Register Argument 5)

- Specifier 5: Reads R9 (Register Argument 6)

• Once the 5 register arguments are exhausted, any further format specifiers read consecutive 8-byte words directly from the stack!

- Specifier 6: Reads [RSP + 0x00] (First stack argument)

- Specifier 7: Reads [RSP + 0x08] (Second stack argument)

- Specifier k: Reads [RSP + (k - 6) * 8]

Step 2: Calculate stack offset to Canary.

The Canary is located at [RSP + 0x48].

Converting 0x48 to decimal:

4 × 16 + 8 = 64 + 8 = 72 bytes.

Each stack argument slot occupies 8 bytes:

Stack Slot Index = 72 / 8 = 9 slots above the base stack pointer.

Step 3: Calculate the positional parameter N.

We compute the total parameter index N:

• Register arguments: 5 parameters (positions 1 through 5).
• Stack slot offset: The canary is located at stack slot index 9 (which means 9 slots above [RSP]).

Let us verify the exact mapping:

• Slot 0 ([RSP + 0x00]): Position 6
• Slot 1 ([RSP + 0x08]): Position 7
• Slot 2 ([RSP + 0x10]): Position 8
• Slot 3 ([RSP + 0x18]): Position 9
• Slot 4 ([RSP + 0x20]): Position 10
• Slot 5 ([RSP + 0x28]): Position 11
• Slot 6 ([RSP + 0x30]): Position 12
• Slot 7 ([RSP + 0x38]): Position 13
• Slot 8 ([RSP + 0x40]): Position 14
• Slot 9 ([RSP + 0x48]): Position 15!

Formula:

N = 5 + (Offset) / (8) + 1 = 5 + (72) / (8) + 1 = 5 + 9 + 1 = 15

Exploitation String:

By sending the format string:

%15p

printf directly accesses the 15th logical argument, printing the hexadecimal value of [RSP + 0x48] (e.g., 0x7a3f819c4b220000). Notice the concluding 00 byte characteristic of stack canaries. The auditor now possesses the secret canary and can overwrite the stack while preserving the canary value, completely bypassing -fstack-protector.

Problem 4: `ptmalloc` Chunk Metadata Corruption & Alignment Dynamics#

An application contains a heap buffer overflow vulnerability in a dynamic logging module.

• Allocation Sequence:

- The application executes: char *chunk_A = (char *)malloc(40);

- Immediately following, it executes: char *chunk_B = (char *)malloc(60);

• Memory Environment: 64-bit Linux running glibc 2.31 (ptmalloc). The minimum chunk size is 32 bytes, and all chunks are aligned to 16 bytes.
• Calculations Required:

1. Calculate the exact total chunk size in bytes allocated for chunk_A (including metadata and alignment padding).

2. Determine the exact value stored in the size field of chunk_A's header, accounting for the A, M, and P flag bits (assuming allocation from the main arena, heap brk, and preceding chunk is in use).

3. If an attacker overflows chunk_A and overwrites chunk_B's header with prev_size = 0x80 and clears the PREV_INUSE (P) bit of chunk_B's size field, describe the exact mechanical corruption that occurs when free(chunk_B) is subsequently called.

Step-by-Step Solution:

Step 1: Calculate total chunk size for `chunk_A`.

In ptmalloc on x86-64:

• Requested payload: 40 bytes.
• Chunk header overhead: 16 bytes (prev_size + size).
• Note on space reuse: If the previous chunk is allocated, prev_size belongs to the previous chunk's space. However, the current chunk must provide at least 8 bytes for its own size field, plus user data, plus alignment.

Formula for chunk size:

Raw Size = Requested + 8 (for size field)

Raw Size = 40 + 8 = 48 bytes.

We must align to the nearest higher multiple of 16:

48 ±od{16} = 0.

Since 48 is already a multiple of 16, and 48 ≥ 32 (minimum chunk size):

Total Chunk Size for chunk_A = 48 bytes.

Step 2: Determine the value in `chunk_A`'s `size` field.

The chunk size is 48 = 0x30 in hexadecimal.

The control bits:

• A (NON_MAIN_ARENA, bit 2) = 0 (main arena).
• M (IS_MMAPPED, bit 1) = 0 (allocated via brk, not mmap).
• P (PREV_INUSE, bit 0) = 1 (preceding chunk is allocated).

Value of flags: 001_2 = 1.

Value written to size field:

Size Field Value = 48 + 1 = 49 = 0x31.

Step 3: Analyze corruption when clearing PREV_INUSE on `chunk_B`.

• When free(chunk_B) is executed, ptmalloc examines Bit 0 (PREV_INUSE) of chunk_B's size field.
• Because the attacker cleared PREV_INUSE (P = 0), the allocator falsely assumes that the chunk physically preceding chunk_B in memory is FREE.
• To prevent memory fragmentation, ptmalloc attempts backward consolidation (coalescing):

1. The allocator reads chunk_B->prev_size (which the attacker set to 0x80 = 128 bytes).

2. The allocator calculates the address of the "free" previous chunk:

Fake Chunk Address = Address(chunk_B) - prev_size = chunk_B - 128

3. The allocator attempts to unlink the fake chunk from its free bin:

```c

// Unlink macro in ptmalloc

FD = fake_chunk->fd;

BK = fake_chunk->bk;

FD->bk = BK;

BK->fd = FD;

```

• Exploitation Consequence:

In older glibc versions, this primitive executed an arbitrary 8-byte write (write-what-where): by crafting fake fd and bk pointers inside the buffer, the attacker could overwrite arbitrary function pointers (such as __free_hook or Global Offset Table entries).

In modern glibc (2.27+), hardened sanity checks verify FD->bk == P && BK->fd == P (safe unlinking). If this check fails, the allocator triggers corrupted double-linked list and immediately aborts the process via abort().