Streams, Buffers & File I/O Pipelines

At the core of Node.js's I/O performance lies its data streaming architecture. In traditional web servers, processing a file or network payload involves buffering the entire dataset into memory before passing it to the application layer. When dealing with gigabyte-scale datasets, video media streaming, or concurrent multi-tenant uploads, whole-payload buffering triggers Out-Of-Memory (OOM) crashes, saturates the garbage collector, and introduces massive processing latency.

Node.js solves this through Buffers (raw C++ allocated binary memory blocks outside the V8 heap) and Streams (continuous, chunked data pipelines with automated backpressure handling).

This guide provides an exhaustive, university-grade architectural analysis of Node.js binary memory management and stream pipelines. We dissect the internal mechanics of the Node.js 8 KB slab allocator, formulate mathematical backpressure flow control models, build resilient custom transform pipelines, diagnose memory retention leaks in buffer slicing, and walk through four comprehensive quantitative checkpoint problems.

1. Buffers & Binary Memory Management in Node.js#

JavaScript was originally designed for web browsers to manipulate text and DOM structures, lacking native primitives for handling raw binary octet streams. Node.js introduced the Buffer class to interface directly with low-level TCP sockets, TLS encryption, and filesystem blocks.

+-------------------------------------------------------------------------------------------------+
|                                 CPYTHON / NODE.JS MEMORY TOPOLOGY                               |
|                                                                                                 |
|   V8 Managed JavaScript Heap (< 2-4 GB Limit)                                                   |
|   +---------------------------------------------+                                               |
|   | FastJS Objects, Arrays, Strings, Closures   |                                               |
|   | Scavenged & Swept by V8 Garbage Collector   |                                               |
|   +---------------------------------------------+                                               |
|                          |                                                                      |
|                          | Buffer JS instance points to external memory                         |
|                          v                                                                      |
|   Node.js C++ Off-Heap Native Memory (No V8 Size Limit)                                         |
|   +---------------------------------------------+                                               |
|   | 8 KB Buffer Slabs (Allocated via malloc)    |                                               |
|   | Raw TCP Socket Payloads                     |                                               |
|   | File System I/O Buffers                     |                                               |
|   +---------------------------------------------+                                               |
+-------------------------------------------------------------------------------------------------+

The 8 KB Slab Allocator Architecture#

Invoking the operating system's malloc() for every small buffer (e.g., a 32-byte header or 64-byte token) incurs high kernel context-switching latency and system heap fragmentation.

To eliminate this, Node.js uses an internal Slab Allocator:

• Node.js pre-allocates an internal slab of 8,192 bytes (8 KB) via Buffer.poolSize.
• When an application calls Buffer.from() or Buffer.allocUnsafe() for a size smaller than half the pool size (< 4,096 bytes), Node.js carves the memory directly out of the active 8 KB slab using a fast pointer-bumping algorithm:

Offset_{new} = Offset_{current} + Size

• Only when the current 8 KB slab is exhausted does Node.js request a fresh slab from the C++ heap.

`Buffer.alloc` vs. `Buffer.allocUnsafe`#

• `Buffer.alloc(size)`:

Allocates a new Buffer and explicitly zeroes out every byte in memory.

Pros: Completely secure. Guarantees no sensitive residual data leaks.

Cons: Slower due to memory zeroing operations.

• `Buffer.allocUnsafe(size)`:

Allocates a segment of memory from the slab without zeroing out its contents.

Pros: Extremely fast.

Cons: Major Security Vulnerability if Misused. The allocated buffer contains whatever residual memory previously occupied that physical address (such as decrypted SSL secrets, database passwords, or private user messages). Never transmit an uninitialized buffer across the network!

Slicing and the Memory Retention Trap#

In standard JavaScript arrays, calling .slice() creates a copy. In Node.js Buffer:

• buf.subarray(start, end) (and legacy buf.slice()) does not copy memory.
• It creates a new Buffer object pointing to the exact same underlying memory buffer via pointer offset and length fields.

+-------------------------------------------------------------------------------------------------+
|                                 BUFFER SLICE MEMORY RETENTION LEAK                              |
|                                                                                                 |
|   [Active 8 KB Native Slab: 8,192 Bytes]                                                        |
|   +-----------------------------------------------------------------------------------------+   |
|   | 16-byte Token | 8,176 Bytes of Unused Memory / Dead Data                                |   |
|   +-----------------------------------------------------------------------------------------+   |
|          ^                                                                                      |
|          | slice.buffer holds strong reference to ENTIRE 8 KB slab!                             |
|   [Small Token Buffer: 16 bytes]                                                                |
|                                                                                                 |
|   Even though only 16 bytes are used, the entire 8,192 bytes CANNOT be garbage-collected!        |
|   100,000 tokens hold 800 Megabytes of dead memory!                                             |
+-------------------------------------------------------------------------------------------------+

Remediation: Explicit Memory Copying:

When retaining a small slice from a large buffer, allocate a dedicated buffer and copy the bytes:

// Secure: Allocates exactly 16 bytes; decouples from 8 KB slab
const isolatedToken = Buffer.alloc(16);
sourceBuffer.copy(isolatedToken, 0, start, end);

2. The Four Fundamental Stream Topologies#

A Stream is an abstract interface for working with streaming data in Node.js. All streams are instances of EventEmitter.

+-------------------------------------------------------------------------------------------------+
|                                 THE FOUR STREAM TOPOLOGIES                                      |
+-------------------+-----------------------+-----------------------+-----------------------------+
| Stream Type       | Directionality        | Primary Methods       | Canonical Use Case          |
+-------------------+-----------------------+-----------------------+-----------------------------+
| Readable          | Source (Read Only)    | read(), pause(),      | fs.createReadStream(),      |
|                   |                       | resume(), pipe()      | incoming HTTP req           |
+-------------------+-----------------------+-----------------------+-----------------------------+
| Writable          | Sink (Write Only)     | write(), end(),       | fs.createWriteStream(),     |
|                   |                       | cork(), uncork()      | outgoing HTTP res           |
+-------------------+-----------------------+-----------------------+-----------------------------+
| Duplex            | Bidirectional (Both)  | write(), read()       | net.Socket, TCP sockets,    |
|                   | Independent Read/Write|                       | WebSockets                  |
+-------------------+-----------------------+-----------------------+-----------------------------+
| Transform         | Bidirectional (Both)  | _transform(),         | zlib.createGzip(),          |
|                   | Output computed from  | _flush()              | crypto.createCipheriv()     |
|                   | input chunk           |                       |                             |
+-------------------+-----------------------+-----------------------+-----------------------------+

3. Backpressure & Flow Control Mechanics#

The most critical architectural problem in streaming I/O is Backpressure:

What happens when data is read from a fast source (e.g., local NVMe SSD reading at 500 MB/s) and written to a slow destination (e.g., mobile client downloading over 3G at 500 KB/s)?

+-------------------------------------------------------------------------------------------------+
|                                 BACKPRESSURE FLOW CONTROL LOOP                                  |
|                                                                                                 |
|   [Fast Readable Source]  -------------------- Writes Chunk -------------------> [Slow Writable] |
|   (500 MB/s SSD)                                                                (500 KB/s Net)  |
|                                                                                        |        |
|                                                                           Internal Buffer Fill: |
|                                                                           bufferedBytes >= HWM? |
|                                                                                 /          \    |
|                                                                               YES          NO   |
|                                                                               /              \  |
|   [Pause Readable!] <--- writable.write(chunk) returns FALSE <---------------+     [Return True]|
|   readable.pause()                                                                              |
|                                                                                                 |
|   [Slow Writable flushes data to physical socket...]                                            |
|   bufferedBytes drops below highWaterMark threshold                                             |
|                                 |                                                               |
|                                 v                                                               |
|   [Writable emits 'drain' event] =============================================> [Resume Source] |
|                                                                                  readable.resume()
+-------------------------------------------------------------------------------------------------+

The `highWaterMark` Threshold#

Every stream maintains an internal queue buffer governed by the highWaterMark property:

• Default for binary byte streams: 16,384 bytes (16 KB).
• Default for Object Mode streams: 16 objects.

When an application calls writable.write(chunk):

1. Node.js writes the chunk directly to the underlying resource if the buffer is empty and the OS socket buffer is ready.
2. If the resource is busy, Node.js queues the chunk in the stream's internal memory buffer.
3. If the total queued bytes exceed highWaterMark, writable.write(chunk) returns `false`.
4. Returning false is the formal signal instructing the readable stream to pause reading immediately.
5. Once the slow consumer drains the buffer, it emits the `'drain'` event, signaling the producer to resume reading.

Without backpressure handling, Node.js buffers incoming chunks into V8 RAM without limit, consuming gigabytes of memory until the process terminates with an Out-Of-Memory crash.

4. Modern Stream Composition with `stream.pipeline`#

Historically, developers piped streams using the legacy method:

readable.pipe(transform).pipe(writable)

Why Legacy `readable.pipe()` is Anti-Pattern in Production#

1. Broken Error Propagation: If transform throws an error, pipe() does not automatically destroy readable or writable. File descriptors and sockets remain open permanently, leaking system resources.
2. Missing Cleanup on Abort: If an HTTP client aborts an upload halfway through, pipe() does not close the underlying writable file stream.

Modern Solution: `node:stream/promises`#

In Node.js 16+, pipelines are composed using pipeline from node:stream/promises:

import { pipeline } from "node:stream/promises";
import fs from "node:fs";
import zlib from "node:zlib";
import crypto from "node:crypto";

async function compressAndEncryptFile(sourcePath, destPath, secretKey, iv) {
  const cipher = crypto.createCipheriv("aes-256-gcm", secretKey, iv);
  
  await pipeline(
    fs.createReadStream(sourcePath),  // Fast disk source
    zlib.createGzip(),                 // Transform 1: Compression
    cipher,                            // Transform 2: Encryption
    fs.createWriteStream(destPath)    // Destination sink
  );

  console.log("Pipeline executed with zero leaks and automatic backpressure!");
}

If any stream in the pipeline encounters an error or aborts, pipeline automatically calls .destroy() on every single stream in the chain, flushes memory buffers, closes all file descriptors, and rejects the Promise with the original error.

5. Anti-Patterns & Common Pitfalls#

The "Buffered File I/O for Large Payloads" Anti-Pattern#

Calling fs.readFile() on multi-gigabyte files loads the entire file into a single contiguous memory block in V8:

// CATASTROPHIC ANTI-PATTERN
app.get("/download-dump", async (req, res) => {
  const data = await fs.promises.readFile("/var/logs/system-dump.tar"); // 3 GB file!
  res.send(data); // V8 heap exceeds memory cap (1.4 GB / 2 GB)! PROCESS CRASHES!
});

Remediation: Always stream file downloads directly to the HTTP response:

app.get("/download-dump", (req, res) => {
  const stream = fs.createReadStream("/var/logs/system-dump.tar");
  stream.pipe(res);
});

Forgetting to Handle Backpressure in Custom Loops#

When generating data programmatically in a loop and pushing to a writable stream:

// ANTI-PATTERN: Ignoring write() return value
for (let i = 0; i < 10_000_000; i++) {
  writable.write(Buffer.from("log entry\n")); // Silently buffers millions of lines in RAM!
}

Remediation: Await the 'drain' event whenever write() returns false:

async function writeLotsOfData(writable) {
  for (let i = 0; i < 10_000_000; i++) {
    const canContinue = writable.write(Buffer.from("log entry\n"));
    if (!canContinue) {
      await events.once(writable, "drain"); // Yields until buffer clears!
    }
  }
}

6. Checkpoint Practice Problems#

The following four multi-part quantitative problems evaluate buffer slab memory retention mathematics, backpressure queue saturation dynamics, custom Transform stream framing, and whole-payload vs. streaming memory cost modeling. Complete step-by-step solutions follow each problem prompt.

Problem 1: Buffer Slab Allocator Memory Utilization & Slice Retention Analysis#

Context:

A high-throughput API gateway running Node.js 20 processes N = 20,000 authentication requests per minute. Each request parses an Authorization header containing an 8 KB base64 string, extracts a small 32-byte authentication token, and caches that token in an in-memory session cache:

const sessionCache = new Map();

function authenticateRequest(req) {
  // Reads incoming header into an 8 KB buffer
  const rawHeader = Buffer.from(req.headers['authorization']); // 8,192 bytes
  // Extracts the 32-byte token using slice / subarray
  const token = rawHeader.subarray(0, 32);
  sessionCache.set(req.id, token);
}

Memory Parameters:

• Active cached sessions maintained concurrently: N = 20,000 tokens.
• Node.js buffer slab size: S_{slab} = 8,192 bytes (8 KB).
• Target token length: S_{token} = 32 bytes.
• Each Buffer JavaScript wrapper instance consumes 48 bytes on the V8 heap.

Questions:

1. Because rawHeader.subarray(0, 32) shares underlying memory with rawHeader, how much total native memory is retained on the C++ heap to store the 20,000 cached tokens?
2. Calculate the Memory Wastage Percentage:

Wastage = [(Retained Memory - Useful Token Memory) / Retained Memory] × 100\%.

1. Refactor the token extraction using Buffer.copy() or Buffer.from() to allocate an isolated memory block.
2. Calculate the new total native memory consumed by the 20,000 tokens after refactoring, and compute the absolute Megabytes of RAM reclaimed.

Solution:

Step 1: Total Native Memory Retained Under Naive Slicing

• Each token was created via rawHeader.subarray(0, 32).
• The JavaScript Buffer instance holds a reference to the ArrayBuffer of rawHeader.
• Because each request allocated a distinct 8,192-byte buffer, the underlying ArrayBuffer is 8,192 bytes.
• V8's garbage collector cannot deallocate an ArrayBuffer until all references pointing to any subsegment within it are released.
• Therefore, each 32-byte token prevents its entire 8,192-byte parent buffer from being garbage collected!

Total Native Memory Retained:

Retained_naive = 20,000 × 8,192 bytes = 163,840,000 bytes (156.25 MB).

Step 2: Memory Wastage Percentage

• Useful token memory:

Useful_bytes = 20,000 × 32 bytes = 640,000 bytes (0.61 MB).

• Memory Wastage:

Wastage = [(163,840,000 - 640,000) / 163,840,000] × 100% ≈ 99.61% WASTED MEMORY.

Insight: 155.6 MB out of 156.25 MB is dead memory held hostage by 32-byte slices!

Step 3: Refactored Isolated Token Allocation

function authenticateRequest(req) {
  const rawHeader = Buffer.from(req.headers['authorization']);
  // Allocate dedicated, decoupled 32-byte buffer
  const token = Buffer.allocUnsafe(32);
  rawHeader.copy(token, 0, 0, 32);
  sessionCache.set(req.id, token);
  // rawHeader can now be completely garbage-collected!
}

Step 4: Memory Consumption After Refactoring

• Because 32 bytes < 4,096 bytes, Buffer.allocUnsafe(32) allocates from Node.js's internal shared 8 KB slab.
• Slices packed together: 8,192 / 32 = 256 tokens per slab.
• Number of slabs required: 20,000 / 256 = 78.125 → 79 slabs.

Total Native Memory Retained:

Retained_optimized = 79 × 8,192 bytes = 647,168 bytes (0.617 MB).

Absolute Memory Reclaimed:

Reclaimed = 156.25 MB - 0.617 MB = 155.63 Megabytes of RAM reclaimed (99.6% reduction!).

Problem 2: Backpressure Flow Control & Buffer Saturation Dynamics#

Context:

A data pipeline transfers uncompressed database records from an in-memory readable generator into a network TCP socket writable stream:

• Producer Generation Rate: R_{prod} = 40.0 MB/sec = 41,943,040 bytes/sec.
• Consumer Network Drain Rate: R_{cons} = 4.0 MB/sec = 4,194,304 bytes/sec.
• The writable stream is configured with a high-water mark of HWM = 64 KB = 65,536 bytes.
• Each chunk written by the producer has a size of S_{chunk} = 8 KB = 8,192 bytes.

At timestamp t = 0 ms, the writable buffer is completely empty. The producer starts pumping chunks continuously.

Questions:

1. Calculate the net rate of buffer accumulation (Δ R = R_{prod} - R_{cons}) in MB/sec and bytes/millisecond.
2. Calculate the exact elapsed time t_{pause} (in milliseconds) and the number of chunks written before writable.write() returns false, triggering backpressure.
3. Once backpressure is triggered and the producer pauses writing, calculate how long (t_{drain}) it takes for the slow consumer to completely drain the 65,536-byte buffer and emit the 'drain' event.
4. Over a sustained 10-second transfer, what is the effective average throughput of the pipeline, and what is the duty cycle (percentage of time reading vs. paused)?

Solution:

Step 1: Net Accumulation Rate

• Production Rate R_{prod} = 40.0 MB/s = 41,943,040 bytes/s.
• Consumption Rate R_{cons} = 4.0 MB/s = 4,194,304 bytes/s.

Net accumulation rate:

Δ R = 40.0 - 4.0 = 36.0 MB/sec.

In bytes per millisecond:

Δ R = 36,000,000 / 1,000 = 37,748.74 bytes/ms (36 KB/ms).

Step 2: Time to Trigger Backpressure (t_{pause})

Buffer capacity limit HWM = 65,536 bytes.

Number of 8,192-byte chunks required to reach or exceed HWM:

Chunks = 65,536 / 8,192 = 8 chunks.

Time required to write 8 chunks at R_{prod} = 40 MB/s:

Total bytes generated = 65,536 bytes.

t_{pause} = 65,536 / 41,943,040 s ≈ 0.0015625 s = 1.56 milliseconds.

On the 8th chunk, buffer reaches 65,536 bytes. writable.write() returns false. Producer pauses immediately.

Step 3: Drain Duration (t_{drain})

The consumer drains at R_{cons} = 4,194,304 bytes/sec.

Time to drain 65,536 bytes:

t_{drain} = 65,536 / 4,194,304 s = 0.015625 s = 15.625 milliseconds.

At t = 1.56 + 15.63 = 17.19 ms, the writable stream emits 'drain'.

Step 4: Pipeline Throughput and Duty Cycle

• One complete cycle period T_{cycle} = t_{pause} + t_{drain} = 1.5625 + 15.625 = 17.1875 ms.
• During each 17.19 ms cycle, 65,536 bytes are transferred.
• Effective Throughput = 65,536 bytes / 0.0171875 s = 4,194,304 bytes/sec (4.0 MB/sec).

Insight: The pipeline throughput automatically throttles down to match the slow consumer's capacity with zero memory accumulation!

Duty Cycle:

• Active Producer Reading Time: t_{pause} = 1.5625 ms (9.09% of cycle).
• Paused Waiting for Drain Time: t_{drain} = 15.625 ms (90.91% of cycle).

Duty Cycle = 1.5625 / 17.1875 × 100\% = 9.09\%.

Problem 3: Custom Transform Stream Implementation & Stream Slicing#

Context:

A telemetry ingestion engine receives a continuous byte stream containing newline-delimited JSON (NDJSON) packets over a TCP socket.

Because TCP is a stream-oriented protocol without message boundaries, network packets arrive fragmented into arbitrary chunks (e.g., a single JSON string might be split across Chunk A and Chunk B, or Chunk C might contain three concatenated JSON objects).

TCP Stream: '{"id":1}\n{"id":2' ---> [Chunk 1: 17 bytes]
            '}\n{"id":3}\n'     ---> [Chunk 2: 13 bytes]

Questions:

1. Implement a custom Node.js Transform stream NdjsonParserTransform that handles arbitrary chunk boundaries, buffers partial line fragments, and emits parsed JavaScript objects in Object Mode.
2. Trace what happens to the internal buffer when Chunk 1 ('{"id":1}\n{"id":2') is pushed into the stream.
3. Trace what happens when Chunk 2 ('}\n{"id":3}\n') arrives.
4. Implement the _flush(callback) method to handle edge cases where the stream closes while holding an un-terminated trailing line.

Solution:

Step 1: Production Implementation of `NdjsonParserTransform`

import { Transform } from "node:stream";

export class NdjsonParserTransform extends Transform {
  constructor(options = {}) {
    super({
      ...options,
      readableObjectMode: true, // Emits parsed JS objects
      writableObjectMode: false // Consumes raw binary/string buffers
    });
    this.buffer = "";
  }

  _transform(chunk, encoding, callback) {
    // Append newly arrived bytes to existing buffer
    this.buffer += chunk.toString("utf8");
    
    // Split on newline delimiter
    const lines = this.buffer.split("\n");
    
    // The last element is either an empty string (if ended with \n)
    // or an incomplete partial JSON fragment. Keep it in buffer!
    this.buffer = lines.pop();

    for (const line of lines) {
      const trimmed = line.trim();
      if (trimmed) {
        try {
          const parsed = JSON.parse(trimmed);
          this.push(parsed); // Push parsed object downstream
        } catch (err) {
          return callback(new Error(`Failed to parse NDJSON line: {err.message}`));
        }
      }
    }

    callback(); // Signal ready for next chunk
  }

  _flush(callback) {
    // Process any remaining bytes when input stream closes
    if (this.buffer && this.buffer.trim()) {
      try {
        const parsed = JSON.parse(this.buffer.trim());
        this.push(parsed);
      } catch (err) {
        return callback(new Error(`Corrupted trailing NDJSON record: {err.message}`));
      }
    }
    callback();
  }
}

Step 2: Trace Chunk 1 (`'{"id":1}\n{"id":2'`)

1. this.buffer becomes '{"id":1}\n{"id":2'.
2. this.buffer.split('\n') produces ['{"id":1}', '{"id":2'].
3. lines.pop() removes '{"id":2', leaving this.buffer = '{"id":2'.
4. Loop processes remaining lines: JSON.parse('{"id":1}') succeeds.
5. Emits downstream: { id: 1 }.
6. State at end of Chunk 1: Output queue has { id: 1 }, internal buffer holds '{"id":2'.

Step 3: Trace Chunk 2 (`'}\n{"id":3}\n'`)

1. this.buffer prepends old fragment: '{"id":2' + '}\n{"id":3}\n' = '{"id":2}\n{"id":3}\n'.
2. Split on \n produces: ['{"id":2}', '{"id":3}', ''].
3. lines.pop() removes '', leaving this.buffer = ''.
4. Loop parses line 1: JSON.parse('{"id":2}') -> Emits { id: 2 }.
5. Loop parses line 2: JSON.parse('{"id":3}') -> Emits { id: 3 }.
6. All records reconstructed and parsed without error.

Step 4: Role of `_flush`

If the client disconnects or closes the file stream without emitting a trailing \n, _flush executes before stream termination, guaranteeing that the final record is parsed rather than silently dropped.

Problem 4: Memory Footprint & Throughput Comparison: Buffered File I/O vs. Streaming Pipeline#

Context:

A video processing platform ingests large media files (S_{file} = 4.0 GB = 4,096 MB) uploaded by users. The server must compute an MD5 verification checksum and write the file to persistent disk.

The team tests two architectures on a server allocated 2.0 GB of RAM (--max-old-space-size=2048):

• Architecture A (Whole-Payload Buffering):

```javascript

const data = await fs.promises.readFile("/tmp/upload.raw"); // 4 GB buffer

const hash = crypto.createHash("md5").update(data).digest("hex");

await fs.promises.writeFile("/storage/video.mp4", data);

```

• Architecture B (Streaming Pipeline):

```javascript

const hashStream = crypto.createHash("md5");

await pipeline(

fs.createReadStream("/tmp/upload.raw", { highWaterMark: 64 * 1024 }),

new PassThrough().on("data", chunk => hashStream.update(chunk)),

fs.createWriteStream("/storage/video.mp4", { highWaterMark: 64 * 1024 })

);

const hash = hashStream.digest("hex");

```

Questions:

1. Analyze what occurs when Architecture A executes on the 2.0 GB RAM server. Identify the exact runtime failure point.
2. For Architecture B, calculate the peak resident memory (RAM) consumed during the entire 4 GB file transfer.
3. Calculate the Time to First Byte (TTFB) written to destination disk for Architecture A vs. Architecture B, assuming disk read latency of 100 MB/sec.
4. Conclude the architectural trade-offs: why is streaming mandatory for enterprise systems?

Solution:

Step 1: Architecture A Failure Analysis

• V8 Max Heap Cap: 2,048 MB.
• Attempting fs.promises.readFile() allocates a 4,096 MB Buffer.
• CPython / Node.js enforces a maximum buffer allocation limit (buffer.constants.MAX_LENGTH ≈ 4 GB on 64-bit systems).
• Even if native allocation succeeds, loading 4 GB requires allocating multiple internal JavaScript references and strings.
• Peak memory requirement: 4,096 MB (source buffer) + 4,096 MB (destination write copy) = 8,192 MB (8.0 GB)!
• Result: The process immediately crashes with `FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory`. The file cannot be processed.

Step 2: Peak Resident Memory in Architecture B (Streaming)

In Architecture B, data flows through in small 64 KB chunks:

• Read stream buffer: 64 KB.
• PassThrough stream buffer: 64 KB.
• Write stream buffer: 64 KB.
• Crypto hash internal state: 256 bytes.
• Node.js runtime baseline: ≈ 35 MB.

Total Peak Memory = 35 MB + (3 × 64 KB) = 35.19 Megabytes!

Result: A 4 Gigabyte file is processed comfortably inside 35 Megabytes of RAM, completely unaffected by file size.

Step 3: Time to First Byte (TTFB) Written to Destination Disk

Disk read throughput = 100 MB/sec.

• Architecture A: Must read entire 4,096 MB before writing byte 1:

TTFB_A = 4,096 MB / 100 MB/s = 40.96 seconds.

The disk sits idle for 41 seconds before write begins.

• Architecture B: First 64 KB chunk is read and piped immediately:

TTFB_B = 64 KB / 100,000 KB/s = 0.00064 s = 0.64 milliseconds.

Writing begins in under 1 millisecond.

Step 4: Strategic Conclusion

Streaming decouples memory consumption from data size (O(1) memory complexity relative to file size N). Whole-payload buffering exhibits O(N) memory complexity that fundamentally fails to scale for media, analytics, and high-concurrency systems.