Network scanning and vulnerability assessment constitute the transition from passive intelligence gathering to active technical interrogation. During this phase, security practitioners probe the target's network perimeter to discover live systems, map active listening services, fingerprint underlying operating system kernels, and systematically identify exploitable security vulnerabilities.
This guide provides an exhaustive, university-grade exploration of transport layer protocol mechanics (RFC 793, RFC 768), low-level port scanning algorithms, stateful firewall evasion techniques, operating system fingerprinting heuristics, vulnerability assessment architectures, and the formal mathematical scoring algorithms of the Common Vulnerability Scoring System (CVSS v3.1).
1. Transport Layer Protocols & State Machine Mechanics#
Every network scan fundamentally interacts with the transport layer of the OSI model. A rigorous understanding of Transmission Control Protocol (TCP, RFC 793) and User Datagram Protocol (UDP, RFC 768) packet structures and state machine transitions is necessary to interpret scan responses and deduce firewall behaviors.
TCP Three-Way Handshake (Normal Connection Establishment)
Client (Scanner) Server (Target)
| |
| ----------- 1. [SYN] Seq=X -------------> | (Enters SYN_RECEIVED)
(Enters SYN_SENT)| |
| <--- 2. [SYN, ACK] Seq=Y, Ack=X+1 ------ |
(Enters ESTAB) | |
| ----------- 3. [ACK] Ack=Y+1 -----------> | (Enters ESTABLISHED)
| |TCP Header Anatomy and Control Flags#
The TCP header comprises 20 bytes of mandatory fields (extending up to 60 bytes with options). The 12-bit flags field governs connection state transitions:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Destination Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Acknowledgment Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data | |C|E|U|A|P|R|S|F| |
| Offset| Res. |W|C|R|C|S|S|Y|I| Window |
| | |R|E|G|K|H|T|N|N| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum | Urgent Pointer |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The Critical TCP Flags in Port Scanning:
- SYN (Synchronize): Initiates a connection. Synchronizes sequence numbers between endpoints.
- ACK (Acknowledgment): Confirms receipt of previously transmitted data or control frames. The Acknowledgment Number field is valid only when ACK is asserted.
- RST (Reset): Abruptly terminates a connection. Emitted by operating systems when receiving packets destined for a closed port, an invalid sequence number, or a half-open socket.
- FIN (Finish): Initiates an orderly, four-way connection teardown, asserting the sender has finished transmitting data.
- PSH (Push): Commands the receiving TCP stack to bypass internal buffering and immediately deliver queued data to the user application.
- URG (Urgent): Signals that the Urgent Pointer field points to out-of-band data requiring priority processing.
RFC 793 State Transitions and the Reset Rule#
RFC 793 mandates exact behavior for closed ports:
- Section 3.4 (Reset Generation): "If the connection does not exist (CLOSED) then a reset is sent in response to any incoming segment except another reset. In particular, SYNs addressed to a non-existent connection are rejected by this means."
- If an incoming packet lacks an ACK flag, the returned RST packet sets
ACK = incoming_seq + incoming_lengthandSeq = 0. - If the incoming packet asserts an ACK flag, the returned RST packet sets
Seq = incoming_ackand omits the ACK flag.
This RFC standard rule enables port scanners to deduce the precise state of remote listening sockets.
2. Taxonomy of Port Scanning Methodologies#
Port scanning techniques manipulate TCP and UDP control headers to infer whether a port is Open (listening application), Closed (kernel resets connection), or Filtered (firewall drops packet or sends ICMP administratively prohibited error).
+-----------------------------------------------------------------------------------------+
| PORT SCANNING TAXONOMY |
+-----------------------------------------------------------------------------------------+
| Scan Technique | Probe Packet | Open Response | Closed Response |
+-----------------------+-------------------+-------------------+-------------------------+
| TCP Connect (-sT) | [SYN] | [SYN, ACK] | [RST, ACK] |
| TCP SYN Stealth (-sS)| [SYN] | [SYN, ACK] | [RST] |
| TCP FIN (-sF) | [FIN] | No response | [RST, ACK] |
| TCP Null (-sN) | None (0x00) | No response | [RST, ACK] |
| TCP Xmas (-sX) | [FIN, PSH, URG] | No response | [RST, ACK] |
| TCP ACK (-sA) | [ACK] | [RST] (Unfilter) | [RST] (Unfiltered) |
| UDP Scan (-sU) | UDP Datagram | Application data | ICMP Type 3 Code 3 |
+-----------------------------------------------------------------------------------------+1. TCP Connect Scan (`-sT`)#
The TCP Connect scan uses the standard Berkeley socket() and connect() system calls provided by the operating system kernel.
- Mechanics: The kernel initiates a full 3-way handshake:
SYN ──> SYN/ACK ──> ACK. Once established, the scanner immediately issues aclose()orFINteardown. - Advantages: Does not require administrative (root/raw socket) privileges. Works transparently across IPv4 and IPv6 network stacks.
- Disadvantages: Heavily logged. Because the 3-way handshake completes, the listening application's
accept()call unblocks, logging connection attempts in application access logs (e.g., Apache, Nginx, SSH auth logs). It is also slow because the operating system manages socket state overhead.
2. TCP SYN Stealth Scan (`-sS`)#
The SYN scan (often termed half-open scanning) is the industry standard for fast, high-volume port enumeration.
- Mechanics: The scanner crafts raw IP/TCP packets, transmitting a single
SYNpacket to the target port.
- If the port is Open, the target responds with SYN/ACK.
- The scanner's raw socket driver immediately transmits a RST packet, abruptly tearing down the half-open connection before the operating system kernel's accept() queue registers it.
- If the port is Closed, the target responds with RST.
- If the port is Filtered, no response is returned (dropped by firewall) or the firewall returns an ICMP Destination Unreachable (Type 3, Code 1, 2, 3, 9, 10, or 13).
- Advantages: Evades simple application-layer connection logging. Significantly faster than connect scanning.
TCP SYN Half-Open Scan Sequence (Open Port)
Scanner Target Port 443
| |
| ------------ 1. [SYN] Seq=100 ------------> | (Port is listening)
| |
| <--- 2. [SYN, ACK] Seq=500, Ack=101 ------- |
| |
| ------------ 3. [RST] Seq=101 ------------> | (Connection aborted!)
| |3. Inverse TCP Scans: FIN, Null, and Xmas Scans#
Inverse scanning techniques exploit subtle nuances in RFC 793 to bypass stateless access control lists and packet filters that block incoming SYN packets.
- TCP FIN Scan (`-sF`): Transmits a standalone
FINcontrol frame without prior handshake. - TCP Null Scan (`-sN`): Transmits a packet with all 6 control flags cleared (0x00).
- TCP Xmas Tree Scan (`-sX`): Transmits a packet with
FIN,PSH, andURGflags asserted simultaneously (0x29).
Evaluation Logic (RFC 793):
- If the destination port is Closed: The RFC mandates that the system return a
RSTpacket. - If the destination port is Open: The RFC states that incoming packets lacking SYN, ACK, or RST must be dropped silently without response!
- State Inference:
- RST received ──> Port is Closed.
- No response (timeout) ──> Port is Open|Filtered.
- The Windows / Non-Compliant Operating System Exception:
Microsoft Windows, Cisco IOS, BSDI, and IBM OS/400 implement TCP stacks that violate RFC 793: they return a RST packet for any unsolicited segment arriving without a SYN or ACK, regardless of whether the port is open or closed. Consequently, FIN, Null, and Xmas scans cannot distinguish open from closed ports on Windows hosts (all ports appear closed). However, this discrepancy provides a valuable OS fingerprinting signature.
4. TCP ACK Scan (`-sA`) & Firewall Rule Mapping#
The ACK scan is not used to determine whether a port is open. Instead, it is a specialized reconnaissance tool to map firewall rule sets and differentiate between Stateful and Stateless firewalls.
- Mechanics: The scanner transmits an unsolicited
ACKpacket (Seq=random, Ack=0). - Stateful Firewalls: Track connection states in a connection table (e.g., Linux
ip_conntrack/netfilter). An incoming ACK packet that does not match an active, established connection is recognized as out-of-state and silently dropped. - Stateless Firewalls: Simple router ACLs inspect packets individually. A stateless rule permitting outbound web traffic often permits any incoming packet with the
ACKflag set. - Response Analysis:
- RST received ──> The packet reached the host. The port is Unfiltered (the firewall allowed the packet through, and the target's TCP stack responded with RST).
- No response (timeout) or ICMP Type 3 error ──> The port is Filtered (a stateful firewall or ACL blocked the packet).
3. UDP Scanning Architecture & ICMP Rate Limiting#
Scanning User Datagram Protocol (UDP, RFC 768) ports presents distinct architectural challenges compared to TCP. UDP is connectionless; applications are not required to acknowledge receipt of incoming datagrams.
UDP Scan Outcomes
1. Open Port (Application Active)
Scanner ---------------- UDP Request (e.g. DNS Query) -------> Target (Port 53)
Scanner <--------------- UDP Response (DNS Answer) ---------- Target
2. Closed Port
Scanner ---------------- UDP Datagram ----------------------> Target (Port 999)
Scanner <--------------- ICMP Type 3, Code 3 (Unreachable) - Target
3. Filtered Port
Scanner ---------------- UDP Datagram ----------------------> Target (Port 161)
Target Firewall drops packet (No response returned / ICMP Type 3 Code 13)The Open|Filtered Ambiguity#
When a scanner transmits a UDP packet to a target port:
- If the target application responds with a UDP packet (e.g., a DNS query response on port 53, an SNMP reply on port 161, or an NTP response on port 123), the port is confirmed Open.
- If the port is Closed, the host operating system generates an ICMP Port Unreachable (Type 3, Code 3) packet.
- If the port is Open but the application does not generate a reply to unexpected payloads, or if a firewall silently drops the packet, the scanner receives no response.
Consequently, UDP scanners classify silent ports as Open|Filtered, requiring targeted application-layer probes to verify service presence.
Operating System ICMP Rate Limiting#
The primary operational constraint in UDP scanning is ICMP rate limiting enforced by operating system kernels to mitigate denial-of-service amplification:
- Linux Kernel: Enforces
net.ipv4.icmp_ratelimit = 1000(by default, limiting ICMP generation to at most one packet per 1,000 milliseconds / 1 second per destination or socket bucket). - Impact: Scanning all 65,535 UDP ports on a Linux server receiving at most 1 ICMP Port Unreachable per second requires:
Time = (65,535 ports) / (1 response/second) = 65,535 seconds ≈ 18.2 hours- Mitigation: Security assessors scan only the top 100 or top 1,000 well-known UDP ports (e.g., DNS 53, DHCP 67/68, TFTP 69, NTP 123, SNMP 161, ISAKMP 500, Syslog 514) and craft valid service-specific application payloads rather than empty UDP datagrams.
4. Service Identification & Operating System Fingerprinting#
Discovering that port 80 or 443 is open provides minimal offensive or defensive value without knowing the exact software version running on that port. Fingerprinting identifies the specific daemon, version, and operating system.
Service Version Detection (`-sV`)#
Modern scanners utilize a multi-stage probing pipeline:
- Null Probe: The scanner opens a connection and waits up to several seconds without sending data. Many services (SSH, FTP, SMTP, Telnet) immediately emit a welcome banner upon connection:
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
220 mail.enterprise.com ESMTP Postfix (Ubuntu)
- Targeted Service Probes: If the null probe yields no data, the scanner transmits protocol-specific requests (
GetRequest,GenericLines,Help,SSLSessionReq). - Regex Signature Matching: Responses are matched against a comprehensive database of regular expressions (such as
nmap-service-probes) containing thousands of service signatures, extracting product names, versions, hostnames, and operating system details.
TCP/IP Stack OS Fingerprinting (`-O`)#
Operating systems implement the TCP/IP protocol specifications with subtle variations in default parameters, flag handling, and sequence number generation. By transmitting a battery of 16 specialized probes (including invalid flag combinations, specific TCP options, and ICMP requests), scanners analyze the response matrix:
+-----------------------------------------------------------------------------------------+
| OS TCP/IP FINGERPRINTING SIGNATURE MATRIX |
+-----------------------------------------------------------------------------------------+
| Operating System | Default TTL | Initial Window Size | DF (Don't Fragment) Bit|
+-----------------------+---------------+-----------------------+-------------------------+
| Linux Kernel 5.x/6.x | 64 | 29,200 or 64,240 | Set (1) |
| Microsoft Windows 10/11/Server | 128 | 64,240 or 65,535 | Set (1) |
| FreeBSD / OpenBSD | 64 | 65,535 | Set (1) |
| Cisco IOS 12.x/15.x | 255 | 4,128 | Not Set (0) |
| Solaris / AIX | 255 | 8,760 or 32,768 | Not Set (0) |
+-----------------------------------------------------------------------------------------+Key Fingerprinting Vectors:
- Initial Time to Live (TTL): Reflected in incoming IP headers. Windows stacks initialize TTL to 128; Linux kernels initialize TTL to 64; enterprise routers initialize TTL to 255. Even after traversing H router hops, the remaining TTL (TTL_{received} = TTL_{initial} - H) immediately indicates the operating system family.
- TCP Options Ordering: The exact sequence of options (MSS, NOP, Window Scale, SACK-Permitted, Timestamp) in a
SYN/ACKpacket is hardcoded into the kernel driver. Linux, Windows, and macOS use distinct ordering. - TCP Initial Sequence Number (ISN) Predictability:
Historic stacks used linear ISN counters, enabling blind TCP spoofing (Mitnick attack). Modern operating systems utilize cryptographically secure pseudorandom number generators (CSPRNGs) with timestamp offsets to generate unpredictable ISNs. Scanners measure the statistical variance of ISN differences over time to identify legacy, vulnerable stacks.
5. Vulnerability Assessment Principles & The CVSS v3.1 Scoring Algorithm#
A vulnerability assessment is the automated, systematic identification, quantification, and prioritization of security vulnerabilities in a target environment.
Vulnerability Assessment vs. Penetration Testing#
- Vulnerability Assessment (VA): Prioritizes breadth over depth. Scans thousands of endpoints, cataloging all potential missing patches, misconfigurations, and outdated libraries. Does not exploit vulnerabilities to verify exploitability; results may contain false positives.
- Penetration Testing: Prioritizes depth and validation over breadth. Actively exploits discovered vulnerabilities, chains multiple low-severity issues into high-impact compromises, pivots into internal networks, and assesses the true business risk.
The Common Vulnerability Scoring System (CVSS v3.1)#
The Forum of Incident Response and Security Teams (FIRST) maintains the CVSS specification. CVSS v3.1 calculates a score between 0.0 and 10.0 based on three metric groups: Base, Temporal, and Environmental.
CVSS v3.1 Metric Architecture
+-----------------------------------+
| CVSS Metric Groups |
+-----------------+-----------------+
|
+-------------------------+-------------------------+
| | |
v v v
[ Base Metrics ] [ Temporal Metrics ] [ Environmental Metrics ]
- Exploitability - Exploit Code Maturity - Modified Base Metrics
- Scope - Remediation Level - Confidentiality Req
- Impact (CIA) - Report Confidence - Integrity/Avail ReqThe Formal Mathematical Derivation of the CVSS v3.1 Base Score#
The CVSS v3.1 Base Score is calculated via a non-linear piecewise function dependent on the Scope (S) metric:
1. Exploitability Sub-Score (ESS)
Exploitability = 8.22 × AV × AC × PR × UIWhere numerical metric values are:
- Attack Vector (AV): Network (0.85), Adjacent (0.62), Local (0.55), Physical (0.20)
- Attack Complexity (AC): Low (0.77), High (0.44)
- Privileges Required (PR):
- If Scope is Unchanged: None (0.85), Low (0.62), High (0.27)
- If Scope is Changed: None (0.85), Low (0.68), High (0.50)
- User Interaction (UI): None (0.85), Required (0.62)
2. Impact Sub-Score (ISS)
The raw Impact Sub-Score measures Confidentiality (C), Integrity (I), and Availability (A) impacts (None = 0.0, Low = 0.22, High = 0.56):
ISS = 1 - (1 - Impact_C) × (1 - Impact_I) × (1 - Impact_A)3. Modified Impact (Impact)
The calculated Impact depends on whether Scope is Unchanged or Changed:
- If Scope is Unchanged:
Impact = 6.42 × ISS- If Scope is Changed:
Impact = 7.52 × (ISS - 0.029) - 3.25 × (ISS - 0.02)^{15}4. Base Score Function
If Impact ≤ 0, the BaseScore = 0.0.
Otherwise:
- If Scope is Unchanged:
BaseScore = Roundup(min((Impact + Exploitability), 10.0))- If Scope is Changed:
BaseScore = Roundup(min(1.08 × (Impact + Exploitability), 10.0))Where Roundup(x) rounds up to the nearest single decimal place (e.g., 4.02 ──> 4.1, 4.00 ──> 4.0).
6. Enterprise Vulnerability Scanner Architecture#
Enterprise vulnerability scanners (e.g., Nessus, OpenVAS, Qualys, Rapid7 Nexpose) operate across distributed architectures:
+---------------------------------------------------------------------------------+
| ENTERPRISE SCANNER ARCHITECTURE |
+---------------------------------------------------------------------------------+
| |
| +-----------------------+ +----------------------------------+ |
| | Management Console & | <==========> | Vulnerability Database / Feeds | |
| | Policy Orchestration | | (NVD CVEs, OVAL, Vendor Errata) | |
| +-----------+-----------+ +----------------------------------+ |
| | |
| v |
| +-------------------------------------------------------------------------+ |
| | Distributed Scanning Engines / Daemons | |
| +-------------------------------------------------------------------------+ |
| | (Raw Sockets) | (SSH / WinRM / WMI) |
| v v |
| +---------------------------+ +-------------------------------+ |
| | Unauthenticated Network | | Authenticated Credentialed | |
| | Probing (Banners, SSL) | | Audit (RPM, DEB, Registry) | |
| +---------------------------+ +-------------------------------+ |
+---------------------------------------------------------------------------------+Authenticated vs. Unauthenticated Scanning#
- Unauthenticated (Network-Only) Scans:
- The scanner connects to externally listening ports from across the network.
- Relies entirely on service banners, SSL certificates, and protocol error behaviors.
- Limitations: High false-positive rate. Cannot evaluate applications listening exclusively on localhost (127.0.0.1), cannot audit file permissions, cannot inspect patched packages if backported by OS vendors (e.g., Debian/Red Hat security backports preserve legacy upstream version strings).
- Authenticated (Credentialed) Scans:
- The scanner authenticates directly to the target system via SSH (Linux) or WinRM/WMI (Windows) using dedicated, low-privilege audit credentials.
- Executes local inspection commands: queries package managers (rpm -qa, dpkg -l), inspects Windows Registry keys (HKLM > Software > Microsoft > Windows NT > CurrentVersion), and reviews kernel sysctl configurations.
- Benefits: Near-zero false-positive rate. Accurately identifies backported patches, audits local privilege escalation vectors, and evaluates configurations without transmitting intrusive network exploits.
7. Production-Grade Network Scanner Implementation#
The following complete TypeScript implementation creates an asynchronous, multi-threaded network scanner with concurrency pool management, socket timeout teardown, raw banner acquisition, and basic service identification:
/**
* Asynchronous Network Scanner & Service Identification Engine
* Built for Node.js 18+ utilizing native net and promises
*/
import * as net from 'node:net';
export interface ScanTarget {
host: string;
ports: number[];
timeoutMs: number;
}
export interface PortScanResult {
port: number;
state: 'OPEN' | 'CLOSED' | 'FILTERED';
banner?: string;
service?: string;
}
export class NetworkScanner {
private host: string;
private ports: number[];
private timeoutMs: number;
private concurrency: number;
constructor(host: string, ports: number[], timeoutMs: number = 1500, concurrency: number = 50) {
this.host = host;
this.ports = ports;
this.timeoutMs = timeoutMs;
this.concurrency = concurrency;
}
// Probe an individual TCP port with banner grabbing
private probePort(port: number): Promise<PortScanResult> {
return new Promise((resolve) => {
const socket = new net.Socket();
let isResolved = false;
let banner = '';
const finalize = (state: 'OPEN' | 'CLOSED' | 'FILTERED') => {
if (isResolved) return;
isResolved = true;
socket.removeAllListeners();
socket.destroy();
resolve({
port,
state,
banner: banner.trim() || undefined,
service: this.inferService(port, banner)
});
};
socket.setTimeout(this.timeoutMs);
socket.connect(port, this.host, () => {
// Connected successfully -> Port is OPEN. Wait briefly for banner.
// Send a generic probe to trigger a response
socket.write('HEAD / HTTP/1.0\r\n\r\n');
});
socket.on('data', (data) => {
banner += data.toString('utf8', 0, Math.min(data.length, 512));
finalize('OPEN');
});
socket.on('timeout', () => {
// If connected but no data returned, port is open without banner
if (!socket.connecting && socket.writable) {
finalize('OPEN');
} else {
finalize('FILTERED');
}
});
socket.on('error', (err: any) => {
if (err.code === 'ECONNREFUSED') {
finalize('CLOSED');
} else if (err.code === 'EHOSTUNREACH' || err.code === 'ETIMEDOUT') {
finalize('FILTERED');
} else {
finalize('CLOSED');
}
});
});
}
// Infer service based on port number and received banner
private inferService(port: number, banner: string): string {
const lowerBanner = banner.toLowerCase();
if (lowerBanner.includes('ssh')) return 'SSH';
if (lowerBanner.includes('http') || lowerBanner.includes('html')) return 'HTTP';
if (lowerBanner.includes('ftp')) return 'FTP';
if (lowerBanner.includes('smtp') || lowerBanner.includes('esmtp')) return 'SMTP';
// Fallback to standard IANA assignments
const commonServices: Record<number, string> = {
21: 'FTP',
22: 'SSH',
25: 'SMTP',
53: 'DNS',
80: 'HTTP',
110: 'POP3',
143: 'IMAP',
443: 'HTTPS',
3306: 'MySQL',
5432: 'PostgreSQL',
8080: 'HTTP-Proxy'
};
return commonServices[port] || 'Unknown';
}
// Concurrency-controlled execution pool
public async executeScan(): Promise<PortScanResult[]> {
console.log(`[*] Initiating scan against {this.host} across {this.ports.length} ports...`);
const results: PortScanResult[] = [];
const queue = [...this.ports];
const workers = Array.from({ length: this.concurrency }, async () => {
while (queue.length > 0) {
const port = queue.shift();
if (port === undefined) break;
const res = await this.probePort(port);
if (res.state === 'OPEN') {
results.push(res);
console.log(` [+] Port {res.port}/tcp OPEN ({res.service}) Banner: "{res.banner || 'None'}"`);
}
}
});
await Promise.all(workers);
return results;
}
}
// Module verification execution
if (require.main === module) {
const commonPorts = [21, 22, 23, 25, 53, 80, 110, 143, 443, 3306, 5432, 8080];
const scanner = new NetworkScanner('127.0.0.1', commonPorts, 1000, 10);
scanner.executeScan().then((openPorts) => {
console.log(`\n[+] Scan Complete. Total Open Ports Found: {openPorts.length}`);
});
}8. Mathematical Foundations of Network Scanning#
Understanding scanning dynamics requires quantitative models of bandwidth consumption, scan duration, packet loss retransmission probabilities, and Bayesian alert classification.
Bandwidth, Packet Size, and Completion Time#
A stateless scanner transmitting probes to N_{hosts} target hosts across N_{ports} ports transmits a total packet volume of:
P_{total} = N_{hosts} × N_{ports} × RWhere R is the probe redundancy factor (number of retries per port, typically R = 1 or R = 2).
Each TCP SYN probe consists of:
- Ethernet Frame Header: 14 bytes + 4 bytes FCS = 18 bytes
- IPv4 Header: 20 bytes (without options)
- TCP Header: 20 bytes (or 24 bytes with MSS option)
- Preamble + Inter-packet Gap: 20 bytes
Total Frame Size = 18 + 20 + 20 + 20 = 78 bytes (624 bits)Given an allocated upload bandwidth of B bits per second:
Scan Duration T = (N_{hosts} × N_{ports} × R × 624) / (B)Example:
Scanning an entire /16 network (N_{hosts} = 65,536) across the top 1,000 ports (N_{ports} = 1,000) with R = 1 over a 10 Mbps (10 × 10^6 bps) link:
P_{total} = 65,536 × 1,000 = 65,536,000 packetsTotal Bits = 65,536,000 × 624 = 40,894,464,000 bitsT = (40,894,464,000) / (10,000,000) = 4,089.4 seconds ≈ 68.16 minutes### Packet Loss and Retransmission Probability
If a network drops packets with an independent probability p_{loss}, the probability that an open port is incorrectly categorized as closed/filtered after k probes is the probability that all k probes (or their returning SYN/ACKs) are lost:
P(False Negative) = p_{loss}^kThe probability of discovering the open port with at least one successful probe is:
P(Detection) = 1 - p_{loss}^kIf p_{loss} = 0.15 (15% packet loss on a congested link):
- With k = 1 probe: P(Detection) = 1 - 0.15 = 0.85 (85% accuracy, 15% missed ports).
- With k = 2 probes: P(Detection) = 1 - 0.15^2 = 1 - 0.0225 = 0.9775 (97.75% accuracy).
- With k = 3 probes: P(Detection) = 1 - 0.15^3 = 1 - 0.003375 = 0.9966 (99.66% accuracy).
This demonstrates why setting a probe retry count of k = 2 or k = 3 is essential in production scanners operating over lossy WAN connections.
Bayesian Probability in Vulnerability Triage#
Vulnerability scanners report potential vulnerabilities based on banner matching and signature heuristics. Bayes' Theorem formalizes the True Positive probability:
P(Vulnerable | Alert) = (P(Alert | Vulnerable) · P(Vulnerable)) / (P(Alert))Where:
P(Alert) = P(Alert | Vulnerable) · P(Vulnerable) + P(Alert | Not Vulnerable) · P(Not Vulnerable)If a vulnerability has a low prior prevalence in the environment (P(Vulnerable) = 0.01), the scanner has a sensitivity of 95% (P(Alert | Vulnerable) = 0.95), and a false-positive rate of 5% (P(Alert | Not Vulnerable) = 0.05):
P(Alert) = (0.95 × 0.01) + (0.05 × 0.99) = 0.0095 + 0.0495 = 0.059P(Vulnerable | Alert) = (0.0095) / (0.059) ≈ 0.161 (16.1%)Even with a 95% accurate test, when baseline prevalence is low, 83.9% of reported alerts are false positives! This mathematical reality explains why manual penetration testing is indispensable for validating vulnerability scanner outputs.
9. Real-World Pitfalls, Detection Signatures & Evasion Mechanics#
Network scanning triggers multiple security alarms if executed without architectural awareness.
+-----------------------------------------------------------------------------------------+
| SCANNING PITFALL & MITIGATION MATRIX |
+-----------------------------------------------------------------------------------------+
| Defense Mechanism | Trigger Signature | Offensive / Audit Bypass |
+---------------------------------+--------------------------+----------------------------+
| Stateful Connection Exhaustion | Millions of half-open | Rate-limit SYN probing; |
| (Linux conntrack table crash) | SYN sockets per second | send RST to clear state |
+---------------------------------+--------------------------+----------------------------+
| Suricata / Snort IDS Alerts | Sequential port sweeps | Randomize port and IP |
| | (1, 2, 3... 1024) | order; slow packet rates |
+---------------------------------+--------------------------+----------------------------+
| Active Shunning / Fail2ban | Repeated connection to | Decoy IP scanning (-D); |
| | TCP port with no data | distributed multi-source |
+---------------------------------+--------------------------+----------------------------+
| Fragment Reassembly Inspection | Tiny IP fragments | Ensure fragment offsets |
| | (overlapping offsets) | comply with modern RFCs |
+---------------------------------+--------------------------+----------------------------+1. State Table Exhaustion in Network Firewalls#
Stateful firewalls and NAT gateways maintain an in-memory connection tracking table (e.g., nf_conntrack in Linux). High-speed SYN scanning (such as masscan running at 100,000 packets/second) across internal network paths can exhaust the firewall's state table within seconds, causing a complete denial-of-service (DoS) that drops all legitimate enterprise traffic.
- Remediation: Throttle scan throughput to safe thresholds (e.g., under 500 packets/second) when scanning through intermediate firewall appliances, or execute scans from within the target subnet.
2. Evading Network Intrusion Detection Systems (NIDS)#
NIDS sensors (Snort, Suricata, Zeek) identify port scans using threshold-based detection rules (e.g., a single source IP probing more than 15 unique destination ports on a single host within a 2-second window).
- Decoy Scanning (`-D`): Intersperses probes from the assessor's real IP with probes spoofed from legitimate third-party IP addresses:
nmap -D RND:10,192.168.1.5,RND:5 192.168.1.100
The target IDS logs alerts containing 16 different IP addresses simultaneously, preventing automated shunning scripts from identifying the authentic source.
- Packet Fragmentation (`-f`, `--mtu`): Splits the 20-byte TCP header across multiple IP fragments (e.g., 8 bytes in fragment 1, 12 bytes in fragment 2). Older stateless packet filters fail to inspect the TCP port numbers because the port fields reside in the second fragment.
10. Checkpoint Practice Problems & Step-by-Step Solutions#
Problem 1: TCP Handshake & Inverse Scan Packet Exchange Analysis#
Scenario: A penetration tester probes port 22 (SSH) and port 80 (HTTP) on a hardened FreeBSD server. An intermediate stateless packet filter allows all inbound traffic to port 80 and port 22, but drops incoming packets that assert the SYN flag without the ACK flag on port 22.
The tester transmits three distinct probe frames to both ports:
- Probe 1:
TCP [SYN]packet to Port 80 and Port 22. - Probe 2:
TCP [FIN]packet to Port 80 and Port 22. - Probe 3:
TCP [ACK]packet to Port 80 and Port 22.
Assume port 80 is Open (listening web server) and port 22 is Open (listening SSH daemon). FreeBSD strictly complies with RFC 793.
Questions:
- Detail the exact response packet (including TCP flags) received for Probe 1 on Port 80 and Port 22.
- Detail the exact response packet (including TCP flags) received for Probe 2 on Port 80 and Port 22.
- Detail the exact response packet (including TCP flags) received for Probe 3 on Port 80 and Port 22.
- Explain how the tester deduces that the firewall is filtering
SYNpackets on port 22.
Solution:
Step 1: Evaluate Probe 1 (`TCP [SYN]`):
- Port 80: The packet filter permits incoming SYN packets to port 80. Port 80 is listening.
- Response: Target responds with `TCP [SYN, ACK]`.
- Port 22: The packet filter rule explicitly drops incoming packets that assert
SYNwithoutACK.
- Response: The packet is dropped by the filter. No response (Timeout).
Step 2: Evaluate Probe 2 (`TCP [FIN]` - Inverse Scan):
- Port 80: The packet filter allows the packet. The packet reaches FreeBSD's TCP stack on port 80. Port 80 is Open. RFC 793 mandates that any segment arriving at an open port lacking SYN, ACK, or RST must be dropped silently.
- Response: No response (Silent Drop).
- Port 22: The packet filter permits the FIN packet (since it does not assert SYN). The packet reaches FreeBSD's open SSH socket. By RFC 793, it is dropped silently.
- Response: No response (Silent Drop).
Step 3: Evaluate Probe 3 (`TCP [ACK]` Scan):
- Port 80: The packet filter permits incoming ACK packets. The packet reaches port 80. Because the ACK does not correspond to an established connection, RFC 793 mandates that the host return a reset.
- Response: `TCP [RST]` (Port state inferred as Unfiltered).
- Port 22: The packet filter permits the ACK packet (the drop rule only applies to SYN-without-ACK). The packet reaches port 22. RFC 793 mandates a reset.
- Response: `TCP [RST]` (Port state inferred as Unfiltered).
Step 4: Deducing the Firewall Rule:
- On Port 22, the tester observed:
- SYN probe ──> Timeout (Filtered).
- ACK probe ──> RST returned (Unfiltered).
- Because the
ACKprobe returnedRST, the tester knows that port 22 is physically reachable through the network routing path. The fact thatSYNtimed out whileACKreturnedRSTmathematically proves that a stateless packet filter or ACL is actively dropping incoming connection requests (SYN) while permitting other TCP control flags!
Problem 2: Full Mathematical Derivation of a CVSS v3.1 Base Score#
Scenario: A vulnerability assessment identifies an unauthenticated Remote Code Execution (RCE) flaw in an enterprise hypervisor web console (CVE-2026-99411).
The vulnerability parameters are assessed as:
- Attack Vector (AV): Network (
AV:N) - Attack Complexity (AC): Low (
AC:L) - Privileges Required (PR): None (
PR:N) - User Interaction (UI): None (
UI:N) - Scope (S): Changed (
S:C) (The vulnerability allows an attacker inside the web console guest container to compromise the physical bare-metal host operating system). - Confidentiality Impact (C): High (
C:H) - Integrity Impact (I): High (
I:H) - Availability Impact (A): High (
A:H)
Questions:
- Assign the appropriate numerical coefficient to each metric under CVSS v3.1 specifications.
- Calculate the Exploitability Sub-Score (ESS).
- Calculate the Impact Sub-Score (ISS) and Modified Impact.
- Calculate the final CVSS v3.1 Base Score and assign its qualitative severity rating (None, Low, Medium, High, Critical).
Solution:
Step 1: Metric Coefficients Assignment:
- Attack Vector (Network,
AV:N): 0.85 - Attack Complexity (Low,
AC:L): 0.77 - Privileges Required (None,
PR:Nwith Scope Changed): 0.85 - User Interaction (None,
UI:N): 0.85 - Impact Confidentiality (High,
C:H): 0.56 - Impact Integrity (High,
I:H): 0.56 - Impact Availability (High,
A:H): 0.56
Step 2: Calculate Exploitability Sub-Score (ESS):
ESS = 8.22 × AV × AC × PR × UIESS = 8.22 × 0.85 × 0.77 × 0.85 × 0.85ESS = 8.22 × 0.5574075 × 0.7225 = 8.22 × 0.4027269 = 3.3104Step 3: Calculate Impact Sub-Score (ISS) & Modified Impact:
First, calculate ISS:
ISS = 1 - (1 - Impact_C) × (1 - Impact_I) × (1 - Impact_A)ISS = 1 - (1 - 0.56) × (1 - 0.56) × (1 - 0.56)ISS = 1 - (0.44 × 0.44 × 0.44) = 1 - 0.085184 = 0.914816Because Scope is Changed (`S:C`), calculate Modified Impact:
Impact = 7.52 × (ISS - 0.029) - 3.25 × (ISS - 0.02)^{15}ISS - 0.029 = 0.914816 - 0.029 = 0.8858167.52 × 0.885816 = 6.661336Now evaluate the second term:
ISS - 0.02 = 0.914816 - 0.02 = 0.894816(0.894816)^{15} ≈ 0.1873763.25 × 0.187376 ≈ 0.608972Subtract the second term from the first:
Impact = 6.661336 - 0.608972 = 6.052364Step 4: Calculate Final Base Score:
Because Scope is Changed, apply the changed scope formula:
BaseScore = Roundup(min(1.08 × (Impact + Exploitability), 10.0))Impact + Exploitability = 6.052364 + 3.3104 = 9.3627641.08 × 9.362764 = 10.111785Apply the cap of 10.0:
min(10.111785, 10.0) = 10.0BaseScore = 10.0Qualitative Severity:
- CVSS Range 9.0 - 10.0: CRITICAL.
The vulnerability receives the maximum possible rating of 10.0 Critical, requiring immediate zero-day remediation.
Problem 3: Large-Scale Subnet Scanning Bandwidth, Timing & Packet Loss#
Scenario: A security audit firm is commissioned to perform a comprehensive vulnerability scan across an enterprise's external network footprint comprising four contiguous `/20` IPv4 subnets. The scan policy mandates probing the top 2,000 TCP ports on every IP address.
To account for internet transit packet drops, the scan tool is configured with a probe redundancy factor of R = 2 (each port receives two probes). The scanning engine transmits standard 78-byte TCP SYN frames.
Questions:
- Calculate the total number of IP addresses across four
/20subnets. - Calculate the total number of packets that will be generated during this scan.
- If the audit firm's uplink bandwidth is capped at 25 Megabits per second (Mbps) dedicated to scanning traffic, calculate the minimum theoretical completion time in hours.
- If an unmanaged intermediate router exhibits an average packet loss rate of p_{loss} = 0.08 (8%), calculate the probability that an open port fails to respond to both probes (resulting in a false negative).
Solution:
Step 1: Calculate Total IP Addresses:
A /20 subnet contains:
N_{subnet} = 2^{32 - 20} = 2^{12} = 4,096 IP addressesAcross four /20 subnets:
N_{total} = 4 × 4,096 = 16,384 IP addressesStep 2: Calculate Total Packets Generated:
Given N_{ports} = 2,000 and R = 2:
P_{total} = N_{total} × N_{ports} × RP_{total} = 16,384 × 2,000 × 2 = 65,536,000 packetsStep 3: Calculate Minimum Theoretical Completion Time:
Each frame size is 78 bytes:
Total Data Volume in Bytes = 65,536,000 × 78 bytes = 5,111,808,000 bytesConvert to bits:
Total Bits = 5,111,808,000 × 8 = 40,894,464,000 bitsUplink bandwidth B = 25 Mbps = 25,000,000 bits/second:
T_{seconds} = (40,894,464,000) / (25,000,000) ≈ 1,635.78 secondsConvert to hours:
T_{hours} = (1,635.78) / (3,600) ≈ 0.454 hours ≈ 27.26 minutesStep 4: Calculate False Negative Probability:
With packet loss p_{loss} = 0.08 and R = 2 independent probes:
P(False Negative) = p_{loss}^2 = (0.08)^2 = 0.0064 (0.64%)The dual-probe policy ensures a 99.36% detection probability, keeping missed open ports well under 1%.
Problem 4: Zombie / Idle Scan IP ID Progression & Blind Port State Deduction#
Scenario: An offensive operator executes an Idle Scan (nmap -sI) to anonymously probe port 443 on target Target-Host (198.51.100.80) using a vulnerable, quiescent printer Zombie-Host (203.0.113.15) that utilizes an incremental IP Identification (IP ID) counter.
The test procedure executes as follows:
- Probe Phase 1: The operator sends a
SYN/ACKpacket toZombie-Host.Zombie-Hostreturns aRSTcontainingIP ID = 31024. - Exploitation Phase: The operator sends a spoofed
SYNpacket toTarget-Host:443, spoofing the source IP address asZombie-Host(203.0.113.15). - Probe Phase 2: The operator sends a second
SYN/ACKpacket toZombie-Host.Zombie-Hostreturns aRSTcontainingIP ID = 31026.
Questions:
- What was the exact sequence of packets exchanged between
Target-HostandZombie-Hostduring Phase 2? - Deduce whether port 443 on
Target-Hostis Open, Closed, or Filtered. - What would the resulting IP ID have been if port 443 on
Target-Hostwere Closed? - What would the resulting IP ID have been if port 443 on
Target-Hostwere Filtered by a drop firewall?
Solution:
Step 1: Reconstruct the Packet Flow:
- In Phase 1, the scanner checked
Zombie-Host's counter:IP ID = 31024. - In Phase 2, the spoofed
SYNarrived atTarget-Host:443. - Because
Target-Host:443is Open,Target-Hostresponded to the perceived sender (Zombie-Host) with a `SYN/ACK`. Zombie-Hostreceived an unsolicitedSYN/ACK. According to RFC 793, receiving an unsolicitedSYN/ACKrequires emitting a `RST`.- Emitting this
RSTpacket causedZombie-Host's kernel to increment its internal IP ID counter by 1 (31024 ──> 31025). - In Phase 3, the scanner probed
Zombie-Hostwith anotherSYN/ACK.Zombie-Hostemitted its normalRSTback to the scanner, incrementing its IP ID counter by 1 again (31025 ──> 31026). - The scanner received
IP ID = 31026.
Step 2: Port State Deduction:
Δ IP ID = Final IP ID - Initial IP ID = 31026 - 31024 = 2An increment of 2 proves that Zombie-Host transmitted an intermediate packet in response to traffic from Target-Host.
Therefore, port 443 on Target-Host is OPEN!
Step 3: Outcome if Port 443 Were Closed:
- If port 443 were closed,
Target-Hostwould have sent aRSTtoZombie-Host. - By RFC 793, an incoming
RSTsegment is dropped silently without response! Zombie-Hostwould not have transmitted any packet.- In Phase 3,
Zombie-Hostwould respond to the scanner withIP ID = 31025(Δ IP ID = 1). - Thus, if Closed, the resulting IP ID would be 31025.
Step 4: Outcome if Port 443 Were Filtered:
- If port 443 were filtered (dropped),
Target-Hostwould have sent nothing. Zombie-Hostwould receive nothing and transmit nothing.- In Phase 3,
Zombie-Hostwould respond to the scanner withIP ID = 31025(Δ IP ID = 1). - Thus, if Filtered, the resulting IP ID would also be 31025.