Epsilon-Delta Limit Definition, Derivatives & Difference Quotients

Calculus is the mathematical foundation of continuous variation, instantaneous rates of change, optimization landscapes, and physical simulation. Whether analyzing the trajectory of an autonomous vehicle, training multi-layer neural networks via backpropagation, or modeling fluid turbulence, calculus provides the analytical and computational tools required to model and optimize dynamic systems.

1. Foundational Architecture of Limits & The Epsilon-Delta Formalism#

In elementary algebra and precalculus, evaluating a function typically involves plugging in a discrete input value. However, modeling continuous physical phenomena—such as instantaneous velocity or asymptotic runtime behavior—demands examining the behavior of a function arbitrarily close to a point, rather than strictly at the point itself.

The Intuitive vs. Rigorous Limit#

Intuitively, we say that the limit of f(x) as x approaches c is L:

lim_{x → c} f(x) = L

if we can make the output values f(x) arbitrarily close to L by taking input values x sufficiently close to c (with x ≠ c).

While intuitive, this informal definition led to deep foundational paradoxes in early mathematical history regarding "infinitesimal" quantities. In the nineteenth century, mathematicians Augustin-Louis Cauchy and Karl Weierstrass established the modern, rigorous foundation of analysis: the Epsilon-Delta (ε, δ) Definition of a Limit.

The Formal Epsilon-Delta Definition#

Definition: Let f be a real-valued function defined on an open interval containing c, except possibly at c itself. We write:

lim_{x → c} f(x) = L

if and only if:

For every real number ε > 0 (no matter how small), there exists a corresponding real number δ > 0 such that:

whenever 0 < |x - c| < δ, it is strictly guaranteed that |f(x) - L| < ε.

+-----------------------------------------------------------------------------+
|                 THE EPSILON-DELTA GAME-THEORETIC MECHANISM                  |
+-----------------------------------------------------------------------------+
|                                                                             |
|  Output Space (y-axis):                                                     |
|       L + epsilon  +-------------------------------------+                  |
|                    |     Allowed Output Target Region    |                  |
|       Target L     * - - - - - - - - - - - - - - - - - - *                  |
|                    |     (Width: 2 * epsilon)            |                  |
|       L - epsilon  +-------------------------------------+                  |
|                                                                             |
|  Input Space (x-axis):                                                      |
|                 c - delta       Point c       c + delta                     |
|                    [---------------*---------------]                        |
|                         Required Input Domain Gate                          |
|                             (Width: 2 * delta)                              |
|                                                                             |
|  Principle: The challenger dictates target tolerance epsilon > 0.          |
|             The defender must construct a valid window delta > 0            |
|             such that every input within delta maps inside epsilon.         |
+-----------------------------------------------------------------------------+

One-Sided Limits & The Limit Existence Theorem#

In many engineering and piecewise systems, a function behaves differently when approached from the left versus the right:

• Left-Hand Limit: lim_{x → c^-} f(x) examines inputs x < c.
• Right-Hand Limit: lim_{x → c^+} f(x) examines inputs x > c.

The Limit Existence Theorem:

The two-sided limit lim_{x → c} f(x) exists and equals L if and only if both one-sided limits exist and are equal:

lim_{x → c^-} f(x) = lim_{x → c^+} f(x) = L.

Mathematical Continuity at a Point#

A function f is defined to be continuous at a point c if and only if three strict conditions are satisfied:

1. f(c) is defined (c resides within the domain of f).
2. The limit lim_{x → c} f(x) exists.
3. The limit equals the function value: lim_{x → c} f(x) = f(c).

If any of these conditions fail, the function exhibits a discontinuity:

• Removable Discontinuity: The limit exists, but f(c) is undefined or does not equal the limit (a "hole" in the graph).
• Jump Discontinuity: Both one-sided limits exist, but they are unequal: lim_{x → c^-} f(x) ≠ lim_{x → c^+} f(x) (e.g., the Heaviside step function).
• Essential / Infinite Discontinuity: One or both one-sided limits do not exist or diverge to ±∞ (e.g., f(x) = 1/x or f(x) = sin(1/x) near x = 0).

Key Analytical Theorems for Continuous Functions#

If a function f is continuous on a closed interval [a, b]:

1. The Intermediate Value Theorem (IVT): If u is any value between f(a) and f(b), there exists at least one point c in [a, b] such that f(c) = u. (This guarantees root existence in numerical root-finding algorithms like bisection).
2. The Extreme Value Theorem (EVT): The function f is guaranteed to attain an absolute maximum and an absolute minimum at least once on [a, b].

2. The Derivative: Instantaneous Rate of Change & Geometric Tangency#

Differential calculus formalizes the concept of instantaneous rate of change by transitioning from average rates of change across finite intervals to infinitesimal limits.

Secant Line to Tangent Line: The Difference Quotient#

Consider a smooth curve y = f(x). To measure the average rate of change between two points (x, f(x)) and (x + h, f(x + h)), we construct a secant line connecting them. The slope of this secant line is given by the Newton Difference Quotient:

m_secant = [ f(x + h) - f(x) ] / [ (x + h) - x ] = [ f(x + h) - f(x) ] / h.

As the interval width h approaches zero (h → 0), the second point slides along the curve toward the first point. The secant line pivots and, in the limit, becomes the tangent line to the curve at x.

Formal Definition of the Derivative#

The derivative of function f at point x, denoted f'(x) or df/dx, is the limit of the difference quotient as h approaches zero:

f'(x) = lim_{h → 0} [ f(x + h) - f(x) ] / h

provided this limit exists. If the limit exists, f is said to be differentiable at x.

An algebraically equivalent formulation at a fixed point c is:

f'(c) = lim_{x → c} [ f(x) - f(c) ] / [ x - c ].

Symmetric Difference Quotient#

In numerical analysis, the symmetric difference quotient provides superior computational accuracy:

f'(x) = lim_{h → 0} [ f(x + h) - f(x - h) ] / [ 2 · h ].

Because it evaluates the function symmetrically on both sides of x, odd-order error terms cancel out, improving truncation accuracy from O(h) to O(h^2).

Differentiability Implies Continuity#

Theorem: If a function f is differentiable at point c, then f is necessarily continuous at c.

Rigorous Analytical Proof:

We must prove that lim_{x → c} f(x) = f(c), or equivalently, lim_{x → c} [ f(x) - f(c) ] = 0.

For any x ≠ c, rewrite the difference:

f(x) - f(c) = [ (f(x) - f(c)) / (x - c) ] · (x - c).

Take the limit as x → c on both sides:

lim_{x → c} [ f(x) - f(c) ] = lim_{x → c} [ (f(x) - f(c)) / (x - c) ] · lim_{x → c} [ x - c ].

By definition of the derivative, the first limit is f'(c) (a finite real number). The second limit is 0.

lim_{x → c} [ f(x) - f(c) ] = f'(c) · 0 = 0.

Therefore, lim_{x → c} f(x) = f(c), proving continuity.

Crucial Counterexample (Continuity Does NOT Imply Differentiability):

The converse of this theorem is false. A function can be continuous everywhere while failing to be differentiable:

• The Absolute Value Function f(x) = |x|: Continuous at x = 0. However:

Left-hand difference quotient: lim_{h → 0^-} [ |0 + h| - 0 ] / h = lim_{h → 0^-} (-h) / h = -1.

Right-hand difference quotient: lim_{h → 0^+} [ |0 + h| - 0 ] / h = lim_{h → 0^+} (+h) / h = +1.

Because the one-sided limits differ (-1 ≠ +1), f'(0) does not exist (a sharp corner or "cusp").

• The Weierstrass Function: A continuous fractal curve that is differentiable nowhere on the entire real line!

3. Fundamental Rules of Differentiation & Algebraic Mechanics#

Rather than computing limits of difference quotients from scratch for every function, differential calculus establishes algebraic transformation rules.

1. The Power Rule#

For any real exponent n in R:

d/dx [ x^n ] = n · x^(n-1).

2. Linearity of the Derivative Operator#

The derivative is a linear operator. For any differentiable functions f and g and constants a, b in R:

d/dx [ a · f(x) + b · g(x) ] = a · f'(x) + b · g'(x).

3. The Product Rule#

When differentiating the product of two functions u(x) · v(x):

d/dx [ u · v ] = u' · v + u · v'.

Geometric Derivation:

Consider a rectangle with side lengths u and v and area A = u · v. If the side lengths increase by small increments Δu and Δv, the new area is:

(u + Δu) · (v + Δv) = u · v + u · Δv + v · Δu + Δu · Δv.

The change in area is:

ΔA = u · Δv + v · Δu + Δu · Δv.

Dividing by Δt and taking the limit as Δt → 0:

dA/dt = u · (dv/dt) + v · (du/dt) + 0, because the second-order corner term (Δu · Δv) vanishes in the limit.

4. The Quotient Rule#

When differentiating the ratio of two functions u(x) / v(x) (where v(x) ≠ 0):

d/dx [ u / v ] = [ u' · v - u · v' ] / [ v^2 ].

5. The Chain Rule: Composite Function Differentiation#

When a function is composed of nested functions, y = f(g(x)):

d/dx [ f(g(x)) ] = f'(g(x)) · g'(x).

In Leibniz notation, let y = f(u) where u = g(x):

dy/dx = (dy/du) · (du/dx).

Physical Intuition:

If a bicycle's pedals turn gear A, gear A turns gear B at 3 times its speed (du/dx = 3), and gear B turns the wheel at 2 times its speed (dy/du = 2), the wheel rotates at 2 · 3 = 6 times the pedal speed (dy/dx = 6). Derivatives multiply across composite chains.

4. Taylor Series & Higher-Order Polynomial Approximations#

In applied computing and scientific simulation, complex non-linear functions (e.g., trigonometric, exponential, logarithmic) are approximated locally using polynomial series.

First-Order Local Linearization (Tangent Line Approximation)#

Near a known reference point x0, the tangent line provides the optimal first-order linear approximation of f(x):

L(x) = f(x0) + f'(x0) · (x - x0).

For an input perturbation Δx = x - x0:

Δy ≈ f'(x0) · Δx.

Second-Order Quadratic Approximation (Curvature)#

Linear approximations fail to capture curvature (bending). Adding the second derivative f''(x0) produces the osculating parabola:

Q(x) = f(x0) + f'(x0) · (x - x0) + (1/2) · f''(x0) · (x - x0)^2.

The second derivative f''(x0) measures the rate of change of the slope:

• f''(x0) > 0: The curve is concave upward (convex). The tangent line lies strictly below the curve.
• f''(x0) < 0: The curve is concave downward. The tangent line lies strictly above the curve.
• f''(x0) = 0 with a change in concavity: An inflection point.

The General Taylor Series#

If f is infinitely differentiable in a neighborhood around x0, its Taylor series expansion is:

f(x) = sum(k=0 to infinity, [ f^(k)(x0) / k! ] · (x - x0)^k)

f(x) = f(x0) + f'(x0)(x - x0) + [ f''(x0) / 2! ](x - x0)^2 + [ f'''(x0) / 3! ](x - x0)^3 + ...

When expanded specifically around x0 = 0, the series is termed a Maclaurin Series:

• Exponential: e^x = 1 + x + x^2 / 2! + x^3 / 3! + ... = sum(k=0 to infinity, x^k / k!)
• Sine: sin(x) = x - x^3 / 3! + x^5 / 5! - x^7 / 7! + ...
• Cosine: cos(x) = 1 - x^2 / 2! + x^4 / 4! - x^6 / 6! + ...

Taylor's Remainder Theorem (Lagrange Form)#

Theorem: Let f be (n + 1)-times continuously differentiable on an interval containing x0 and x. The approximation error of truncating the Taylor polynomial at degree n is:

R_n(x) = f(x) - P_n(x) = [ f^(n+1)(ξ) / (n + 1)! ] · (x - x0)^(n + 1)

for some unknown point ξ residing strictly between x0 and x. This formula enables rigorous quantification of worst-case truncation error bounds in scientific numerical libraries.

5. Computational & Numerical Differentiation in Software#

In computational science, derivatives must frequently be calculated for functions that exist only as black-box simulation subroutines or compiled code.

Finite Difference Methods & The Truncation-Roundoff Dilemma#

Finite difference approximations replace mathematical limits with a small, finite step size h > 0:

1. Forward Difference:

f'(x) ≈ [ f(x + h) - f(x) ] / h.

By Taylor expansion: f(x + h) = f(x) + h · f'(x) + (h^2 / 2) · f''(ξ).

Truncation Error: E_trunc = O(h). (First-order accuracy).

1. Central Difference:

f'(x) ≈ [ f(x + h) - f(x - h) ] / [ 2 · h ].

By Taylor expansion:

f(x + h) = f(x) + h · f'(x) + (h^2 / 2) · f''(x) + (h^3 / 6) · f'''(x) + O(h^4)

f(x - h) = f(x) - h · f'(x) + (h^2 / 2) · f''(x) - (h^3 / 6) · f'''(x) + O(h^4)

Subtracting the two expansions cancels the even powers:

f(x + h) - f(x - h) = 2 · h · f'(x) + (h^3 / 3) · f'''(x) + O(h^5)

Dividing by 2h:

[ f(x + h) - f(x - h) ] / [ 2 · h ] = f'(x) + (h^2 / 6) · f'''(ξ).

Truncation Error: E_trunc = O(h^2). (Second-order accuracy).

+-----------------------------------------------------------------------------+
|                 NUMERICAL DIFFERENTIATION STEP SIZE DILEMMA                 |
+-----------------------------------------------------------------------------+
| Log Error                                                                   |
|     ^                                                                       |
|     |   \                                                  /                |
|     |    \  Roundoff Error Dominates                      / Truncation Error|
|     |     \ (Subtractive Cancellation)                   /  Dominates       |
|     |      \    ~ epsilon_machine / h                   /   ~ h^2           |
|     |       \                                          /                    |
|     |        `---.                                .---'                     |
|     |             `----.                    .----'                          |
|     |                   `---- Optimal h ----'                               |
|     +-------------------------------------------------------------------->  |
|     1e-16                 h_opt ≈ 1e-5                       1e-1    Log h  |
+-----------------------------------------------------------------------------+

The Catastrophic Cancellation Dilemma:

Engineers often assume that choosing h as small as possible (e.g., h = 10^(-16)) maximizes accuracy. This is a critical misconception.

In finite-precision floating-point arithmetic (IEEE-754 double precision with ε_machine ≈ 10^(-16)):

• Truncation error decreases as h → 0: E_trunc ≈ C1 · h^2.
• Roundoff error increases as h → 0: Computing f(x + h) - f(x - h) involves subtracting two nearly identical floating-point numbers, destroying significant digits. The roundoff error scales as E_round ≈ ε_machine / h.

The total error is minimized when the two error curves cross:

• For Forward Difference: h_optimal ≈ sqrt(ε_machine) ≈ sqrt(10^(-16)) = 10^(-8).
• For Central Difference: h_optimal ≈ (ε_machine)^(1/3) ≈ (10^(-16))^(1/3) ≈ 10^(-5.3) ≈ 10^(-5).

The Complex Step Derivative: Eliminating Cancellation Error#

To bypass catastrophic cancellation entirely, advanced scientific engines deploy the Complex Step Derivative Method:

Evaluate f using complex arithmetic with a purely imaginary perturbation i · h:

f(x + i · h) = f(x) + i · h · f'(x) - (h^2 / 2) · f''(x) - i · (h^3 / 6) · f'''(x) + ...

Extracting the imaginary component:

Im(f(x + i · h)) = h · f'(x) - (h^3 / 6) · f'''(x) + ...

Dividing by h:

f'(x) = Im(f(x + i · h)) / h + O(h^2).

Notice: There is NO subtraction of function values! Because no subtraction occurs, roundoff cancellation is completely eliminated. One can set h = 10^(-30) safely, achieving full machine-precision accuracy.

6. Systems & Computer Science Engineering Applications#

Derivatives and rates of change are pervasive across software engineering, compiler design, and Artificial Intelligence.

1. Deep Learning: Activation Functions & Gradient Flow#

In neural networks, weights are updated via gradient descent: W_new = W_old - η · (∂L / ∂W).

The derivative of the activation function directly determines whether training succeeds or stalls:

• Sigmoid: σ(x) = 1 / (1 + e^(-x)).

Derivative: σ'(x) = σ(x) · (1 - σ(x)).

The maximum value of σ'(x) is 0.25 (at x = 0). When stacking 20 layers, multiplying gradients across layers scales as (0.25)^20 ≈ 10^(-12). The gradient vanishes exponentially (Vanishing Gradient Problem), freezing weight updates in early layers.

• ReLU (Rectified Linear Unit): f(x) = max(0, x).

Derivative: f'(x) = 1 if x > 0, and 0 if x < 0.

Because the derivative is exactly 1 for all positive activations, gradients propagate across hundreds of layers without diminishing, enabling deep residual architectures (ResNets, Transformers).

2. High-Performance Graphics: Fast Inverse Square Root#

In 3D game engines (such as the famous id Software Quake III engine), rendering lighting and reflection vectors requires normalizing millions of 3D vectors per second:

v_norm = v / sqrt(x^2 + y^2 + z^2) = v · (1 / sqrt(magnitude)).

Calculating 1 / sqrt(x) via standard division and square root instructions was computationally expensive on early CPUs. John Carmack and engineers implemented the Fast Inverse Square Root algorithm:

1. Generate an astonishingly accurate initial guess using IEEE-754 bit-level integer manipulation with the magic constant 0x5f3759df.
2. Refine the approximation using a single iteration of Newton-Raphson root-finding:

To find 1 / sqrt(a), define f(x) = 1 / x^2 - a = 0.

Derivative: f'(x) = -2 / x^3.

Newton step: x_{k+1} = x_k - [ f(x_k) / f'(x_k) ] = x_k - [ (x_k^(-2) - a) / (-2 · x_k^(-3)) ]

Simplifying: x_{k+1} = x_k · (1.5 - 0.5 · a · x_k^2).

A single Newton iteration doubles the number of accurate digits, delivering 99.9% precision in a fraction of CPU cycles!

3. Distributed Systems: Latency Rate of Change (TCP BBR)#

In cloud infrastructure and edge networking, congestion control algorithms (like Google's BBR: Bottleneck Bandwidth and RTT) monitor network traffic rates. Rather than waiting for packet loss (the primitive approach of TCP Reno), BBR models the derivative of delivery rate with respect to inflight data volume:

d(Delivery_Rate) / d(Inflight_Bytes).

When this derivative approaches zero while the derivative of round-trip time d(RTT)/d(Inflight_Bytes) becomes strictly positive, the network bottleneck queue has begun filling. The algorithm immediately throttles transmission, maximizing throughput while eliminating bufferbloat latency.

7. Common Architectural Pitfalls & Misconceptions#

• Pitfall 1: Confusing Continuity with Differentiability.

A function can be continuous everywhere while failing to be differentiable at sharp points, corners, or vertical tangents. Never assume that because a sensor stream or cost curve is continuous, standard gradient-based optimization can be applied without subgradient smoothing.

• Pitfall 2: Selecting an Extremely Small Step Size in Numerical Differentiation.

Setting h = 1e-16 in standard forward or central difference code causes severe subtractive cancellation, producing completely garbage gradients (often 0.0 or random noise). Always use h ≈ 1e-5 for central differences or switch to complex-step differentiation.

• Pitfall 3: Misapplying L'Hôpital's Rule to Non-Indeterminate Forms.

L'Hôpital's Rule states that lim [ f(x) / g(x) ] = lim [ f'(x) / g'(x) ] ONLY if the original limit is an indeterminate form (0 / 0 or ±∞ / ±∞). Applying L'Hôpital's rule to determinate fractions (e.g., lim_{x → 0} [ (x + 1) / (x + 2) ] = 1/2) yields completely erroneous answers (lim [ 1 / 1 ] = 1).

• Pitfall 4: Neglecting the Chain Rule in Automatic Differentiation Graphs.

When manually implementing custom backpropagation layers in PyTorch or CUDA, developers frequently compute local partial derivatives while omitting the incoming adjoint gradient multiplication from downstream layers. Always remember: total gradient = local gradient × upstream gradient.

8. Summary & Checkpoint Exercises#

Solidify your mastery of limits, epsilon-delta formulations, derivatives, and Taylor approximations with these four checkpoint problems. Full step-by-step mathematical solutions are provided.

Problem 1 (Rigorous Epsilon-Delta Limit Proof):

Prove rigorously using the formal Epsilon-Delta (ε, δ) definition of a limit that:

lim_{x → 3} (4x - 5) = 7.

Solution:

Step 1: Identify Components.

f(x) = 4x - 5, c = 3, L = 7.

We must show that for any given ε > 0, there exists a δ > 0 such that:

whenever 0 < |x - 3| < δ, it is strictly guaranteed that |(4x - 5) - 7| < ε.

Step 2: Algebraic Simplification of the Output Error.

Evaluate |f(x) - L|:

|(4x - 5) - 7| = |4x - 12|.

Factor out the constant 4:

|4 · (x - 3)| = 4 · |x - 3|.

We want:

4 · |x - 3| < ε.

Divide both sides by 4:

|x - 3| < ε / 4.

Step 3: Choose δ and Complete the Proof.

Choose δ = ε / 4.

Now write the formal proof verification:

Assume 0 < |x - 3| < δ = ε / 4.

Then:

|f(x) - L| = |(4x - 5) - 7| = |4x - 12| = 4 · |x - 3| < 4 · (ε / 4) = ε.

Thus, whenever 0 < |x - 3| < δ, |f(x) - 7| < ε holds unconditionally.

This concludes the rigorous epsilon-delta proof: lim_{x → 3} (4x - 5) = 7.

Problem 2 (First-Principles Difference Quotient Derivative Derivation):

Using the formal limit definition of the derivative (the difference quotient), derive the derivative of:

f(x) = 1 / sqrt(x) for x > 0.

Verify your result using the power rule.

Solution:

Step 1: Set up the Difference Quotient.

f'(x) = lim_{h → 0} [ f(x + h) - f(x) ] / h

= lim_{h → 0} [ (1 / sqrt(x + h)) - (1 / sqrt(x)) ] / h.

Step 2: Find a Common Denominator.

The numerator is:

[ sqrt(x) - sqrt(x + h) ] / [ sqrt(x + h) · sqrt(x) ].

Substitute back into the difference quotient:

f'(x) = lim_{h → 0} [ sqrt(x) - sqrt(x + h) ] / [ h · sqrt(x + h) · sqrt(x) ].

Step 3: Rationalize the Numerator.

Multiply numerator and denominator by the conjugate [ sqrt(x) + sqrt(x + h) ]:

Numerator becomes:

[ sqrt(x) - sqrt(x + h) ] · [ sqrt(x) + sqrt(x + h) ] = (sqrt(x))^2 - (sqrt(x + h))^2 = x - (x + h) = -h.

Substitute into the limit expression:

f'(x) = lim_{h → 0} [ -h ] / [ h · sqrt(x + h) · sqrt(x) · (sqrt(x) + sqrt(x + h)) ].

Step 4: Cancel the Factor h (eliminating the 0/0 indeterminate form).

f'(x) = lim_{h → 0} [ -1 ] / [ sqrt(x + h) · sqrt(x) · (sqrt(x) + sqrt(x + h)) ].

Step 5: Evaluate the Limit as h → 0.

As h → 0, sqrt(x + h) → sqrt(x):

f'(x) = [ -1 ] / [ sqrt(x) · sqrt(x) · (sqrt(x) + sqrt(x)) ]

= [ -1 ] / [ x · (2 · sqrt(x)) ]

= -1 / [ 2 · x^(3/2) ].

Verification via Power Rule:

Rewrite f(x) = x^(-1/2).

f'(x) = (-1/2) · x^(-1/2 - 1) = (-1/2) · x^(-3/2) = -1 / [ 2 · x^(3/2) ].

Both analytical derivations match identically.

Problem 3 (Taylor Series Expansion and Truncation Error Bound):

Consider the natural logarithm function f(x) = ln(x) expanded around reference point x0 = 1.

1. Derive the third-degree Taylor polynomial P3(x).
2. Use P3(x) to approximate ln(1.2).
3. Use Taylor's Remainder Theorem (Lagrange Form) to compute a rigorous upper bound on the absolute approximation error.

Solution:

Part 1: Compute Derivatives at x0 = 1.

• f(x) = ln(x) => f(1) = ln(1) = 0
• f'(x) = 1/x = x^(-1) => f'(1) = 1/1 = 1
• f''(x) = -x^(-2) => f''(1) = -1/(1^2) = -1
• f'''(x) = 2 x^(-3) => f'''(1) = 2/(1^3) = 2
• f^(4)(x) = -6 x^(-4) => (needed for remainder term)

Assemble the Taylor Polynomial P3(x):

P3(x) = f(1) + f'(1)(x - 1) + [ f''(1) / 2! ](x - 1)^2 + [ f'''(1) / 3! ](x - 1)^3

P3(x) = 0 + 1 · (x - 1) - [ 1 / 2 ] · (x - 1)^2 + [ 2 / 6 ] · (x - 1)^3

P3(x) = (x - 1) - (1/2)(x - 1)^2 + (1/3)(x - 1)^3.

Part 2: Approximate ln(1.2).

Here x - 1 = 1.2 - 1 = 0.2.

P3(1.2) = 0.2 - (1/2)(0.2)^2 + (1/3)(0.2)^3

= 0.2 - 0.5(0.04) + (1/3)(0.008)

= 0.2 - 0.02 + 0.0026667

= 0.1826667.

(Analytical true value: ln(1.2) ≈ 0.1823216).

Part 3: Compute Lagrange Error Bound.

The remainder term for n = 3 is:

R3(x) = [ f^(4)(ξ) / 4! ] · (x - 1)^4

where ξ lies strictly between 1 and 1.2.

Since f^(4)(ξ) = -6 / ξ^4:

|R3(1.2)| = | [ -6 / ξ^4 ] / 24 | · (0.2)^4 = [ 1 / (4 · ξ^4) ] · 0.0016 = 0.0004 / ξ^4.

To maximize this error bound across ξ in [1, 1.2], we minimize the denominator by picking ξ = 1:

|R3(1.2)| <= 0.0004 / (1)^4 = 0.0004 = 4.0 × 10^(-4).

Actual Error Check:

|ln(1.2) - P3(1.2)| = |0.1823216 - 0.1826667| = 0.0003451 = 3.451 × 10^(-4).

Since 3.451 × 10^(-4) < 4.0 × 10^(-4), the Lagrange theoretical bound is verified.

Problem 4 (Optimal Numerical Differentiation Step Size & Error Estimation):

Suppose you are computing the derivative of f(x) = e^x at x = 1.0 using the Central Difference formula:

f'_approx(x) = [ f(x + h) - f(x - h) ] / [ 2 · h ].

Assume double-precision floating-point arithmetic with machine precision ε_machine = 1.11 × 10^(-16).

1. State the formula for the total error (truncation error + roundoff error) as a function of step size h.
2. Calculate the theoretical optimal step size h_opt that minimizes total error.
3. Calculate the expected minimum total error at h_opt.

Solution:

Part 1: Total Error Formulation.

1. Truncation Error:

From Taylor expansion, the truncation error of central differences is:

E_trunc(h) = (h^2 / 6) · |f'''(x)|.

For f(x) = e^x, f'''(x) = e^x. At x = 1.0, |f'''(1.0)| = e ≈ 2.71828.

E_trunc(h) = (e / 6) · h^2.

1. Roundoff Error:

In evaluating f(x + h) and f(x - h), each function evaluation incurs roundoff on the order of ε_machine · |f(x)|.

Subtracting them introduces uncertainty bounded by 2 · ε_machine · e.

Dividing by 2h yields the roundoff error bound:

E_round(h) = [ 2 · ε_machine · e ] / [ 2 · h ] = (e · ε_machine) / h.

Total Error Function:

E_total(h) = E_trunc(h) + E_round(h) = (e / 6) · h^2 + (e · ε_machine) / h.

Part 2: Calculate Optimal Step Size h_opt.

To minimize E_total(h), set the derivative with respect to h to zero:

d(E_total) / dh = (2 · e / 6) · h - (e · ε_machine) / h^2 = 0

(e / 3) · h = (e · ε_machine) / h^2

h^3 = 3 · ε_machine

h_opt = (3 · ε_machine)^(1/3).

Substitute ε_machine = 1.11 × 10^(-16):

3 · ε_machine = 3 · (1.11 × 10^(-16)) = 3.33 × 10^(-16).

h_opt = (3.33 × 10^(-16))^(1/3) ≈ 6.93 × 10^(-6).

Part 3: Expected Minimum Error at h_opt.

Evaluate E_total at h_opt = 6.93 × 10^(-6):

• Truncation Error:

E_trunc = (2.71828 / 6) · (6.93 × 10^(-6))^2 = 0.45305 · (4.802 × 10^(-11)) ≈ 2.175 × 10^(-11).

• Roundoff Error:

E_round = (2.71828 · 1.11 × 10^(-16)) / (6.93 × 10^(-6)) = (3.017 × 10^(-16)) / (6.93 × 10^(-6)) ≈ 4.354 × 10^(-11).

• Total Minimum Error:

E_total ≈ 2.175 × 10^(-11) + 4.354 × 10^(-11) ≈ 6.53 × 10^(-11).

Conclusion:

Choosing h ≈ 7 × 10^(-6) yields approximately 10 to 11 reliable decimal digits of precision for the derivative. Choosing h = 10^(-16) would destroy all precision, yielding an error of order 1.