Deployment Strategies - Blue-Green & Canary

In distributed microservice architectures and mission-critical cloud applications, deploying software updates without service interruption is a fundamental engineering requirement. The practice of scheduling multi-hour maintenance windows—taking services offline at midnight while database administrators execute manual SQL scripts and engineers swap executable binaries—is obsolete. Modern continuous delivery mandates zero-downtime deployments, rapid defect containment, and immediate, automated rollbacks.

Achieving zero-downtime delivery is not merely an operational convenience; it requires deliberate architectural decoupling between deployment (installing software into an environment) and release (making functionality visible to end users). This guide explores the mechanical, mathematical, and architectural foundations of modern deployment strategies, focusing deeply on Blue-Green environments, progressive Canary analysis, service mesh traffic management, and backward-compatible database schema migrations.

Comparative Taxonomy of Deployment Strategies#

Deployment strategies exist along a spectrum of trade-offs balancing infrastructure cost, operational complexity, blast radius containment, and data consistency risks.

+---------------------------------------------------------------------------------------------------+
|                              Deployment Strategy Comparative Matrix                               |
+-------------------+-------------+-------------------+---------------------+-----------------------+
| Strategy          | Downtime    | Infrastructure    | Rollback Duration   | Traffic Version       |
|                   | Risk        | Overhead          |                     | Concurrency           |
+-------------------+-------------+-------------------+---------------------+-----------------------+
| Recreate          | Yes (High)  | 0% (Minimal)      | Slow (Redeploy old) | Single (No skew)      |
| Rolling Update    | None (Zero) | ~25% (maxSurge)   | Moderate (Stepwise) | Mixed (v1 and v2)     |
| Blue-Green        | None (Zero) | 100% (2x capacity)| Instant (< 5 sec)   | Single at cutover     |
| Canary            | None (Zero) | ~10% - 25%        | Rapid (< 30 sec)    | Dual (Controlled %)   |
| Shadow (Dark)     | None (Zero) | 100%              | Instant (Drop fork) | Forked (Duplicated)   |
+-------------------+-------------+-------------------+---------------------+-----------------------+

1. Recreate (Big-Bang) Strategy#

In a recreate deployment, all existing instances running Version 1 (v1) are terminated simultaneously. Once all v1 instances have exited, the new Version 2 (v2) instances are provisioned and booted.

• Advantages: Eliminates version concurrency; at no point do v1 and v2 run simultaneously or interact with the shared database concurrently. Simplifies database migrations. Zero extra infrastructure cost.
• Disadvantages: Inherent downtime during the termination and boot interval (Downtime = T_shutdown + T_boot + T_healthcheck). Unacceptable for consumer-facing or high-availability enterprise applications.

2. Rolling Update Strategy#

A rolling update replaces pods or virtual machines incrementally. A subset of v1 instances is taken offline while an equivalent or surge capacity of v2 instances is spawned. Traffic is load-balanced across the pool as new instances pass readiness probes.

• Advantages: Zero downtime; minimal infrastructure surge capacity (e.g., 25% additional resources during rollout).
• Disadvantages: Introduces version skew. For a transient period (often minutes to hours), both v1 and v2 instances handle live production traffic simultaneously. Application code and database schemas must remain strictly backward- and forward-compatible.

Time 0: [v1] [v1] [v1] [v1]  (100% v1)
Time 1: [v1] [v1] [v1] [v2]  (75% v1, 25% v2)
Time 2: [v1] [v1] [v2] [v2]  (50% v1, 50% v2)
Time 3: [v1] [v2] [v2] [v2]  (25% v1, 75% v2)
Time 4: [v2] [v2] [v2] [v2]  (100% v2 complete)

3. Blue-Green (Red-Black) Deployment#

Blue-Green deployment provisions two identical, independent production environments:

• Blue Environment: The currently active production stack handling 100% of live user traffic.
• Green Environment: The idle, standby stack where the new version (v2) is deployed and verified using internal synthetic smoke tests without exposing it to public users.

Once the Green environment is verified, a high-level networking router, load balancer, or DNS record atomically switches live traffic from Blue to Green.

• Advantages: Instantaneous cutover; instant rollback (switching the router back to Blue if an anomaly is detected); zero mixed-version traffic.
• Disadvantages: Double infrastructure cost (requires 2× production capacity during deployment); long-lived stateful connections (WebSockets, long-polling HTTP) may experience abrupt termination upon switchover.

+---------------------------------------------------------------------------------------+
|                               Blue-Green Cutover Flow                                 |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   Live User Traffic ---> [ Edge Router / Load Balancer ]                              |
|                                     |                                                 |
|                        +------------+------------+                                    |
|                        | (Active: 100%)          | (Standby: 0% / Internal QA)        |
|                        v                         v                                    |
|             +---------------------+   +---------------------+                         |
|             |  Blue Environment   |   |  Green Environment  |                         |
|             |  (Running v1.0.0)   |   |  (Running v2.0.0)   |                         |
|             +---------------------+   +---------------------+                         |
|                        |                         |                                    |
|                        +------------+------------+                                    |
|                                     v                                                 |
|                        [ Shared Production Database ]                                 |
|                                                                                       |
+---------------------------------------------------------------------------------------+

4. Canary Deployment & Progressive Delivery#

Canary deployment introduces the new version (v2) to a tiny fraction of live user traffic (e.g., 1%, 5%, or 10%) while the overwhelming majority (90% to 99%) remains on the stable baseline (v1). Telemetry metrics (HTTP error rates, P99 latency, CPU utilization, unhandled exceptions) from the canary cohort are continuously compared against baseline metrics.

If the canary metrics remain healthy across designated evaluation intervals, traffic is progressively incremented (e.g., 5% -> 25% -> 50% -> 100%). If anomalies or metric degradation occur, the canary traffic is immediately severed, returning 100% of traffic to the stable baseline.

Traffic Routing & Shifting Mechanics#

Implementing granular traffic shifting requires intelligent ingress proxies, API gateways, or service meshes capable of weighted routing and header-based steering.

1. Ingress & Weighted Target Groups#

Modern application load balancers (such as AWS Application Load Balancer or NGINX Ingress Controller) support weighted routing rules at the HTTP listener level:

# Kubernetes NGINX Ingress Canary Configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: order-service-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
    - host: api.devthesis.com
      http:
        paths:
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: order-service-v2
                port:
                  number: 8080

Under this configuration, the NGINX Ingress controller uses an internal pseudo-random number generator per HTTP request: exactly 10% of incoming requests matching /orders are forwarded to order-service-v2, while 90% continue to the production service.

2. Service Mesh Traffic Splitting with Istio#

In service mesh architectures, Envoy sidecar proxies intercept all east-west (service-to-service) and north-south (ingress) traffic. Istio controls traffic distribution through VirtualService and DestinationRule custom resources:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: payment-service-subsets
spec:
  host: payment-service
  subsets:
    - name: stable
      labels:
        version: v1.12.0
    - name: canary
      labels:
        version: v1.13.0-rc1
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payment-service-routing
spec:
  hosts:
    - payment-service
  http:
    # Header-based override for internal QA testing
    - match:
        - headers:
            x-qa-tester:
              exact: "true"
      route:
        - destination:
            host: payment-service
            subset: canary
    # Statistical weighted traffic split for public traffic
    - route:
        - destination:
            host: payment-service
            subset: stable
          weight: 95
        - destination:
            host: payment-service
            subset: canary
          weight: 5

Automated Canary Analysis (ACA) & Metric Evaluation#

Manual observation of dashboards during a deployment is error-prone, subjective, and unsustainable at scale. Automated Canary Analysis (ACA) executes automated statistical evaluations comparing telemetry from the canary cohort against the baseline cohort over identical time windows.

Core Canary Telemetry Metrics#

1. HTTP Error Rate (5xx Ratio):

```

R_error = (∑ requests with HTTP 5xx) / (∑ total requests)

```

1. P95 and P99 Latency: Percentile response times measured in milliseconds, capturing tail latency degradations caused by unoptimized database queries, thread pool contention, or garbage collection pauses.
2. Application Resource Saturation: Container CPU throttling percentage, Resident Set Size (RSS) memory consumption, and heap allocation rates.
3. Custom Business Invariants: E-commerce checkout success rate, payment gateway authorization success rate, shopping cart abandonments.

Statistical Hypothesis Testing: Mann-Whitney U Test#

Simple threshold comparisons (e.g., error_rate > 1%) frequently fail during traffic fluctuations or micro-bursts, causing false rollbacks. Production-grade ACA engines (such as Netflix Kayenta or Argo Rollouts Analysis) use non-parametric statistical hypothesis testing, most notably the Mann-Whitney U Test (Wilcoxon rank-sum test).

The Mann-Whitney U test evaluates whether the probability distribution of metric samples collected from the Canary group is statistically stochastically greater than or equal to the Baseline group, without assuming a normal (Gaussian) distribution:

• Null Hypothesis (H0): The latency or error distributions of the Baseline and Canary populations are identical.
• Alternative Hypothesis (H1): The Canary population exhibits a systematic upward shift in latency or error rates.

If the calculated p-value drops below the significance threshold (e.g., α = 0.05 or α = 0.01), H0 is rejected: the regression is statistically proven to be non-random, and an automated rollback is triggered immediately.

Progressive Rollout with Argo Rollouts#

Argo Rollouts replaces the standard Kubernetes Deployment object with a Rollout Custom Resource Definition, managing canary steps and automated analysis templates natively:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: catalog-service-rollout
spec:
  replicas: 20
  strategy:
    canary:
      analysis:
        templates:
          - templateName: success-rate-check
        args:
          - name: service-name
            value: catalog-service-canary
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 20
        - pause: { duration: 15m }
        - setWeight: 50
        - pause: { duration: 20m }
        - setWeight: 100
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.995
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus-server.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="catalog-service-canary", status=~"2.."}[2m]))
            /
            sum(rate(http_requests_total{service="catalog-service-canary"}[2m]))

In this specification, if the canary's success rate drops below 99.5% for 3 consecutive 1-minute sampling evaluations, Argo Rollouts automatically aborts the rollout, scales the canary deployment to 0 replicas, and restores 100% traffic to stable instances without human intervention.

Database Schema Evolution: The Expand and Contract Pattern#

The most difficult challenge in zero-downtime deployments (both Rolling and Blue-Green) is the stateful layer: the relational database. During a rolling update or canary rollout, code v1 and code v2 execute concurrently and connect to the identical database schema.

If a developer executes a breaking database change—such as renaming column user_address to shipping_address or dropping a deprecated column—the existing v1 instances immediately crash with fatal SQL syntax errors (column does not exist).

The Expand / Contract (Parallel Run) Pattern#

To eliminate database-induced outages, schema changes must be decomposed into multiple non-breaking, backward-compatible evolutionary phases across independent deployment cycles.

+-----------------------------------------------------------------------------------------+
|                  Database Schema Evolution: Expand and Contract                         |
+-----------------------------------------------------------------------------------------+
| Phase 1: Expand                                                                         |
| - Add new column 'shipping_address' (NULLABLE).                                         |
| - Deploy code v1.1: Reads 'user_address', writes to BOTH 'user_address' & 'shipping'    |
+-----------------------------------------------------------------------------------------+
                                           |
                                           v
+-----------------------------------------------------------------------------------------+
| Phase 2: Asynchronous Backfill                                                          |
| - Run background migration script: UPDATE users SET shipping_address = user_address     |
|   WHERE shipping_address IS NULL; (Chunked to prevent lock contention).                 |
+-----------------------------------------------------------------------------------------+
                                           |
                                           v
+-----------------------------------------------------------------------------------------+
| Phase 3: Transition                                                                     |
| - Deploy code v1.2: Reads 'shipping_address', writes to BOTH.                           |
| - Verify all v1.0 and v1.1 instances are completely decommissioned.                     |
+-----------------------------------------------------------------------------------------+
                                           |
                                           v
+-----------------------------------------------------------------------------------------+
| Phase 4: Contract                                                                       |
| - Deploy code v2.0: Reads and writes ONLY to 'shipping_address'.                        |
| - Execute migration: ALTER TABLE users DROP COLUMN user_address;                        |
+-----------------------------------------------------------------------------------------+

Invariants of Zero-Downtime Database Migrations#

1. Never rename columns in-place: Always create the new column alongside the old one.
2. Never add a `NOT NULL` column without a default value: Existing running code that inserts rows without the new column will fail. Add columns as nullable first; backfill existing records; apply NOT NULL constraints in a subsequent phase.
3. Execute DDL operations concurrently: In PostgreSQL, adding an index locks the table against writes (ACCESS EXCLUSIVE) unless executed with CREATE INDEX CONCURRENTLY. In MySQL, use online DDL or tools like pt-online-schema-change / gh-ost.

Decoupling Deployment from Release: Feature Flags#

Deployment is an engineering action that places code onto production servers. Release is a business action that makes capabilities available to end users. Conflating these two concepts creates high-stakes, risky 'big-bang' releases.

Feature Flag Architecture#

Feature flags (or feature toggles) wrap new application code in dynamic conditional switches evaluated at runtime:

// Production TypeScript Feature Flag Evaluation
import { UnleashClient } from 'unleash-client';

export async function processOrderPayment(order: Order, user: User): Promise<PaymentResult> {
  const isStripeV3Enabled = await unleashClient.isEnabled('enable-stripe-v3-sdk', {
    userId: user.id,
    properties: {
      userTier: user.tier,
      country: user.country
    }
  });

  if (isStripeV3Enabled) {
    // New code path: Canary tested and isolated
    return paymentServiceV3.charge(order);
  } else {
    // Proven stable baseline path
    return legacyPaymentService.charge(order);
  }
}

Operational Advantages of Feature Flags#

• Instant Kill Switches: If an unexpected bug escapes into production, disabling the feature flag deactivates the broken code path in milliseconds without triggering a full Git commit, CI build, or deployment pipeline.
• Targeted Beta Testing: Expose new features exclusively to internal employees (user.email.endsWith('@devthesis.com')) or enterprise pilot customers before general availability.
• Elimination of Long-Lived Feature Branches: Developers merge incomplete features into main continuously behind a disabled toggle, preventing painful merge conflicts and facilitating continuous integration.

Traffic Shadowing (Dark Launches) & Asynchronous Mirroring#

While canary deployments expose a small percentage of real user requests to new code, certain mission-critical financial, compliance, or core database systems carry zero tolerance for customer-facing errors. In such domains, Traffic Shadowing (also known as Dark Launching or Request Mirroring) provides the ultimate zero-risk testing methodology.

Architectural Mechanics of Shadowing#

In a traffic shadowing architecture, the ingress proxy or service mesh (such as Envoy) duplicates 100% (or a configured fraction) of live production HTTP/gRPC requests:

1. Primary Route: The original request is dispatched to the stable production service (v1). The production service processes the request, executes database operations, and returns the HTTP response to the client. The client's response time and SLA are completely isolated.
2. Shadow Route: An asynchronous copy of the request payload, headers, and metadata is dispatched out-of-band to the new service version (v2).
3. Response Dropping: The shadow service processes the request, but the proxy automatically discards the shadow response. No response from v2 is ever returned to the end user.

                                      +------------------------+
                                      | Production Client      |
                                      +------------------------+
                                                  |
                                                  | 1. HTTP Request
                                                  v
                                      +------------------------+
                                      | Envoy / Ingress Proxy  |
                                      +------------------------+
                                        /                    \
                   2. Live Traffic     /                      \  3. Async Cloned
                      (Blocks Client) /                        \    Request
                                     v                          v
                      +---------------------+    +---------------------+
                      | Stable Service (v1) |    | Shadow Service (v2) |
                      | (Live Production)   |    | (Dark Launch)       |
                      +---------------------+    +---------------------+
                                     |                          |
                     4. Response     |                          | 5. Response
                        Returned     v                          v    Silently
                      +---------------------+                   x    Discarded
                      | Client Receives 200 |
                      +---------------------+

Side-Effect Isolation & Mocking for Mutating Requests#

A critical challenge with traffic shadowing arises when handling state-mutating requests (e.g., POST /payments/charge or PUT /orders/cancel). If both v1 and v2 write to the production database or trigger third-party credit card APIs, duplicate charges or severe data corruption will occur.

To safely shadow mutating traffic:

• Read-Only Shadowing: Configure mirroring rules exclusively for idempotent HTTP methods (GET, HEAD, OPTIONS).
• Dry-Run Mode & Synthetic Datastores: Instrument the shadow service with an environment flag (SHADOW_MODE=true). When enabled, the application routes all database write queries to an ephemeral read-replica or sandboxed datastore, and replaces external payment APIs with simulated mock stubs.
• Diffing Proxies (e.g., Twitter Diffy): A specialized proxy compares the HTTP status code, headers, and JSON response bodies emitted by v1 and v2, logging structural differences or semantic regressions to an analytics engine without impacting live users.

Session Persistence, Sticky Canary Routing, and Consistent Hashing#

Standard percentage-based canary traffic routing operates statelessly at the per-request level: each HTTP request has an independent 5% or 10% probability of landing on the canary instance. While suitable for purely stateless APIs, this approach can degrade user experience in stateful, single-page, or conversational applications.

The Problem of State Flapping#

If an active user submits multiple requests during a single browsing session (e.g., browsing a catalog, adding items to a cart, proceeding to checkout):

• Request 1 lands on v1 (stable).
• Request 2 lands on v2 (canary).
• Request 3 lands on v1 (stable).

If v2 introduces new client-side JavaScript assets or alters internal state models, the user experiences inconsistent UI behavior, cache invalidation loops, or missing session data.

Consistent Hashing & Deterministic Hashing Buckets#

To provide consistent user experiences during canary rollouts, ingress proxies implement sticky canary routing based on stable client identifiers:

1. User Identifier Hashing: The proxy extracts a stable request attribute—such as the authenticated User-ID header, an encrypted session cookie (session_id), or the client IP address.
2. Hash Normalization: The attribute is passed through a fast, uniform cryptographic or non-cryptographic hash function (such as MurmurHash3 or MD5):

```

H = MurmurHash3(user_id) mod 100

```

1. Bucket Assignment: If the calculated integer H falls within the canary percentage threshold [0, P_canary - 1], the request is forwarded to the canary. Otherwise, it is routed to the stable baseline.

+---------------------------------------------------------------------------------------+
|                         Consistent Canary Hash Partitioning                           |
+---------------------------------------------------------------------------------------+
|  User "Alice" (ID: 8492) ---> MurmurHash3("8492") % 100 = 4   ---> Canary (<= 10%)    |
|  User "Bob"   (ID: 1033) ---> MurmurHash3("1033") % 100 = 87  ---> Stable (> 10%)     |
|  User "Carol" (ID: 5521) ---> MurmurHash3("5521") % 100 = 9   ---> Canary (<= 10%)    |
+---------------------------------------------------------------------------------------+

Because hash values are deterministic, User Alice will consistently route to the canary across all subsequent HTTP requests throughout her entire session. When the canary weight is subsequently increased from 10% to 25%, all existing canary users remain on the canary, while an additional 15% of the user population is deterministically added without disrupting existing routing decisions.

Common Architectural Pitfalls in Progressive Delivery#

1. Inadequate Canary Sample Sizes & Low-Traffic Services#

If a service receives only 10 requests per minute and a canary is allocated 5% of traffic, the canary processes only 0.5 requests per minute (1 request every 2 minutes). Over a 10-minute evaluation window, the canary receives 5 requests. If 1 request fails due to an unrelated network glitch, the measured error rate is 20%, triggering a false rollback.

• Mitigation: Canary analysis must enforce minimum sample count thresholds (count(requests) >= 500). For low-traffic services, synthetic traffic generation or header-based dark routing must be used instead of percentage-based splitting.

2. Stateful Session Affinity Leaks#

When applications rely on server-side in-memory user sessions without distributed session stores (e.g., Redis), switching an active session to a canary instance can result in user logout or missing shopping cart items.

• Mitigation: Implement stateless architectures with externalized session storage (JWT or distributed cache); configure consistent hashing or cookie-based sticky sessions for canary cohorts during testing.

3. Premature Teardown of the Blue Stack#

Tearing down or terminating the Blue environment immediately upon routing 100% of traffic to Green removes the safety net. If a latent bug (such as a memory leak or slow connection exhaustion) manifests 30 minutes later, instant rollback is impossible.

• Mitigation: Enforce a mandatory bake period (e.g., 2 to 4 hours) during which the Blue environment remains fully warm and idle. The old stack is decommissioned only after the bake period passes with zero alerts.

Checkpoint Practice Problems#

Problem 1 (Kubernetes Rolling Update Surge & Availability Sizing): A Kubernetes deployment runs N = 20 replicas of an e-commerce API. To maintain service level agreements (SLAs), the system must never drop below 16 healthy pods during any phase of an update, and the cluster cannot accommodate more than 24 total pods due to node memory limits.

1. Determine the appropriate integer values or percentages for maxUnavailable and maxSurge.
2. Compute the minimum and maximum number of running pods at any instant during the rolling update.

Solution:

1. Calculate maxUnavailable:

* The minimum required healthy pods is 16 out of 20.

* Maximum allowed missing pods = 20 - 16 = 4.

* maxUnavailable = 4 (or 20% of 20).

1. Calculate maxSurge:

* The maximum permitted total pods is 24.

* Maximum allowed extra pods above desired = 24 - 20 = 4.

* maxSurge = 4 (or 20% of 20).

1. Compute Pod boundaries:

* Minimum healthy pods during rollout = Replicas - maxUnavailable = 20 - 4 = 16 pods.

* Maximum pods running simultaneously = Replicas + maxSurge = 20 + 4 = 24 pods.

Configuration: maxUnavailable: 4 (or 20%) and maxSurge: 4 (or 20%).

Problem 2 (Canary Error Rate Hypothesis Testing & Z-Score): During a canary analysis, the baseline service processes N_base = 10,000 requests with 50 errors (p_base = 50 / 10,000 = 0.0050 or 0.50%). The canary service processes N_canary = 2,500 requests with 25 errors (p_canary = 25 / 2,500 = 0.0100 or 1.00%).

The standard error (SE) for the difference between two independent proportions is given by:

p_pool = (errors_base + errors_canary) / (N_base + N_canary)
SE = √[ p_pool · (1 - p_pool) · ( (1 / N_base) + (1 / N_canary) ) ]

The test statistic Z is:

Z = (p_canary - p_base) / SE

Using a one-tailed critical value Z_crit = 2.33 (corresponding to α = 0.01 or 99% confidence):

1. Compute p_pool and SE.
2. Compute the test statistic Z.
3. State whether the canary regression is statistically significant at α = 0.01, and whether an automated rollback should trigger.

Solution:

1. Compute pooled proportion p_pool:

```

p_pool = (50 + 25) / (10,000 + 2,500) = 75 / 12,500 = 0.0060

1 - p_pool = 1 - 0.0060 = 0.9940

```

1. Compute sample size factor:

```

(1 / N_base) + (1 / N_canary) = (1 / 10,000) + (1 / 2,500) = 0.0001 + 0.0004 = 0.0005

```

1. Compute SE:

```

Variance = 0.0060 · 0.9940 · 0.0005 = 0.005964 · 0.0005 = 0.000002982

SE = √(0.000002982) ≈ 0.0017268

```

1. Compute test statistic Z:

```

p_canary - p_base = 0.0100 - 0.0050 = 0.0050

Z = 0.0050 / 0.0017268 ≈ 2.895

```

1. Evaluate against Z_crit = 2.33:

Since Z = 2.895 > 2.33, the observed doubling of error rate in the canary cohort is statistically significant with greater than 99% confidence (p < 0.01). The pipeline must trigger an immediate automated rollback.

Problem 3 (Infrastructure Cost Comparison: Blue-Green vs Canary): A cloud microservice requires 50 compute nodes running 24/7. Each node costs 0.10 per hour. The company executes 4 deployments per day.

• Under Strategy A (Blue-Green): A full duplicate stack (50 nodes) is provisioned for every deployment. It takes 30 minutes to warm up and test, followed by a mandatory 2-hour bake period before the old stack is terminated (total extra uptime = 2.5 hours per deployment).
• Under Strategy B (Progressive Canary): An additional 5 nodes (10% capacity) are provisioned for canary analysis. The canary evaluation runs for a total of 1.5 hours per deployment before full promotion and node termination.

Calculate:

1. The monthly additional deployment infrastructure cost for Strategy A (assume 30 days/month).
2. The monthly additional deployment infrastructure cost for Strategy B.
3. The percentage cost savings achieved by Strategy B over Strategy A.

Solution:

1. Strategy A (Blue-Green):

* Extra node-hours per deployment: 50 nodes · 2.5 hours = 125 node-hours.

* Daily extra node-hours: 125 · 4 deployments = 500 node-hours/day.

* Monthly extra cost: 500 node-hours/day · 30 days · 0.10/node-hour = 1,500.00 per month.

1. Strategy B (Canary):

* Extra node-hours per deployment: 5 nodes · 1.5 hours = 7.5 node-hours.

* Daily extra node-hours: 7.5 · 4 deployments = 30 node-hours/day.

* Monthly extra cost: 30 node-hours/day · 30 days · 0.10/node-hour = 90.00 per month.

1. Cost Savings:

* Absolute savings: 1,500 - 90 = 1,410.00 per month.

* Percentage savings: (1,410 / 1,500) · 100% = 94.0%.

Strategy B reduces deployment infrastructure overhead by 94.0%.

Problem 4 (Canary Stepped Rollout Schedule & Blast Radius): A progressive delivery pipeline uses a 4-step canary rollout schedule:

• Step 1: 5% traffic for 15 minutes.
• Step 2: 15% traffic for 20 minutes.
• Step 3: 40% traffic for 30 minutes.
• Step 4: 100% traffic (Full promotion).

If a catastrophic defect (which crashes 100% of affected requests) is present in the new release but takes 25 minutes of live traffic to trigger its first manifestation:

1. At which step and minute of the deployment will the bug first manifest?
2. What percentage of total user traffic is exposed to the defect at the moment it triggers?
3. What would the exposed blast radius have been under a direct Recreate deployment?

Solution:

1. Timeline analysis of rollout:

* Minutes 0 to 15: Step 1 (5% traffic). Cumulative time = 15 minutes.

* Minutes 15 to 35: Step 2 (15% traffic). Cumulative time = 35 minutes.

The bug triggers at Minute 25. Minute 25 falls squarely inside Step 2* (which runs from minute 15 to 35).

1. Blast radius at triggering:

During Step 2, the traffic allocation to the canary is 15%*.

* Therefore, exactly 15% of total user traffic experiences the defect before automated analysis halts the pipeline and rolls back.

1. Comparison with Recreate:

Under a Recreate or Big-Bang deployment, 100% of traffic would have been routed to the broken version immediately, causing a 100% service outage* for all active users.

The canary strategy successfully restricted the blast radius to 15%.