MLOps - Model Tracking & Versioning (MLflow)

In academic and experimental environments, machine learning is often treated as a static modeling exercise: training a model once in a Jupyter notebook on a fixed CSV file, tuning hyperparameters, and evaluating test accuracy. In enterprise production environments, however, machine learning systems are continuous, dynamic software applications. Once deployed, models interact with volatile real-world data distributions, degrade over time, and require rigorous reproducibility, governance, auditability, and automated redeployment.

This operational reality led to the emergence of MLOps (Machine Learning Operations): the intersection of machine learning, software engineering, and DevOps. Drawing from Sculley et al.'s foundational insight on the "Hidden Technical Debt in Machine Learning Systems," actual ML model code constitutes only a tiny fraction of a production system. The vast majority consists of data verification, pipeline orchestration, model tracking, feature stores, metadata governance, and serving infrastructure. This guide delivers a comprehensive, university-grade examination of MLOps architectures, focusing deeply on MLflow tracking, DVC dataset versioning, serving topologies, and statistical drift detection.

Machine Learning Technical Debt & The Reproducibility Crisis#

Unlike traditional software engineering where reproducibility depends strictly on deterministic source code (Git commit SHA), machine learning reproducibility requires capturing the confluence of four independent, evolving components:

Machine Learning Reproducibility Vector:
R = ⟨ Code Version, Data Version, Compute Environment, Hyperparameters & Random Seeds ⟩

+---------------------------------------------------------------------------------------+
|                       The Four Pillars of ML Reproducibility                          |
+---------------------------------------------------------------------------------------+
|  1. Exact Code SHA           : Git commit hash of model definition & training scripts  |
|  2. Exact Data Snapshot      : Immutable dataset hash (DVC / Delta Lake time-travel)  |
|  3. Environment Dependencies : Exact pip/conda lockfile, CUDA runtime, Docker image   |
|  4. Parameters & Seed State  : Initial weights, random seed (numpy, torch), learning α|
+---------------------------------------------------------------------------------------+

If a financial fraud model deployed six months ago begins exhibiting anomalous rejection patterns, regulatory auditors require the organization to reproduce the exact model artifact, trace the training data used, inspect the evaluation metrics, and explain its mathematical behavior. Without rigorous MLOps tooling, reconstructing an experimental model trained months earlier is virtually impossible.

MLflow: Architecture & The Four Core Modules#

Developed by Databricks and open-sourced under the Linux Foundation, MLflow is the dominant open-source platform designed to manage the end-to-end machine learning lifecycle.

+---------------------------------------------------------------------------------------+
|                                  MLflow System Architecture                           |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   Data Scientist / Training Pipeline (Python, PyTorch, Scikit-learn, XGBoost)         |
|                                     |                                                 |
|                                     v                                                 |
|   +---------------------------------------------------------------------------------+ |
|   |                             MLflow Tracking Server                              | |
|   +---------------------------------------------------------------------------------+ |
|            |                                                         |                |
|            v                                                         v                |
|   [ Backend Entity Store ]                                  [ Artifact Store ]        |
|   (PostgreSQL / MySQL)                                      (AWS S3 / GCS / Azure)    |
|   - Runs, Experiments, Users                                - Serialized Model Files  |
|   - Hyperparameters (e.g., max_depth)                       - Metric plots, charts    |
|   - Metric Histories (loss per epoch)                       - TensorBoard logs        |
|            |                                                         |                |
|            +----------------------------+----------------------------+                |
|                                         |                                             |
|                                         v                                             |
|   +---------------------------------------------------------------------------------+ |
|   |                             MLflow Model Registry                               | |
|   | - Versioning (v1, v2, v3)                                                       | |
|   | - Stage Transitions (Development -> Staging -> Production -> Archived)           | |
|   | - Governance, Webhooks, Approval Gates                                          | |
|   +---------------------------------------------------------------------------------+ |
|                                         |                                             |
|                                         v                                             |
|   Downstream Model Serving: Docker, Kubernetes, Triton, SageMaker, Spark UDFs         |
|                                                                                       |
+---------------------------------------------------------------------------------------+

1. MLflow Tracking#

MLflow Tracking organizes training runs into logical groupings called Experiments. Within each run, the client SDK logs:

• Parameters: Key-value string pairs recording configuration hyperparameters (e.g., learning_rate: 0.001, n_estimators: 500). Parameters are immutable once written.
• Metrics: Key-value numeric entries (float64) recorded with a timestamp and step counter (e.g., logging train_loss and val_loss across 100 training epochs), enabling visual learning curves in the UI.
• Artifacts: Heavy binary files written directly to cloud object storage (AWS S3, GCS): serialized model weights (model.pkl, weights.pt), confusion matrix PNGs, feature importance charts, and sample predictions.
• Metadata: Git commit SHA, author, start time, end time, and host environment.

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score

# Set remote tracking URI
mlflow.set_tracking_uri("http://mlflow-server.internal:5000")
mlflow.set_experiment("credit_default_prediction")

with mlflow.start_run(run_name="rf_hyperparameter_tuning_v1"):
    # 1. Log Hyperparameters
    n_estimators = 200
    max_depth = 12
    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_param("max_depth", max_depth)

    # 2. Train Model
    clf = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth, random_state=42)
    clf.fit(X_train, y_train)

    # 3. Log Performance Metrics
    predictions = clf.predict(X_val)
    acc = accuracy_score(y_val, predictions)
    f1 = f1_score(y_val, predictions)
    mlflow.log_metric("accuracy", acc)
    mlflow.log_metric("f1_score", f1)

    # 4. Log Model Artifact with Signature
    from mlflow.models.signature import infer_signature
    signature = infer_signature(X_val, predictions)

    mlflow.sklearn.log_model(
        sk_model=clf,
        artifact_path="model",
        registered_model_name="CreditDefaultClassifier",
        signature=signature
    )

2. MLflow Models & Model Flavors#

A major bottleneck in classical ML delivery was the deployment friction between data science frameworks (PyTorch, TensorFlow, Scikit-learn) and production serving runtimes (Java microservices, C++ inference servers, REST APIs).

MLflow Models defines a standardized, self-contained directory format that encapsulates the model along with its runtime flavor:

• `MLmodel` file: A declarative YAML file describing the model flavors, input schema signature, and dependencies.
• Model Flavors: Enables the model to be consumed across different environments:

* python_function (pyfunc): The generic, universal wrapper format. Any model exposing a predict(df) method can be served via a standardized Docker REST API container without custom wrapper code.

* Native Framework Flavors (sklearn, pytorch, onnx): Enables framework-specific performance optimizations (such as tensor batching or GPU acceleration).

3. MLflow Model Registry#

The Model Registry provides a centralized catalog for governing enterprise model promotion:

• Model Lineage: Maps a production model version directly back to the exact experiment run, training parameters, and Git commit that generated it.
• Stage Transitions: Manages the model lifecycle across formal operational states: None → Staging → Production → Archived.
• CI/CD Integration & Webhooks: When a model is transitioned to Staging, MLflow can fire an automated webhook to a GitHub Actions or ArgoCD pipeline, executing automated integration tests, latency benchmarks, and fairness audits before promoting to Production.

Data & Pipeline Versioning with DVC (Data Version Control)#

While Git is the undisputed standard for source code version control, it is fundamentally incapable of managing multi-gigabyte training datasets, video corpora, or heavy neural network checkpoint files:

• Committing a 20 GB dataset to Git bloats the .git directory permanently. Every clone, branch, and checkout operation becomes painfully slow.
• Git LFS (Large File Storage) handles large files but lacks pipeline execution semantics and remote caching capabilities.

DVC Mechanics: Content-Addressable Pointers#

DVC (Data Version Control) solves this by decoupling metadata from physical data:

+---------------------------------------------------------------------------------------+
|                               DVC Content-Addressable Storage                         |
+---------------------------------------------------------------------------------------+
|  Local Workspace                Git Repository                   Remote Object Storage |
|  +--------------------+         +-----------------------+        +--------------------+|
|  | data/raw_data.csv  | ======> | data/raw_data.csv.dvc | -----> | s3://my-dvc-bucket/ |
|  | (15 Gigabytes)     | (Hash)  | - md5: 8a4f91...      |        | files/md5/8a/4f91..|
|  +--------------------+         | - size: 15482910      |        +--------------------+|
|                                 +-----------------------+                              |
|                                 (Tiny text file in Git!)                               |
+---------------------------------------------------------------------------------------+

1. When a developer runs dvc add data/raw_data.csv, DVC computes the MD5 checksum hash of the 15 GB file (e.g., 8a4f91d2e...).
2. DVC creates a tiny, 100-byte text pointer file: data/raw_data.csv.dvc containing the hash, and automatically appends data/raw_data.csv to .gitignore.
3. The developer commits data/raw_data.csv.dvc directly to Git.
4. The heavy 15 GB file is pushed to remote cloud storage via dvc push (s3://dvc-bucket/8a/4f91d2e...).
5. Any team member checking out the Git commit simply runs dvc pull to download the exact binary dataset version associated with that commit.

DVC Pipelines (`dvc.yaml`)#

DVC enables reproducible data science pipelines by defining a directed acyclic graph in dvc.yaml:

stages:
  extract:
    cmd: python src/extract.py
    deps:
      - src/extract.py
    outs:
      - data/raw_transactions.parquet
  featurize:
    cmd: python src/featurize.py
    deps:
      - src/featurize.py
      - data/raw_transactions.parquet
    outs:
      - data/features.parquet
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/features.parquet
    params:
      - train.learning_rate
      - train.n_estimators
    outs:
      - models/model.pkl
    metrics:
      - metrics/eval.json:
          cache: false

When dvc repro is executed, DVC inspects the cryptographic hashes of all dependencies, scripts, and parameters. If an upstream file has not changed, DVC skips that stage entirely and pulls the cached output from storage, enabling blazing-fast incremental pipeline execution.

Production Model Serving Topologies#

Deploying machine learning models to production requires selecting an appropriate serving topology based on latency, throughput, and statefulness requirements.

+----------------------------------------------------------------------------------------------------+
|                                    Model Serving Topologies Matrix                                 |
+-------------------+--------------------+------------------------+----------------------------------+
| Serving Topology  | Latency SLA        | Throughput             | Primary Technologies             |
+-------------------+--------------------+------------------------+----------------------------------+
| Real-Time (REST)  | Low (10 - 100 ms)  | Moderate (Req/sec)     | FastAPI, Triton, BentoML, vLLM   |
| Real-Time (gRPC)  | Ultra-Low (< 10 ms)| High (Binary Protobuf) | TorchServe, Triton Inference     |
| Batch Scoring     | High (Hours/Days)  | Massive (Billions/hr)  | Spark UDFs, Ray, DuckDB          |
| Embedded Edge     | Micro (< 1 ms)     | Local device only      | ONNX Runtime, TensorRT, CoreML   |
| Streaming Event   | Near Real-Time     | Continuous Streams     | Kafka Streams, Apache Flink      |
+-------------------+--------------------+------------------------+----------------------------------+

Production Monitoring: Drift Detection & Decay Mitigation#

Once a model is deployed into production, its predictive accuracy inevitably decays over time due to shifts in real-world human behavior, macroeconomic conditions, or software dependencies.

+---------------------------------------------------------------------------------------+
|                               The Three Types of ML Drift                             |
+---------------------------------------------------------------------------------------+
|  1. Covariate Shift (Data Drift):                                                     |
|     - Input distribution P(X) changes: P_train(X) ≠ P_prod(X)                         |
|     - True mapping P(Y | X) remains invariant.                                        |
|     - Example: A marketing campaign attracts an older demographic.                    |
|                                                                                       |
|  2. Concept Drift:                                                                    |
|     - True relationship P(Y | X) changes: P_train(Y | X) ≠ P_prod(Y | X)              |
|     - Input distribution P(X) may remain unchanged.                                   |
|     - Example: Post-pandemic consumer spending patterns shift fundamentally.          |
|                                                                                       |
|  3. Prior Probability Shift:                                                          |
|     - Target class distribution P(Y) changes: P_train(Y) ≠ P_prod(Y)                  |
|     - Example: Sudden macroeconomic recession increases base default rate from 2% to 8%|
+---------------------------------------------------------------------------------------+

1. Population Stability Index (PSI)#

The Population Stability Index (PSI) is the industry-standard metric for quantifying distribution drift in continuous or categorical features between a baseline training population (Reference) and a live production monitoring window (Actual).

Data is partitioned into B distinct quantile bins (typically B = 10 deciles established on the reference dataset):

PSI = ∑ [b = 1 to B] [ (Actual%_b - Reference%_b) · ln( Actual%_b / Reference%_b ) ]

Notice the mathematical structure:

• (Actual%_b - Reference%_b) measures the linear shift in frequency.
• ln(Actual%_b / Reference%_b) is positive when Actual > Reference, and negative when Actual < Reference.
• The product is guaranteed to be non-negative across every individual bin: each bin contributes ≥ 0 to total PSI.

Regulatory & Operational Action Thresholds#

• PSI < 0.10: No Significant Distribution Shift. The feature distribution is stable; no action required.
• 0.10 ≤ PSI < 0.20: Moderate Distribution Drift. Indicates noticeable divergence; trigger enhanced monitoring and inspect pipeline data quality.
• PSI ≥ 0.20: Severe Distribution Shift. Significant population drift has occurred. Machine learning pipelines should automatically trigger an alert, quarantine the model, or execute an automated retraining DAG on recent data.

2. The Kolmogorov-Smirnov (KS) Two-Sample Test#

For continuous univariate features, the two-sample Kolmogorov-Smirnov test measures the maximum vertical distance between the empirical cumulative distribution functions (eCDF) of the reference dataset F_ref(x) and production dataset F_prod(x):

D_KS = sup_x | F_ref(x) - F_prod(x) |

If the test statistic D_KS exceeds the critical threshold at significance level α (e.g., α = 0.05), we reject the null hypothesis: the production feature distribution has statistically significantly shifted from the training baseline.

Model Lineage & Data Provenance (OpenLineage & Marquez)#

In enterprise data platforms, machine learning models do not exist in isolation; they represent the downstream termination point of vast, multi-stage data supply chains comprising dozens of upstream Airflow DAGs, dbt transformations, and Spark streaming jobs.

The Data Lineage Crisis#

If an upstream database administrator alters the calculation logic for a column net_revenue inside an operational MySQL database, the downstream machine learning model may silently ingest the altered semantic definition without raising any schema syntax errors. The model's predictions will quietly degrade in production.

• OpenLineage: An open industry standard for metadata and lineage collection across modern data systems. It defines a declarative JSON specification capturing:

Datasets*: Inputs and outputs of every stage, identified by persistent uniform resource identifiers (URIs).

Jobs*: The transformation code and version executing on the data.

Runs*: Specific executions with start/end timestamps and operational facets (e.g., row count deltas, null counts).

• Marquez: A reference metadata repository that ingests OpenLineage events from Airflow, Spark, and dbt. Data scientists can inspect a visual DAG graph tracing a model artifact in MLflow directly backward through all upstream ETL transformations all the way to the source transactional database tables.

Automated Retraining Pipelines & Shadow Evaluation#

When statistical drift detectors (such as PSI or KS tests) indicate severe distribution divergence, the MLOps platform initiates an automated remediation workflow:

+---------------------------------------------------------------------------------------+
|                        Automated MLOps Retraining & Promotion                         |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   1. Production Monitor Detects PSI >= 0.20                                           |
|         |                                                                             |
|         v                                                                             |
|   2. Trigger Retraining Airflow DAG                                                   |
|      - Extracts recent 90-day data window from Feature Store                          |
|      - Tunes hyperparameters via Bayesian Optimization (Optuna)                       |
|      - Logs candidate model (v_next) to MLflow Model Registry                         |
|         |                                                                             |
|         v                                                                             |
|   3. Automated Validation Gate                                                        |
|      - Compares v_next performance against v_current (Production Baseline)            |
|      - Enforces Champion/Challenger Gate: F1(v_next) >= F1(v_current) + 0.01          |
|      - Enforces Latency Gate: P99 Inference Latency <= 45 ms                          |
|         |                                                                             |
|         +---> Pass ===> Deploy to Shadow / Canary Route                               |
|         |                                                                             |
|         +---> Fail ===> Alert On-Call Engineer, Halt Promotion                        |
|                                                                                       |
+---------------------------------------------------------------------------------------+

Champion vs. Challenger Deployment#

The newly trained candidate model (Challenger) is not immediately promoted to 100% of live traffic.

• Shadow Mode: Ingress proxies (e.g., Envoy or Istio) fork live production requests asynchronously to the Challenger model. The Challenger computes predictions, logs its outputs and latency to Datadog/Prometheus, but its predictions are discarded. Live users continue receiving responses from the proven Champion.
• Canary Promotion: Once shadow evaluation verifies that the Challenger does not crash, leak memory, or violate latency SLAs, traffic is incrementally shifted (5% -> 25% -> 100%) as telemetry confirms zero regression in real-world business KPIs.

Explainability & Fairness Auditing in Production (SHAP & LIME)#

Modern regulatory frameworks (e.g., EU AI Act, GDPR Article 22, US Fair Lending laws) prohibit automated black-box decision systems. Organizations must be able to generate human-interpretable explanations for individual model predictions.

SHAP (SHapley Additive exPlanations)#

Grounded in cooperative game theory (Lloyd Shapley, 1953), SHAP computes the marginal contribution of each feature to the difference between the model's actual prediction and the base expected prediction:

f(x) = E[f(x)] + ∑ [j = 1 to d] φ_j(x)

where φ_j(x) is the Shapley value for feature j, uniquely satisfying four foundational game-theoretic axioms: Efficiency, Symmetry, Dummy Player, and Additivity.

• TreeSHAP: High-performance algorithmic optimization for tree ensembles (XGBoost, LightGBM, CatBoost) that evaluates exact Shapley values in O(T · L · D^2) time rather than exponential O(2^d) time.
• Production Integration: High-throughput ML serving engines log top-k positive and negative SHAP feature contributions alongside each prediction, writing them to analytical data lake tables for real-time customer dispute resolution and algorithmic fairness auditing.

Checkpoint Practice Problems#

Problem 1 (Population Stability Index Calculation): A credit risk model monitors a customer's credit_utilization_ratio feature. The training baseline (Reference) and a 30-day production window (Actual) are binned into three representative risk tiers:

• Tier 1 (Low: 0% to 30%): Reference = 60%, Actual = 40%
• Tier 2 (Medium: 30% to 70%): Reference = 30%, Actual = 30%
• Tier 3 (High: 70% to 100%): Reference = 10%, Actual = 30%

1. Express the proportions as decimals for each bin.
2. Compute the individual PSI contribution for Tier 1, Tier 2, and Tier 3.
3. Compute the total Population Stability Index (PSI).
4. Based on standard MLOps operational thresholds, determine whether this feature has experienced severe drift requiring automated retraining.

Solution:

1. Bin proportions:

* Tier 1: Ref = 0.60, Act = 0.40

* Tier 2: Ref = 0.30, Act = 0.30

* Tier 3: Ref = 0.10, Act = 0.30

1. Compute individual contributions:

* Tier 1:

```

Act - Ref = 0.40 - 0.60 = -0.20

ln(Act / Ref) = ln(0.40 / 0.60) = ln(2/3) ≈ -0.405465

PSI_1 = (-0.20) · (-0.405465) = +0.081093

```

* Tier 2:

```

Act - Ref = 0.30 - 0.30 = 0.0

ln(Act / Ref) = ln(0.30 / 0.30) = ln(1.0) = 0.0

PSI_2 = (0.0) · (0.0) = 0.000000

```

* Tier 3:

```

Act - Ref = 0.30 - 0.10 = +0.20

ln(Act / Ref) = ln(0.30 / 0.10) = ln(3.0) ≈ +1.098612

PSI_3 = (+0.20) · (+1.098612) = +0.219722

```

1. Compute Total PSI:

```

Total PSI = PSI_1 + PSI_2 + PSI_3

= 0.081093 + 0.000000 + 0.219722 = 0.300815 ≈ 0.301

```

1. Operational interpretation:

Since PSI = 0.301 ≥ 0.20, this feature exhibits Severe Distribution Shift. The customer base has significantly migrated into high credit utilization. The MLOps pipeline must immediately trigger an alert and initiate automated model retraining.

Problem 2 (Kolmogorov-Smirnov Test Statistic Calculation): A continuous model feature is evaluated across two sample datasets (Reference and Production), each containing N = 5 sorted observations:

• Reference Dataset: X_ref = { 10, 20, 30, 40, 50 }
• Production Dataset: X_prod = { 25, 35, 45, 55, 65 }

1. Construct the combined sorted evaluation points.
2. Compute the empirical cumulative distribution functions F_ref(x) and F_prod(x) at each unique point (where F(x) = (count of samples ≤ x) / N).
3. Compute the absolute difference |F_ref(x) - F_prod(x)| at each point.
4. Determine the Kolmogorov-Smirnov test statistic D_KS.

Solution:

1. Combined sorted evaluation points:

x ∈ { 10, 20, 25, 30, 35, 40, 45, 50, 55, 65 }

1. Evaluate eCDFs:

* x = 10: F_ref = 1/5 = 0.20, F_prod = 0/5 = 0.00 -> Diff = 0.20

* x = 20: F_ref = 2/5 = 0.40, F_prod = 0/5 = 0.00 -> Diff = 0.40

* x = 25: F_ref = 2/5 = 0.40, F_prod = 1/5 = 0.20 -> Diff = 0.20

* x = 30: F_ref = 3/5 = 0.60, F_prod = 1/5 = 0.20 -> Diff = 0.40

* x = 35: F_ref = 3/5 = 0.60, F_prod = 2/5 = 0.40 -> Diff = 0.20

* x = 40: F_ref = 4/5 = 0.80, F_prod = 2/5 = 0.40 -> Diff = 0.40

* x = 45: F_ref = 4/5 = 0.80, F_prod = 3/5 = 0.60 -> Diff = 0.20

* x = 50: F_ref = 5/5 = 1.00, F_prod = 3/5 = 0.60 -> Diff = 0.40

* x = 55: F_ref = 5/5 = 1.00, F_prod = 4/5 = 0.80 -> Diff = 0.20

* x = 65: F_ref = 5/5 = 1.00, F_prod = 5/5 = 1.00 -> Diff = 0.00

1. Maximum difference:

The maximum absolute difference occurs at x ∈ {20, 30, 40, 50}:

```

D_KS = max | F_ref(x) - F_prod(x) | = 0.40

```

The KS test statistic is D_KS = 0.40.

Problem 3 (Real-Time Inference Serving Concurrency & Latency Sizing): An enterprise fraud detection microservice is deployed on a Kubernetes cluster.

• Peak traffic demand: R = 2,500 requests per second.
• Target service level objective: P99 response time must be strictly under 50 milliseconds.
• Single-threaded model inference benchmark: A single CPU worker process takes on average T_inf = 12 milliseconds of CPU compute time per inference request.
• Operational headroom buffer: Nodes must not exceed 75% average CPU utilization during peak traffic to prevent queuing delays.

Calculate:

1. The minimum theoretical number of concurrent worker processes needed to process 2,500 requests/sec with zero queuing delay.
2. The operational number of worker processes required when factoring in the 75% utilization headroom constraint.
3. If each worker pod can host 4 worker processes, determine the minimum number of Kubernetes pods that must be provisioned.

Solution:

1. Theoretical minimum worker concurrency (Little's Law):

```

Concurrency L = Arrival Rate λ · Service Time W

λ = 2,500 req/sec

W = 12 ms = 0.012 seconds

L_min = 2,500 · 0.012 = 30 concurrent processes

```

1. Operational worker requirement (factoring 75% utilization cap):

```

Workers_operational = L_min / Target_utilization

Workers_operational = 30 / 0.75 = 40 worker processes

```

1. Pod provisioning:

```

Pods = 40 workers / 4 workers_per_pod = 10 pods

```

The engineering team must deploy at least 10 worker pods (comprising 40 worker processes) to comfortably sustain 2,500 req/sec within the SLA.

Problem 4 (DVC Content-Addressable Remote Storage Deduplication): A data science team runs 10 consecutive model training experiments across two weeks.

• The base training dataset has an uncompressed size of 8.0 GB.
• In Experiment 1, the full 8.0 GB dataset is committed.
• In Experiments 2 through 10, each experiment appends an incremental daily delta of 100 MB (0.1 GB) of new records, leaving the historical 8.0 GB completely unmodified.
• S3 storage costs 0.023 per GB per month.

1. Under a naive legacy backup strategy where the entire dataset is copied into a separate timestamped S3 directory for each of the 10 experiments, compute the total S3 storage consumed and monthly cost.
2. Under DVC content-addressable chunked storage where identical immutable data chunks are stored once and deduplicated via hash pointers, compute the total storage consumed and monthly cost.
3. Calculate the percentage storage savings achieved by DVC over the naive approach.

Solution:

1. Naive Strategy (Full copies per experiment):

* Exp 1: 8.0 GB

* Exp 2: 8.1 GB

* Exp 3: 8.2 GB

* ...

* Exp 10: 8.9 GB

* Total storage = 8.0 + 8.1 + 8.2 + 8.3 + 8.4 + 8.5 + 8.6 + 8.7 + 8.8 + 8.9 = 84.5 GB.

* Monthly cost = 84.5 GB · 0.023/GB = 1.9435 per month.

1. DVC Content-Addressable Strategy (Deduplication):

* The base 8.0 GB data blocks are stored exactly once.

* Only the 9 daily deltas are stored additionally:

9 deltas · 0.1 GB = 0.9 GB.

* Total storage = 8.0 GB + 0.9 GB = 8.9 GB.

* Monthly cost = 8.9 GB · 0.023/GB = 0.2047 per month.

1. Savings:

```

Storage Savings = [ (84.5 GB - 8.9 GB) / 84.5 GB ] · 100%

= [ 75.6 / 84.5 ] · 100% ≈ 89.47%

```

DVC deduplication eliminates 89.5% of storage overhead, reducing footprint from 84.5 GB down to 8.9 GB.