📊 Agile MLOps Lifecycle & Heuristic Baselines¶

In modern MLOps development, starting with a complex machine learning model (like Random Forests or Deep Neural Networks) in "Sprint 1" is often a anti-pattern.

Instead, the Agile MLOps Lifecycle recommends starting with a Sprint 0 heuristic baseline.

Heuristic baseline design patterns¶

A heuristic baseline is a hard-coded or simple statistic-based rule (e.g., pricing a house at a flat $150 per sqft + $50k base). This serves three primary purposes:

1. End-to-End Pipeline Validation: Ensuring that database ingestion, serialization, API servers, and monitoring are fully connected before introducing training noise.
2. Lower Bound Benchmark: Setting a strict minimum quality target. If a complex model cannot outperform a basic rule, it should not be built.
3. Instant Business Value: Providing an immediate, fallback predictor that API clients can invoke on day one.

graph TD
     A["Sprint 0: Define Heuristics"] --> B["Build & Deploy API with Baseline"]
     B --> C["Establish Telemetry & Monitoring"]
     C --> D["Sprint 1: Train Complex ML Model"]
     D --> E{"Outperforms Baseline?"}
     E -->|"Yes"| F["Promote to Active Serving"]
     E -->|"No"| G["Keep Baseline / Investigate"]

In this module, we will explore:

1. Structuring "Sprint 0" and managing Agile-ML scope creep.
2. Implementing a simple heuristic baseline model.
3. Utilizing our unified CLI to run heuristic predictions.

---

🖥️ 1. Heuristic Baseline Execution via CLI¶

Module 18 extends our unified CLI with the mlops baseline command:

• Execute heuristic baseline locally:

  uv run mlops baseline run --input data/housing_raw.csv --output data/baseline_predictions.csv
  

• Run via Docker:

  docker run --rm -v $(pwd)/data:/app/data mlops-cli baseline run
  

Let's run this baseline check programmatically:

In [1]:

# Ensure we run from the project root directory
import os
import sys

# Locate project root (searching upwards for mkdocs.yml)
current_dir = os.path.dirname(os.path.abspath(__file__)) if "__file__" in locals() else os.getcwd()
while current_dir != os.path.dirname(current_dir):
    if os.path.exists(os.path.join(current_dir, "mkdocs.yml")):
        break
    current_dir = os.path.dirname(current_dir)

if os.path.exists(os.path.join(current_dir, "mkdocs.yml")):
    os.chdir(current_dir)
    if current_dir not in sys.path:
        sys.path.insert(0, current_dir)
else:
    print("⚠️ Could not find project root containing mkdocs.yml.")

import subprocess
import pandas as pd

# Generate default dataset if missing
os.makedirs("data", exist_ok=True)
if not os.path.exists("data/housing_raw.csv"):
    pd.DataFrame({
        "area_sqft": [1500, 2000, 2500],
        "bedrooms": [3, 4, 4],
        "price_usd": [275000, 350000, 425000]
    }).to_csv("data/housing_raw.csv", index=False)

# Run baseline command
result = subprocess.run(
    ["mlops", "baseline", "run", "--input", "data/housing_raw.csv", "--output", "data/baseline_predictions.csv"],
    capture_output=True,
    text=True
)
print(result.stdout)

# Verify outputs
if os.path.exists("data/baseline_predictions.csv"):
    print("📋 Baseline Predictions Sample:")
    df = pd.read_csv("data/baseline_predictions.csv")
    print(df)
else:
    print("❌ Failed to output baseline predictions.")

# Ensure we run from the project root directory import os import sys # Locate project root (searching upwards for mkdocs.yml) current_dir = os.path.dirname(os.path.abspath(__file__)) if "__file__" in locals() else os.getcwd() while current_dir != os.path.dirname(current_dir): if os.path.exists(os.path.join(current_dir, "mkdocs.yml")): break current_dir = os.path.dirname(current_dir) if os.path.exists(os.path.join(current_dir, "mkdocs.yml")): os.chdir(current_dir) if current_dir not in sys.path: sys.path.insert(0, current_dir) else: print("⚠️ Could not find project root containing mkdocs.yml.") import subprocess import pandas as pd # Generate default dataset if missing os.makedirs("data", exist_ok=True) if not os.path.exists("data/housing_raw.csv"): pd.DataFrame({ "area_sqft": [1500, 2000, 2500], "bedrooms": [3, 4, 4], "price_usd": [275000, 350000, 425000] }).to_csv("data/housing_raw.csv", index=False) # Run baseline command result = subprocess.run( ["mlops", "baseline", "run", "--input", "data/housing_raw.csv", "--output", "data/baseline_predictions.csv"], capture_output=True, text=True ) print(result.stdout) # Verify outputs if os.path.exists("data/baseline_predictions.csv"): print("📋 Baseline Predictions Sample:") df = pd.read_csv("data/baseline_predictions.csv") print(df) else: print("❌ Failed to output baseline predictions.")

Running Sprint 0 Heuristic baseline rule engine...
Heuristic inference completed. Saved to data/baseline_predictions.csv

📋 Baseline Predictions Sample:
    area_sqft  bedrooms  price_usd  heuristic_price
0        1660         5     259751         399000.0
1        2094         2     531050         464100.0
2        1930         2     611079         439500.0
3        1895         1     518452         434250.0
4        2438         4     256081         515700.0
..        ...       ...        ...              ...
95       1191         5     616960         328650.0
96       2498         4     274123         524700.0
97       1218         2     455628         332700.0
98       3136         4     232989         620400.0
99       1178         3     448356         326700.0

[100 rows x 4 columns]

Let's check the help menu for baseline operations:

In [2]:

result = subprocess.run(["mlops", "baseline", "--help"], capture_output=True, text=True)
print(result.stdout)

result = subprocess.run(["mlops", "baseline", "--help"], capture_output=True, text=True) print(result.stdout)

Usage: mlops baseline [OPTIONS] {run}

  Heuristic baseline model run

Options:
  --input TEXT   Input path
  --output TEXT  Output path
  --help         Show this message and exit.

---

🎉 Module 18 Completed! You have successfully implemented heuristic baselines to benchmark MLOps architectures and ensure reliable Sprint 0 operations!