Skip to content

CI/CD and Massive Regression Testing

In the airline industry, deploying a "buggy" rule doesn't just mean a page fails to load—it means a flight might be grounded because a crew's schedule was built illegally, or the airline might bleed millions of dollars in union penalties.

Because of this, airlines cannot use standard CI/CD pipelines (like simply running 5 minutes of unit tests). They must rely on Massive Regression Testing.

1. The Challenge of Testing Optimization Rules

When a Rave developer changes a rule (e.g., tweaking a duty rig pay calculation), it impacts the entire optimization search space.

  • A rule that costs slightly more might cause the solver to choose an entirely different set of pairings for the whole month.
  • This "butterfly effect" means unit tests are insufficient. You must test the code against reality.

2. The Regression Testing Pipeline

To safely merge code, airlines run automated regression tests against historical "baselines."

flowchart TD
    A[Developer Pushes Rave Code to Git] --> B[CI Server triggers Regression Run]
    B --> C[Run Solver on Historical Data: Month 1]
    B --> D[Run Solver on Historical Data: Month 2]
    C --> E[Generate Output KPIs]
    D --> F[Generate Output KPIs]
    E --> G[Compare against Production Baseline]
    F --> G
    G --> H{Are Costs or Legality Worse?}
    H -->|Yes| I[Block Merge - Flag Developer]
    H -->|No| J[Approve Pull Request]

3. KPI Diffing

Instead of just checking if the code compiles, the CI pipeline (often managed via Jenkins, GitLab CI, or Airflow orchestrators) compares Key Performance Indicators (KPIs).

The pipeline outputs a "diff" report comparing the New Code vs the Current Production Code:

  • Total Optimizer Run Time: Did the new rule slow the solver down from 4 hours to 10 hours?
  • Total Crew Cost: Did the total cost increase by 5%?
  • Unassigned Flights: Did the new legality constraint make it mathematically impossible to cover all flights?

Only if the KPIs show stable or improved performance is the Rave code allowed to be merged into the main branch and deployed to the live system.