Back to Garden|Archive

The Primacy of Simple Models

Start with a simple Model first, if it fails, you know you are in the wrong direction without going too far. So you can change the direction and route.

Wed, Nov 26th
datasciencestrategyproblem-framingperspectivemachinelearningsystemthinking
Created: 2025-12-15Updated: 2025-12-15

“If you can’t get at least moderately good or promising results from a simple statistical method, proceeding with a more sophisticated method is dangerous.”

Core Principle

A simple model is not just a baseline, it is a diagnostic instrument.
If a simple method cannot extract meaningful signal, it is almost always because:

  • the data is inadequate,
  • the features are mis-specified,
  • the assumptions are broken,
  • or the problem definition is flawed.

In such cases, adding sophistication doesn’t fix the foundation, it buries the flaw under complexity and makes it harder to detect.

Why This Matters

1. Sophisticated methods amplify both signal and noise

If your data contains weak signal or poorly engineered features, complex models will:

  • fit noise,
  • hallucinate patterns,
  • overfit aggressively.

2. Configuration errors compound with complexity

A mis-tuned simple model (e.g., bad regularization in logistic regression)
→ becomes a catastrophe when scaled into a deep model with thousands of parameters.

3. Complexity obscures interpretability

When you can’t explain failure in a simple model,
you definitely can’t explain it in a sophisticated one.

4. Sophistication is not a substitute for understanding

It reveals nothing new unless the foundation (data → features → assumptions) is correct.

Different Perspective

Model sophistication = higher energy state**.
Higher energy states should only be entered if the system is stable at lower levels.

If the foundation model is unstable, the entire modeling tower collapses.

Complex models magnify ontology, meaning they magnify:

  • wrong assumptions,
  • wrong data distributions,
  • wrong target framing,
  • wrong causal structure.

Sophistication isn’t power;
it is amplification.
And it amplifies whatever is already there, including errors.

The “Ladder of Sophistication” Heuristic

Step 1: Build the simplest possible version

Examples:

  • mean baseline
  • linear/logistic regression
  • simple ARIMA
  • simple decision tree
  • naive Bayes
  • 1-feature model

Check:

  • Is the signal detectable at all?
  • Are residuals structured or random?
  • Do the predictions correlate with truth?

Step 2: Increment complexity one notch at a time

For example:

  • linear → polynomial → regularized → tree-based → ensemble → neural
  • ARIMA → SARIMA → Prophet → hybrid ML models

Each step must improve:

  • accuracy
  • calibration
  • stability
  • generalization

If no improvement, STOP.
Investigate assumptions, data quality, feature construction.

Step 3: Only escalate when each layer is validated

This is the equivalent of layer-by-layer integrity checking in systems design.

The Failure Point: Overfitting

“Methods too complex for the data or goals = overfitting.”

Overfitting is not “your model memorized training data.”
That is the symptom.

The deeper cause is mismatch between model complexity and data complexity.

Examples:

  • Complex neural net on tiny dataset.
  • Random forest on meaningless features.
  • Deep forecasting model on non-stationary time series with no preprocessing.
  • High-dimensional embeddings with no domain constraints.

Overfitting = sophistication beyond what the evidence can support.

Practical MLOps Guidance

1. Enforce a “Simple Baseline First” policy

  • Baseline must be logged, versioned, and used as comparison point.
  • No complex model allowed into production unless it beats baseline on multiple metrics.

2. Require monotonic improvement

Model complexity ↑ → must result in:

  • better validation performance
  • better calibration
  • better error distribution behavior
  • no loss in interpretability for high-stakes tasks

3. Monitor for complexity-induced drift

Sophisticated models are:

  • more brittle to distribution shifts,
  • more sensitive to missing values,
  • harder to debug when live data diverges.

Use:

  • PSI, KS tests, SHAP drift, partial dependence drift monitoring.

Reflection Prompts

  • Does the simple version of my model show any meaningful signal?
  • Are my features well-defined, well-scaled, and aligned with assumptions?
  • Did I jump to sophistication out of impatience or unclear problem framing?
  • Am I hiding flawed logic inside a fancier algorithm?
  • Is the validation curve improving monotonically with complexity?
  • If my simple baseline fails, do I understand why?
  • What exactly am I trying to gain from increased sophistication?