Only Production Exists: the limits of staging environments
The following is an AI generated summary of the video
One of the most common questions asked before any software change is:
Has it been tested?
It's a sensible question.
Whether we're changing application code, infrastructure, databases, configurations, security policies, or scaling systems up and down, we want confidence that the change won't cause problems.
The challenge is that confidence and certainty are not the same thing.
The Purpose of Staging
Most software teams use some form of non-production environment to test changes before release.
We might call it:
- Development
- Test
- Integration
- QA
- Staging
The name doesn't matter.
The purpose is the same: create a safe environment where changes can be evaluated before affecting real users.
If something breaks in staging, that's usually a cheap lesson.
If something breaks in production, the lesson can be considerably more expensive.
The Assumption We Make
The closer a staging environment resembles production, the more confidence we have in our testing.
That seems reasonable.
We mirror configurations.
We use Infrastructure as Code.
We replicate services.
We copy deployment pipelines.
We might even mirror production traffic.
The goal is to make staging as close to production as possible.
But there's an important limitation.
No matter how similar staging becomes, it is never production.
The Difference Between Similar and Identical
Imagine owning a classic car and wanting to experiment with repairs.
Rather than risk damaging the original, you buy another car of the same model to practise on.
The lessons you learn are valuable.
But it isn't the same car.
Or imagine repairing a treasured childhood Walkman for a friend.
You buy several identical units from eBay to practise on first.
Again, the experience is useful.
But those devices aren't the one that matters.
They're substitutes.
Models.
Simulations.
Approximations.
The original remains unique.
Production systems are no different.
Staging Is a Simulacrum
Even when a staging environment appears identical, subtle differences remain.
Perhaps the hardware is different.
Perhaps the scale is different.
Perhaps the traffic patterns differ.
Perhaps a configuration value is slightly different.
Even if every measurable attribute were somehow identical, staging would still be a separate instantiation of the system.
It isn't production.
It merely resembles production.
In mathematical terms, the systems may be isomorphic—structurally equivalent—but they are not identical.
The Trap of Perfect Parity
This becomes important because teams can become obsessed with achieving perfect parity between staging and production.
Every difference becomes something to eliminate.
Every inconsistency becomes a project.
Every deviation becomes a problem.
But perfection is unattainable.
No matter how much effort is invested, staging will always remain a model rather than the real thing.
Eventually, teams reach a point of diminishing returns where additional effort delivers increasingly little value.
The environment becomes "good enough" for its purpose.
Pursuing absolute parity beyond that point can become counterproductive.
Target Fixation
Pilots and drivers sometimes experience something called target fixation.
They become so focused on avoiding an obstacle that they steer directly towards it.
Engineering teams can do something similar.
By becoming excessively focused on staging fidelity, they risk losing sight of the actual objective.
The goal isn't to build the world's most perfect staging environment.
The goal is to deliver reliable software safely and efficiently.
A staging environment is a tool in service of that goal.
It is not the goal itself.
Confidence, Not Proof
This is perhaps the most important distinction.
Testing does not prove software is correct.
Testing increases confidence.
Unit tests increase confidence.
Integration tests increase confidence.
Staging environments increase confidence.
None of them provide absolute certainty.
They simply reduce risk.
Understanding this helps teams make better decisions about where to invest time and effort.
Beyond Staging
Rather than pouring every available resource into achieving impossible parity, teams can often gain more value from complementary strategies such as:
- Automation and deployment pipelines.
- Feature flags.
- Canary releases.
- Traffic mirroring.
- Observability and monitoring.
- Rapid rollback mechanisms.
- Self-healing infrastructure.
These approaches acknowledge a simple truth:
Some things can only be learned from real-world operation.
Only Production Exists
Staging environments are enormously valuable.
They catch bugs.
They build confidence.
They reduce risk.
Most teams should absolutely have them.
But it's important to remember what they are.
A staging environment is a model of production.
A useful model.
A powerful model.
But still a model.
Production remains the only environment where real users, real workloads, real data, and real business outcomes exist.
The closer we get to understanding that distinction, the better we become at balancing risk, cost, confidence, and delivery.
Because in the end, only production truly exists.