The Word "Test" Has Eaten Engineering
Three different things now wear the same name, and we are losing the difference between them
David Proctor
Apr 28, 2026
In 2012, an engineer at Microsoft Bing got bored of waiting. A colleague had pitched a tiny tweak to how ad headlines were displayed. The idea sat in a backlog for months, too small to argue about, too speculative to ship. So the engineer did the lazy thing: he ran an A/B test. Within hours the new headline variation was producing abnormally high revenue, triggering a "too good to be true" alert. An analysis showed that the change had increased revenue by an astonishing 12%, which on an annual basis would come to more than $100 million in the United States alone, without hurting key user-experience metrics.
The story is told inside Microsoft as a parable about the power of experimentation. I read it as something else: the moment the word test quietly stopped meaning what it used to.
Once upon a time, a "test" was a thing you ran before you shipped. You wrote some code, you wrote some assertions about that code, you ran the assertions, and a green checkmark gave you permission to merge. Today, when an engineer says the word "test," they could mean any of three completely different activities — with completely different epistemologies, completely different failure modes, and completely different ethical implications.
Everyone is Testing on You, All the Time
Representatives with experience in large-scale experimentation from thirteen different organizations — Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University — were invited to the first Practical Online Controlled Experiments Summit. Together these organizations tested more than one hundred thousand experiment treatments last year.
100K+
Experiments per year
Across 13 major tech organizations combined
25K
Booking.com tests
Roughly 70 experiments per day on live users
10K+
Per company annually
Microsoft, Amazon, Google, Facebook each exceed this threshold
"This is sold as a triumph of empiricism, and in narrow ways it is. Jeff Bezos has spent two decades evangelizing the idea that "Our success at Amazon is a function of how many experiments do we per year, per month, per week, per day.""
"But notice what an A/B "test" is actually testing. It is not testing whether the code works. It is testing whether a particular metric — clicks, conversions, time-on-page — moves up. The thing being verified is not correctness; it is desirability of an outcome the company has chosen to optimize for. Booking.com at least knows this, which is why Booking.com maintains guardrails such as cancellation rates, customer service contact rate, payment failures, refund friction, load times, and accessibility. Most companies running experiments do not have those guardrails. They have a primary metric and a stopwatch."
"When you call this a "test," you are borrowing the moral authority of a unit test — the implication that something has been verified — and applying it to a process whose actual question is "did more people click?" These are not the same question."
The QA Role Didn't Die. It Got Rewritten.
Meanwhile, the original meaning of "test" — the boring, deterministic, pre-production kind — is going through its own quiet revolution. The dedicated QA department, the one whose entire job was to keep bad code from reaching users, is being dismantled.
The Indeed Story
In March 2023, Indeed laid off 15% of its 14,000-person staff. The layoffs resulted in the elimination of the QA role — most QA engineers were let go. Those not shown the door were given 6 months to pass an interview and transition to an SWE position.
What happened next is the depressingly predictable part: an anonymous software engineer told Gergely Orosz that since the layoffs, "The overall quality of tests has nosedived."
Industry-Wide Shifts
43%
Manual testing decline
Since 2023
17%
QA engineering growth
From 2023 to 2025
64%
AI adoption in QA
Actively using or building a roadmap
The cynical reading is that companies used the AI moment as cover for cost-cutting they wanted to do anyway. A QA team of 10 could easily cost $1m per year. The more interesting reading is that the role of testing has migrated up the stack. It has stopped being a department and started being a property of the deploy pipeline, and increasingly, of the running system itself.
Where 'Test' Lives Now
"The only environment that matters is production. For the good of humanity, ditch the rest. — Charity Majors, CEO, Honeycomb"
It is a heresy, and it is also mostly true. If you run a distributed system in 2025, the staging environment is a fond fiction. The interesting failures — cardinality explosions, slow p99 drift, the photo that loads for 99.7% of users but not for the ones in São Paulo on Android 11 — are not catchable in any reasonable approximation of production.
So the third meaning of "test" is born: continuous, instrumented, observation-as-verification. Feature flags. Canary deploys. High-cardinality telemetry. "I hope that we look back in a couple years at the bad old days when we used to just ship code and wait to get paged."