Aviation has outlived its share of dangerous folklore, and so has software. The newest round of confusion gathers around TestOps \u2014 the practice of running an automated test suite as a service whose output, a pass or fail on every change, has to be fast enough to act on and honest enough to trust. Because the term is new, nonsense collects around it quickly. What follows are six myths met repeatedly in engineering organisations, each stated plainly and then answered with what the evidence actually says.<\/p>\n
Several vendors sell software with the term in the name, which creates the impression that adopting it means buying something. The record says otherwise. There is no standards body behind the word; an academic survey of practitioners \u2014 Dole\u017eel’s study at the XP 2020 conference \u2014 found their understandings of it so scattered that no single definition held across interviews. The safest reading, laid out in the TestOps section of the Ops Atlas<\/a>, is that this is a set of operating practices, in the same sense that running a flight department is not a product but a discipline. Tools can support it. Nothing in a box can be it.<\/p>\n The confusion is understandable: both concern quality. The distinction is organisational and it matters. Test management as formalised in the ISO\/IEC\/IEEE 29119 standard \u2014 plans, procedures, documentation \u2014 predates this conversation and remains its own profession. The newer discipline instead treats the checking machinery itself as the thing to operate: budgets, environments, results, costs. It is closer to running infrastructure than to running an inspection department, and it changes who does the work, which is the subject of myth six.<\/p>\n This myth survives because its first half contains truth \u2014 intermittent failures appear in even the largest systems, including Google’s, where a 2016 study measured roughly 1.5 percent of runs misbehaving. The fatal move is the second half. Automatic retries do not remove instability; they conceal it, and along with it they conceal the intermittent genuine defects that present identically. The engineering answer is diagnosis, and the causes are well catalogued:<\/p>\n GitLab’s engineering teams and others document these categories precisely so that each flaky check gets a cause, not just a shrug and a second attempt.<\/p>\n Volume feels like virtue, and it is the most seductive myth on this list because every addition looks like progress in a dashboard. The delivery research behind DORA’s findings points the other way: a handful of checks engineers believe in outperforms hundreds they routinely override. Every untrustworthy check added to a gate does triple damage \u2014 it multiplies false alarms, it trains people to ignore red, and it consumes compute that a reliable check could have used. A suite is an evidence-generating system, and the quality of that evidence, never its raw quantity, is what a release decision actually consumes.<\/p>\n Many organisations still treat a shared pre-production environment as the final word before release. The practice fails predictably: shared machines drift from the code they claim to represent, several changes mingle in one place so attribution becomes guesswork, and access to the environment becomes a scheduling bottleneck. Mature teams treat staging as one instrument among several \u2014 useful for certain whole-system rehearsals, never the deciding vote. The deciding vote comes from cheap, reproducible, per-change environments whose verdicts can be traced to a single commit.<\/p>\n The centralisation argument sounds efficient and operates in the opposite way. DORA’s research is direct: developers should own their tests. When a separate group owns the suite, two failure loops start \u2014 the people closest to the code lose the feedback loop that would teach them their code’s testability, and the central team becomes a queue, which keeps pipelines broken longer and quietly rewards writing code that resists checking. Ownership belongs with the authors; a central group can still run the shared machinery, the way an operator runs the ramp while each captain remains responsible for their own preflight.<\/p>\n Strip the myths out and a workable position remains. Buy tools by category when a habit already hurts, but treat the operating discipline as home-grown. Keep quality ownership distributed to the people who write the code. Diagnose instability rather than retrying past it. Judge the suite by whether its verdicts are believed, and by what a verdict costs \u2014 never by raw counts. And approach artificial intelligence with the same evidence standard the rest of the field was held to: machine learning has one strong published result in this space, Meta’s test-selection model, which caught over 99.9 percent of regressions while running a third of the relevant checks \u2014 but self-healing and auto-generated tests so far lack independent verification, so record every automated repair in a log and leave the choices about deleting tests and setting gate policy with people. Travellers already apply this logic without the vocabulary; the habit of checking claims against records is exactly what guides like what to verify before booking an air charter<\/a> and which charter safety mistakes to avoid<\/a> teach, and the same scepticism serves any organisation evaluating its pipelines. For newer crews, safety tips for first-time private flyers<\/a> make the broader point: trust is built from verifiable answers, one decision at a time \u2014 which is all TestOps ever promised.<\/p>\n","protected":false},"excerpt":{"rendered":" A product in a box, a renamed QA team, retries as medicine: six TestOps myths answered with what the research and incident records actually show.<\/p>\n","protected":false},"author":6,"featured_media":912,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[615],"tags":[],"class_list":["post-911","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aviation"],"_links":{"self":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts\/911","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/comments?post=911"}],"version-history":[{"count":1,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts\/911\/revisions"}],"predecessor-version":[{"id":932,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts\/911\/revisions\/932"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/media\/912"}],"wp:attachment":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/media?parent=911"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/categories?post=911"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/tags?post=911"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Myth 2: it is a renamed QA department<\/h2>\n
Myth 3: some flakiness is normal, so just re-run<\/h2>\n
\n
Myth 4: more checks automatically mean more safety<\/h2>\n
Myth 5: shared staging is the real gate<\/h2>\n
Myth 6: one central team should own everybody’s tests<\/h2>\n
What the TestOps evidence favours instead<\/h2>\n