{"id":1196,"date":"2026-11-17T10:40:00","date_gmt":"2026-11-17T10:40:00","guid":{"rendered":"https:\/\/flyaiaa.com\/blog\/?p=1196"},"modified":"2026-11-20T23:40:00","modified_gmt":"2026-11-20T23:40:00","slug":"eight-lessons-automatic-fixes","status":"publish","type":"post","link":"https:\/\/flyaiaa.com\/blog\/eight-lessons-automatic-fixes\/","title":{"rendered":"Eight Hangar-Tested Lessons on AIOps Automatic Fixes"},"content":{"rendered":"<p>Ask a veteran pilot what separates a smooth flight from a scary one and the answer is rarely heroics. It is usually a boring habit: a checklist followed, a limit respected, a second opinion sought. The same truth governs AIOps when it reaches its riskiest stage, letting software fix problems without a person in the loop. Here are eight lessons about automatic remediation, told the way hangar stories get passed down, drawn from the Ops Atlas and the cases it cites.<\/p>\n<h2>1. Start from the day automation went wrong<\/h2>\n<p>Every cautious rule has a story behind it. One of the best known in operations circles concerns an automated decommissioning job at a very large search company that, through a mistake, erased the disks on servers feeding its content-delivery network. The fix was not to give up on automation. The team added sanity checks that stop a job before it removes too much, put rate limits on how fast it may act, and made the workflow idempotent so that running it twice does no extra damage.<\/p>\n<p>Flying has its own version: a mechanic who finds a mistake does not abandon maintenance, he adds a signature line. That attitude, described in <a href=\"https:\/\/www.xopsschool.com\/aiops\/\">the Ops Atlas reference on AIOps<\/a>, is the foundation for everything below.<\/p>\n<h2>2. Keep the action list short and written down<\/h2>\n<p>A remediation system should be able to do only what is on an approved list. Restart this service, rotate those logs, clear that cache. Anything not on the list goes to a human. Short lists are easy to review and hard to abuse. Long, open-ended ones turn into a second, undocumented administrator with no one watching.<\/p>\n<p>Think of an aircraft&#39;s minimum equipment list. It does not say what might be nice to fix; it says exactly what may be deferred and under which conditions.<\/p>\n<h2>3. Prefer actions you can undo<\/h2>\n<p>Reversible steps are forgiving. Restarting a stuck process or scaling out a pool can be backed out within minutes, while deleting data or changing a schema cannot. The Atlas places reversibility next to allow-listing and rate-limiting as the three properties of safe automation. If you cannot describe the rollback in a sentence, the action is not ready to run alone.<\/p>\n<h2>4. Cap how often it can run<\/h2>\n<p>Loops are the quiet enemy. A fix that triggers the very condition that triggers it will repeat endlessly. The Atlas gives a concrete example for log-volume rotation: at most three runs on any one host in a day, and at most five an hour across the whole fleet. When a cap is hit, the system should stop and call a person, because hitting the cap is itself information.<\/p>\n<h2>5. Run it with the smallest possible permissions<\/h2>\n<p>The Atlas example executes as the account that owns the logs, never as root. A line it quotes from the Ansible documentation explains why this matters: you cannot restrict privilege escalation to selected commands. Once a runner holds broad rights, any bug or crafted input inherits them. Our own note on <a href=\"https:\/\/flyaiaa.com\/blog\/common-charter-safety-mistakes-travelers-should-avoid\/\">frequent safety slips charter passengers should steer clear of<\/a> makes a similar point about not handing authority to whoever happens to be nearby.<\/p>\n<h2>6. Treat the trigger as untrusted<\/h2>\n<p>Alerts arrive from the network, and anything that starts an action must be authenticated. One event source in the Ansible collection that listens for alert webhooks accepts plain, unauthenticated HTTP, so the Atlas advises keeping it on the loopback interface. The programming interface of a widely used open-source alert router ships with a wildcard cross-origin setting, so a single forged request is enough to mute an outage. And if alert text is passed to a language model, remember that someone outside your organisation may have written it.<\/p>\n<h2>7. Verify, then page if it did not work<\/h2>\n<p>An automatic fix that does not check its own result is a coin flip. Build in a verification step: did disk usage fall, did the service answer? When the check fails, escalate to a person immediately with the evidence attached. The Atlas pattern ends with exactly that: verify, and page on failure. Silent failure is the worst outcome, because the problem stays and the dashboard says all is well.<\/p>\n<h2>8. Earn each level of trust in stages<\/h2>\n<p>Finally, no new automation should go straight into production. The Atlas describes a policy lifecycle: proposed, then shadow, where the rule is scored but invisible; then a canary limited to a pair of services; then active, then a review, then eventually retired, with explicit rollback triggers at each step. Fresh rules fail in surprising ways, and shadow mode is the cheap way to find those surprises.<\/p>\n<p>The same staging logic shows up in how careful travellers book their first trip with an operator, as covered in <a href=\"https:\/\/flyaiaa.com\/blog\/booking-mistakes-first-time-charter-travelers-should-avoid\/\">errors first-time charter customers make when reserving<\/a>. Small, verifiable steps beat one large leap.<\/p>\n<h2>The temptation to skip ahead<\/h2>\n<p>Why do teams ignore these lessons? Mostly because the dream is seductive. Self-healing infrastructure is a compelling slogan, and benchmark scores for language-model agents sound impressive. The Atlas is blunt: a system that heals itself is where the field is heading, not an item on a price list, and leaderboard diagnosis scores of 56 to 57 percent describe narrower tasks than handling a live incident. One benchmark had agents on early-2025 models clearing just 11.4 percent of 102 scenarios. Nothing there justifies letting an agent act on its own.<\/p>\n<p>It also helps to remember what comes first in the Atlas ordering. Labels, duplicates, grouping and correlation with changes deliver most of the noise reduction. Automation is item seven of eight, deliberately placed after the dull work, because a remediation triggered by a badly grouped alert is just a faster way to make the wrong move.<\/p>\n<h2>What a fence looks like on paper<\/h2>\n<p>General advice is pleasant to agree with and awkward to apply, so here is the shape of a one-page remediation policy a team might keep beside the runbook. Each automated action gets a name, an owner who answers when it misbehaves, the exact condition that triggers it, the command it is allowed to run, the account it runs as, and the cap on how often it may fire. Beneath those fields sits the verification check and the person to page when the check fails.<\/p>\n<p>Writing it forces useful arguments. People discover that two teams assumed the same script was theirs, or that nobody can say which account the runner uses. Fixing those gaps costs an afternoon, while discovering them during an incident costs far more.<\/p>\n<ul>\n<li>Is the action on the approved list, and who signed it off?<\/li>\n<li>Can it be reversed in one sentence of instructions?<\/li>\n<li>Is there a per-host and a fleet-wide cap?<\/li>\n<li>Does it run with the narrowest account that can do the job?<\/li>\n<li>Is the trigger authenticated and kept off public networks?<\/li>\n<li>Does it verify its result and page on failure?<\/li>\n<li>Has it spent time in shadow and on a canary?<\/li>\n<\/ul>\n<h2>A closing story<\/h2>\n<p>An old flight instructor liked to say that the checklist is not an insult to your skill, it is a record of every mistake someone else already made on your behalf. Automatic remediation rules are the same. Each limit, each permission boundary, each page-on-failure line is a lesson learned somewhere at someone&#39;s expense. Write them down before you need them, and let the software act only inside the fence you have built.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Eight lessons on automatic remediation in AIOps, told like hangar stories: short action lists, reversibility, caps, least privilege, verification and staged trust.<\/p>\n","protected":false},"author":6,"featured_media":1197,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[634],"tags":[],"class_list":["post-1196","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flight-training-tech"],"_links":{"self":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts\/1196","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/comments?post=1196"}],"version-history":[{"count":1,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts\/1196\/revisions"}],"predecessor-version":[{"id":1325,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/posts\/1196\/revisions\/1325"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/media\/1197"}],"wp:attachment":[{"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/media?parent=1196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/categories?post=1196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/flyaiaa.com\/blog\/wp-json\/wp\/v2\/tags?post=1196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}