A Flight Academy IT Team Learns DisasterRecoveryOps in One Week

A Flight Academy IT Team Learns DisasterRecoveryOps in One Week

Nobody in this story is real; treat it as a composite sketch of how a small flight academy’s IT group might learn DisasterRecoveryOps the practical way. Three people look after the scheduling portal, student records and the maintenance-log database for a training operation with a fleet of trainers. They had backups, a nightly job and a folder called “DR plan” that nobody had opened in a year. Then they decided to test it.

Monday: the folder meets reality

The team lead opened the folder and read the plan aloud at the Monday meeting. It said “restore from backup” and gave a phone number that had changed. No target times, no named decision-maker, no order of steps. The database admin admitted he had never restored the scheduling data to anywhere except his own laptop.

They agreed on a rule: no more reading plans, only running them. Before anything else, they wrote down two questions for each of their three systems. How long could it stay down before lessons were cancelled? And how recent a copy would they need? The answers were uncomfortable. Scheduling could tolerate about two hours on a normal day, student records could wait most of a working day, and the maintenance log, which gates whether an aircraft may fly, had to be restorable to within a few minutes.

Tuesday: the first drill goes badly

They chose the scheduling portal for the first rehearsal, using a spare copy of the environment. The database admin switched off the primary at ten sharp. Nobody noticed for six minutes. Then came a stretch the team called “the shrug”: three people asked each other whether this was a real problem, one of them checked the status page, and eventually the team lead said “let us go.” That decision came twenty-two minutes after the failure. The technical restore then took thirty-five minutes.

Added up, the drill took nearly an hour and a quarter against a two-hour target for scheduling. That was a pass, but when they applied the same pattern to the maintenance log, where the target was thirty minutes, the numbers made them wince. The restore alone would have used all of it. They had also learned the hard way what the Ops Atlas teaches as its central lesson: the recovery clock includes the people, not only the machines.

Wednesday: fixing the human part

The team made four changes, all of them boring and all of them effective.

  • The team lead became the person who could invoke the plan, with the database admin as deputy, and a rota ensured one of them was always reachable during flying hours.
  • They calculated a decision deadline for each system: the target time, less the slowest recent drill, less a margin. For the maintenance log the answer was uncomfortably short, which pushed them to rethink the strategy.
  • They wrote an alert message template, so the first person to notice did not have to compose anything.
  • They created a short checklist and put it where the on-call phone is kept.

Thursday: picking a strategy that matches the stakes

For the maintenance log they could not rely on restoring from a nightly copy. The team compared the options the Atlas lists: backup and restore, a pilot light with the data tier always running elsewhere, a warm standby, a hot standby, and multi-site operation. They chose a pilot light for the log, because the data would be continuously copied to a second location and only the application layer would need starting. Scheduling stayed on backup and restore. Student records got a pilot light as well, mostly because it was cheap once the first was built.

They also checked an assumption: spreading the portal across two availability zones, which they already did, was not recovery from a regional event. It was good for a single building problem and nothing more. Written on the whiteboard, that sentence settled a debate that had dragged on for months.

Friday: the second drill, and the lost writes

The second rehearsal was a planned failover of the maintenance log to its standby. Here the team met a subtle problem. Between the last copy and the cut-over, two sign-off entries had been recorded on the old system but not yet replicated. A real failover loses exactly this kind of record. They followed a method they had read about: preserve the old primary untouched, rebuild a clone of the data, diff it against the new primary, then re-enter the missing entries through the application so its checks still run. It took longer than they liked, but nothing was invented and nothing was lost.

Their fencing also had a gap. The portal’s public address moved over cleanly, yet a nightly reporting job kept writing to the old database for another ten minutes because it used a stored connection string. The admin added that job to the checklist as its own step. Fencing, they now understood, is a list, not a single switch.

The weekend: the way back

Nobody wants to rehearse the return, which is exactly why they did. Moving the maintenance log back to its original location was a second migration, with the same risk of missed entries. They scripted it, ran it on the staging copy and discovered that the steps depended on a console menu that only the database admin remembered. By Sunday evening it was a script anyone on the rota could run.

What changed after a week

The folder now held real material: two objectives per system, names for authority and deputy, a decision deadline, an ordered checklist, a record of drill timings and a short list of follow-ups with owners and dates. They also arranged emergency logins stored away from the main hosting account, since a plan that needs a locked service to sign in is useless. They scheduled a repeat drill every quarter for the top tier, and agreed that the real test was two clean runs in a row, not one lucky one.

If a story like this makes you want to dig deeper, the Ops Atlas page on DisasterRecoveryOps collects the standards, strategies and anti-patterns in one place. And if your team is building its skills as it goes, there is a helpful overview of growing a technology career through structured learning, plus a Japan-focused plan for software engineering and cloud work that covers the wider territory.

The takeaway for any academy, large or small: the first drill will probably go badly, and that is the point. Every minute you find in a rehearsal is a minute you will not lose in a real outage.