Hand-Run, Pipeline or Fully Automated: Comparing MLOps Styles

Hand-Run, Pipeline or Fully Automated: Comparing MLOps Styles

Anyone running a flight-operations or training business with a prediction model eventually has to pick an operating style: update it by hand when someone notices a problem, let a schedule do it, or build a pipeline that retrains and releases automatically. Each approach has a legitimate use. Here the three are laid next to each other with the help of the Ops Atlas account of MLOps, to let you decide on evidence instead of fashion and skip automation you could not supervise.

The three options in short

  • Hand-run. People train, test and deploy the model through manual steps, usually from a notebook or a script on someone’s machine.
  • Pipeline-automated. Training is an automated, repeatable pipeline, triggered by people or a calendar, with release still decided by a human.
  • Fully automated delivery. Training, testing and release run through an integrated chain with gates, and retraining can fire on its own conditions.

They echo the three levels in Google’s maturity model as the Atlas reports it: hand-driven work first, then an automated training pipeline, then an automated build-test-release chain. Microsoft’s scale has five levels numbered zero to four and keeps generative applications apart from predictive ones. The point of both ladders, the entry says, is orientation, not a mandate to climb.

Criterion one: reproducibility

With hand-run work, the record of how a model was made often lives in memory and in file names. That works until the person who remembers goes on leave. The Ops Atlas entry on MLOps expects each run to log which code, which data fingerprint, which settings and which environment were used, and it accepts results that fall inside a tolerance rather than demanding identical output. A pipeline records these automatically. Full automation records them and refuses to proceed if they are missing.

Verdict: pipelines win clearly over hand-run work. Between the second and third options there is little difference on this criterion, provided the pipeline is disciplined.

Criterion two: speed of change

Hand-run updates are slow and uneven; a fix can wait weeks for a free afternoon. Automation shortens that to hours. Yet the Atlas notes that speed is not the same as value. Among its anti-patterns is the overnight retrain that promotes itself, because a loop like that can end up consuming what it produced and bring systems down. If a model’s output influences what data it sees next, quick automated cycles can make things worse faster.

Verdict: choose the speed your monitoring can safely absorb. A pipeline with a human decision at the release step captures most of the gain with little of the risk.

Criterion three: protection against a bad release

Here the gap between the options is largest. A hand-run release often relies on a quick glance at one headline metric. The Atlas’s gate expects a frozen, time-ordered test set and a comparison across overall performance, slices, latency and format. Its worked example showed a challenger whose overall AUC rose by 0.009 yet which was rejected because one slice fell by 0.029 when the policy allowed 0.02. After repair it was promoted.

Automated chains can enforce such gates every time, which is their great strength. Hand-run processes can enforce them too, but only through discipline. The entry also cites a review in which leakage mistakes affected 294 papers across 17 fields, a reminder that careless evaluation is common even among experts.

Verdict: automation is superior here only when the gate itself is well designed. An automated bad gate is just a faster way to approve the wrong thing.

Criterion four: monitoring and retraining

Hand-run teams tend to retrain when someone complains. Pipeline teams can retrain on a schedule. Fully automated teams may add triggers. Google Cloud’s list, which the Atlas relays, runs to five: a manual request, a timetable, newly arrived data, falling performance and a large shift in the data distribution. For picking among them, the entry proposes a staleness test that trains to a date and measures weekly decay.

Whatever the style, monitoring must cover three layers: service, data and quality. Labels often come late, so interim proxies stand in. And diagnosis must come before retraining, because skew calls for a pipeline fix, an upstream defect for a fix at the source and only genuine drift for retraining. Automation that retrains on every alert would be treating all three the same, which the entry warns against.

Verdict: the more automated your retraining, the better your diagnosis must be. A hand-run team can afford to think; an automated team must have thought already.

Criterion five: rollback

All three options need a way back, and the cheapest is the same in each. In the Atlas’s words, the way back is to repoint an alias or redirect traffic to an earlier version; kicking off a new training run does not count. A plain non-ML fallback should also be held in reserve and the whole manoeuvre rehearsed every quarter. Hand-run teams often cannot do this because the previous version was overwritten. Pipeline and automated teams, which keep a registry, can.

Verdict: a registry beats everything on this criterion, independent of automation level. Our own advice on avoiding avoidable charter safety slips follows the same philosophy: have the alternative arranged before you need it.

Criterion six: cost and ownership

Full automation is not cheap. It needs engineers who can build and maintain the chain, monitoring that someone reads and clear owners for every stage. The Atlas is direct on this: MLOps is a practice and not a product, and a platform without ownership and gates is the first level with a nicer interface. A small organisation that buys a platform to skip the hard parts ends up where it started, with a bigger invoice.

The entry’s starter advice is the most useful line for a budget conversation: a pipeline you can rerun, a version catalogue that traces ancestry and a retreat path will do more for you than an expensive platform sitting idle. That describes the middle option, pipeline-automated with human release, almost exactly.

What a small flight-training operation might do

Take a flight school whose model estimates which students are at risk of dropping out. The data is small, labels arrive slowly and the stakes of a wrong guess are modest. Full automation would be an expensive answer to a gentle question. A sensible set-up is a single scripted pipeline run each term, a test comparing the new version with the old one across student groups, a registry entry showing exactly what data was used and a note saying who to ask before anything changes. When enrolment patterns shift, someone looks at the diagnostic steps first. That is slower than a robot and far cheaper than an incident.

How the options score at a glance

  • Reproducibility: hand-run weak, pipeline strong, automated strong.
  • Speed: hand-run slow, pipeline moderate, automated fast but riskier.
  • Release protection: hand-run depends on people, pipeline good, automated strongest if the gate is good.
  • Retraining discipline: hand-run reactive, pipeline planned, automated needs excellent diagnosis.
  • Rollback: depends on the registry, not the style.
  • Cost and staffing: hand-run cheap now and expensive later, pipeline moderate, automated highest.

A way to test your own readiness

The Atlas points to a published production-readiness rubric: 28 tests across four groups, in which doing a check by hand is worth 0.5 and automating it is worth 1. The overall result takes the weakest group’s total, not an average. At the bottom end the rubric calls the system a research project rather than a production one, and a result above five counts as exceptional. The minimum-of-sections rule is the clever bit: it stops a strong testing culture in one area from hiding neglect in another. Try scoring your own setup honestly before deciding how far to automate.

Our recommendation

For most small and mid-sized aviation operations, the sensible target is the middle option. Automate the repeatable parts, keep a human at the promotion step, maintain a registry with lineage and rehearse the way back. Move toward continuous automation only when your monitoring is proven and your gates have caught real problems. If you are shopping for tooling to get there, our blueprint for choosing software sensibly gives a method for comparing vendors without being dazzled by demos.

Finally, remember that automation is a decision and not a default. The Atlas quotes Google’s opening rule of thumb: do not launch with machine learning at all unless the problem genuinely requires it. If a simple rule does the job, the wisest MLOps may be no model at all.