Predictive Maintenance System

The brief

You will get this one phrased in a way that sounds like an AI question and is not.

“Design a system that ingests sensor data from industrial machines and predicts failures before they happen.”

Or, the version that carries more of the actual product in it:

“A manufacturer has ten thousand pumps across forty plants. They currently service them on a fixed schedule and still get unplanned outages. Design something better.”

In plain terms, machines emit signals: vibration, temperature, current draw, pressure, acoustic noise, and error codes. Those signals change before a machine breaks. The product watches the signals and notices the change. It then tells a maintenance planner to look at a specific machine within a specific window. The alert has to come early enough that the repair happens on a Tuesday morning rather than at 3am with a production line stopped.

The business case is entirely about the difference between planned and unplanned downtime. Planned downtime is a scheduled hour. Unplanned downtime is an hour of stopped line, plus expedited parts, plus overtime, plus whatever the downstream contractual penalty is. That ratio is often ten to one or worse. It is the reason the system exists, and it drives almost every design decision that follows.

The most valuable thing you can do in the first two minutes of this interview is say out loud that this is not an LLM problem. There is more on that below, but say it early, because the rest of your design will make more sense once you have.

What I’d ask first

What is the failure you are trying to catch, and what does it cost when you miss it? This is the first question, because it fixes the operating point of the entire system. Suppose a missed failure costs $200,000 in stopped line and a false alarm costs a technician half a day. Then you are going to run at a threshold that produces a lot of false alarms, and you need to know that going in. If the ratio is closer to even, you build a much more conservative system.

How much run-to-failure data do you have, and how are failures recorded? “Run-to-failure” means a sensor trace that continues all the way to an actual breakdown, rather than being cut short by a preventive replacement. This is the single hardest constraint in the domain. A well-maintained fleet produces almost no run-to-failure examples by construction, because the machines get serviced before they break. “We have four years of sensor data and eleven confirmed failures” is a different project from “we have three hundred failures with technician-confirmed root causes.”

How far ahead does the prediction need to be, and what is the maximum useful horizon? Maintenance has logistics. A part has to be in stock, a technician has to be scheduled, and the line has to have a window. A prediction seven days out is actionable. A prediction ninety seconds out is a fancy alarm, and the machine already has one of those. A prediction ninety days out is unfalsifiable, and nobody will trust it.

What sensors exist, at what sampling rate, and how do they get off the machine? There is a wide gap between “a PLC exposes ten tags at 1Hz over Modbus” and “we have accelerometers sampling at 25kHz.” The second one cannot send raw data to the cloud, so it forces edge processing. The first one fits comfortably in a normal time-series pipeline.

Are the machines identical? A fleet of one model from one vendor is a tractable modelling problem. Forty models from twelve vendors across four decades of installation is a data integration problem wearing a machine learning costume, and the integration is where the year goes.

Who acts on the alert and what does their day look like? The output of this system is a work order in somebody’s queue. If that person already has forty open work orders, an alert that says “pump 4471 looks unusual” is noise. The design has to produce something rankable and something explainable.

What I’ll design against

Ten thousand rotating assets across forty plants: pumps, compressors, and motors. Roughly two hundred sensor channels per plant. Most run at 1Hz, and a subset of vibration channels at 10kHz or above are handled on the edge. Four years of history, with around 250 recorded failure events. Maybe 90 of those have trustworthy timestamps and a confirmed cause. Target horizon: alert between 3 and 14 days before failure. Cost asymmetry: a missed failure costs about $150k, and a false alarm costs about $400 of technician time plus a slow erosion of trust that is the real cost. Users: plant maintenance planners, plus a small central reliability engineering team.

The design

  MACHINE / EDGE                 INGESTION                STORAGE
 ┌────────────────┐        ┌───────────────────┐    ┌──────────────────┐
 │ PLC / sensors  │        │  MQTT / Kafka     │    │ raw time-series  │
 │ 1Hz tags       ├───────▶│  broker           ├───▶│ (Timescale /     │
 │                │        │  + schema check   │    │  Influx / Parquet│
 │ ┌────────────┐ │        │  + dedupe         │    │  on object store)│
 │ │edge gateway│ │        └─────────┬─────────┘    └────────┬─────────┘
 │ │ 10kHz vib. │ │                  │                       │
 │ │ → FFT feats│ │                  ▼                       ▼
 │ └────────────┘ │        ┌───────────────────┐    ┌──────────────────┐
 └────────────────┘        │ store-and-forward │    │ FEATURE PIPELINE │
        ▲                  │ buffer (offline)  │    │ rolling windows, │
        │                  └───────────────────┘    │ spectral bands,  │
        │                                           │ deltas vs baseline│
        │                                           └────────┬─────────┘
        │                                                    │
        │            ┌───────────────────────────────────────┤
        │            ▼                                       ▼
        │   ┌──────────────────┐                  ┌──────────────────────┐
        │   │ FEATURE STORE    │                  │  TRAINING (offline)  │
        │   │ online + offline │◀────same code────┤  GBDT / survival /   │
        │   └────────┬─────────┘                  │  anomaly detector    │
        │            │                            └──────────┬───────────┘
        │            ▼                                       │
        │   ┌──────────────────┐    ┌──────────────┐         │
        │   │ SCORING SERVICE  │◀───┤ model registry│◀───────┘
        │   │ per asset, hourly│    └──────────────┘
        │   └────────┬─────────┘
        │            ▼
        │   ┌──────────────────┐   ┌───────────────────┐
        │   │ ALERT ENGINE     │   │ EXPLANATION LAYER │
        │   │ hysteresis,      ├──▶│ (LLM: turns       │
        │   │ dedupe, ranking, │   │  features+history │
        │   │ suppression      │   │  into a briefing) │
        │   └────────┬─────────┘   └─────────┬─────────┘
        │            └──────────┬────────────┘
        │                       ▼
        │            ┌──────────────────────┐
        └────feedback┤ PLANNER UI + CMMS    │
             loop    │ work order, triage,  │
                     │ outcome capture      │
                     └──────────────────────┘

Ingestion. Sensors publish to a broker. MQTT runs at the plant edge, and it bridges into Kafka centrally. The two things that matter here are not glamorous. The first is store-and-forward. Plant networks drop, so the edge gateway must buffer locally and replay on reconnect. That means your downstream has to tolerate late-arriving data measured in hours. The second is schema and unit validation at the boundary. A vendor firmware update that silently changes a temperature channel from Celsius to Fahrenheit will produce a model that quietly stops working, and the only place to catch that is at ingest.

Edge processing. High-rate vibration data does not travel. A 25kHz accelerometer produces roughly 2GB per channel per day raw. Across a plant that is untenable, and it is also pointless, because nobody needs the raw waveform downstream. So the gateway computes the features on-site and ships a few dozen numbers per minute instead. Those features are RMS amplitude, kurtosis, and energy in specific frequency bands tied to the bearing’s fault frequencies. Keep raw snapshots on a rolling window at the edge. Then, when something does break, you can pull the last 48 hours of waveform for forensics.

Storage. Use two tiers. A time-series database holds recent data and supports fast range queries per asset. The UI and the scoring service read from that tier. Columnar files on object storage, partitioned by asset and date, hold everything older. Training reads from that tier. Do not try to make one system serve both, because the access patterns are opposite.

Feature pipeline. Features are computed over rolling windows: mean, standard deviation, min, max, slope, and rate-of-change over 1h, 24h and 7d. Add the spectral features from the edge. Add derived context such as operating hours since last service, load level, ambient temperature, and duty cycle. The important discipline is that the same code computes features for training and for serving. A feature store exists for exactly one reason, which is to make the offline and online paths identical. Training-serving skew is where the numbers your model saw in training differ subtly from what it sees in production. It is the most common way a predictive maintenance project quietly fails, and it is a plumbing bug rather than a modelling one.

Models. Three components, layered:

1. A per-asset-class anomaly detector (an isolation forest, or a simple autoencoder, or in many cases just a robust control chart on residuals from a physics baseline). This works with no failure labels at all, and it is what you ship in month two.
2. A supervised failure classifier. Use gradient-boosted trees over the window features, with the target “does this asset fail within the next 14 days.” This needs labels, so it is what you ship once you have enough of them.
3. Optionally a survival / remaining-useful-life model, for the small number of asset classes where you genuinely have run-to-failure curves. Time-to-event modelling handles censored data properly, meaning assets that were serviced before failing. The binary classifier does not.

Alerting. The model produces a score. The alert engine produces a decision. Those are not the same thing. The engine applies four things. It applies hysteresis, so a score must stay elevated for N consecutive windows before firing, which kills most spurious spikes. It deduplicates against open work orders. It suppresses during known maintenance windows and startup transients. It ranks across the fleet, so the planner sees the ten most urgent rather than all four hundred elevated assets.

Human surface. The planner gets a ranked queue inside their existing workflow, integrated with the CMMS. CMMS means computerised maintenance management system, and it is the system of record for work orders. Each alert shows the score, the horizon, the top contributing signals with their recent traces plotted against the asset’s own normal band, and similar historical cases. The surface also captures the outcome. The technician marks whether they found a real problem, what it was, and what they did. That feedback loop is not a nice-to-have. It is the only source of new labels you will ever get.

Where the AI actually is

Be blunt about this, because it is the point of the chapter.

This is a classical machine learning problem, and mostly not even a deep learning one. The prediction layer is gradient-boosted trees over engineered window features, plus an unsupervised anomaly detector, plus possibly a survival model. On tabular sensor features with a few hundred positive examples, a well-tuned GBDT beats a neural network essentially every time. It trains in minutes, runs on a CPU, and produces feature attributions the reliability engineer can argue with. Reaching for a transformer here is a red flag, and reaching for an LLM is a bigger one.

What an LLM must not do: look at sensor values and predict failure. It is the wrong tool on every axis. It has no calibrated notion of probability over a numeric distribution. It costs orders of magnitude more per inference. It cannot run on an edge gateway. You cannot get a reliable confidence out of it, and you cannot explain its output to an auditor after an incident. If someone proposes feeding a window of vibration readings into a language model as text, the answer is no. The reason is that you would be replacing a well-understood estimator with an expensive one that you cannot validate.

Where a language model does earn its place:

• The explanation layer. The model says: score 0.83, top contributors are bearing-band energy up 40% over 10 days and winding temperature trending up. The LLM takes that, plus the asset’s service history and the relevant section of the maintenance manual, and turns it into three sentences a planner reads in ten seconds: “Pump 4471 shows a rising outer-race bearing signature consistent with the failure in June on the sister unit. Recommend inspection within 7 days; spare bearing kit is in stock at this plant.” That is real value, and it is squarely a language task.
• Unstructured maintenance history. Decades of free-text technician notes contain the ground truth about what actually failed. An LLM is the best available tool for turning “replaced brg, noisy, see wo 88213” into a structured failure code. This is a labelling accelerator, and it is probably the highest-ROI use of a model in the whole system.
• Natural language over the fleet. “Show me every compressor whose vibration trend has worsened since the last service.” That is text-to-query over a well-defined schema, with the query shown to the user before it runs.
• Triage assistance. Retrieval over manuals, past work orders, and OEM bulletins, so the technician arrives knowing what three things to check.

Here is the honest split. The model training is maybe two weeks of a data scientist’s time, and it gets revisited quarterly. The year goes to the ingestion reliability, the feature pipeline, the CMMS integration, the alert suppression logic, the backfill tooling, the asset hierarchy modelling, and the planner UI. That is the 20/80 in its most literal form, and in this product it might be closer to 10/90.

Key decisions and tradeoffs

The last row deserves emphasis, because it is the tradeoff generalists miss. The binding constraint on a predictive maintenance system is not model accuracy. The binding constraint is how many alerts a maintenance planner will act on before they start ignoring the tool. Design to that number.

What breaks

Class imbalance, and the metric it corrupts. Failures are perhaps 0.01% of your labelled windows. A model that always predicts “no failure” scores 99.99% accuracy and is worthless. Accuracy and ROC-AUC both look flattering under this imbalance. Precision-recall curves do not, which is why you use them. The failure mode here is not the model. It is a team reporting AUC 0.94 to a customer and being surprised when precision at the actual operating threshold turns out to be 4%.

Label ambiguity: what even is a failure? This is harder than it sounds, and it is where most of the modelling risk actually lives. Is a failure the moment the machine stopped, the moment the operator noticed noise, the moment the work order was opened, or the moment the technician found the cracked bearing? Those can be a week apart. Your prediction horizon is one to two weeks, so a one-week labelling error destroys the signal. There is a worse case. A machine that was serviced preventively, and would have failed, is an unlabelled positive sitting in your negative class. So you need a labelling protocol agreed with the reliability team, written down, and applied retroactively to history. Say in the interview that this document, not the model, is the artifact that determines whether the project works.

Survivorship in the training data. Good maintenance programmes destroy the data needed to model failure. The assets that produce run-to-failure traces are disproportionately the neglected ones, on the least critical lines, at the worst-run plants. So your model learns the failure signature of badly maintained equipment, and it is then deployed on well-maintained equipment.

Concept drift as machines age and operations change. Bearings wear, so the “normal” vibration baseline of a five-year-old pump is not that of a new one. Then the plant changes its product mix, the duty cycle shifts, and every machine’s baseline moves at once. Then a sensor is replaced and its calibration offset changes. Each of these silently degrades the model, and none of them shows up as an error. There are four mitigations. Make baselines per-asset and rolling, rather than fleet-wide and fixed. Monitor drift on the input feature distributions independently of any prediction. Retrain on a schedule, fed by the outcome feedback loop. Add an explicit re-baselining step in the workflow after any service event or sensor swap.

The cold-start problem for new assets. A machine installed last month has no personal baseline. So you need a fleet-level prior for its class, and an explicit “insufficient history” state in the UI, rather than a confident-looking score computed from three weeks of data.

Sensor faults masquerading as machine faults. A loose accelerometer produces a beautiful, dramatic anomaly. So does a failing thermocouple, a dying battery on a wireless node, and a gateway whose clock has drifted. So you need a sensor-health model that runs before the asset-health model. You also need a rule that a signal from a single channel, with no corroboration from correlated channels, gets downgraded rather than escalated.

Timestamp and clock chaos. Plant gateways lose NTP sync. Data arrives hours late after a network outage, out of order, or duplicated after a replay. Daylight saving transitions produce a duplicated hour and a missing one, every year, in every plant in a DST jurisdiction. Any window feature computed over a misaligned timeline is garbage, and it will be garbage silently.

The intervention paradox. Once the system works, failures stop happening. So positive labels dry up, and you can no longer measure whether it still works. There is a worse consequence. An alert that leads to a repair looks in your data exactly like a false positive would have looked if nobody had checked. This is the domain-specific failure a generalist misses entirely. The mitigation is discipline in the outcome capture: the technician must record whether they found a genuine fault, and you must treat “found and fixed” as a true positive even though the failure never occurred.

Alert fatigue, which is the actual killer. Every system in this space dies the same way. There are too many low-confidence alerts, planners stop looking, the tool becomes shelfware, and the post-mortem blames the model.

How you’d evaluate it

Offline, with the only split that means anything: time-based. Random k-fold on time-series data leaks the future into the past, and it will hand you a spectacular, entirely fictional result. Train on 2022–2024 and validate on 2025. Respect the horizon: a window labelled from a failure on day D must not use any feature computed after D minus the horizon.

Metrics that survive the imbalance. Use precision-recall AUC rather than ROC-AUC. Report precision and recall at the operating threshold you will actually deploy, not at the best point on the curve. Then add an event-level metric rather than a window-level one. What fraction of the 90 real failures were flagged at least once inside the useful window of 3 to 14 days before the event, and how many alerts did that cost per week per plant? Window-level metrics flatter you, because one failure generates many positive windows.

A cost curve rather than a single number. Plot expected annual cost against threshold, using the customer’s own downtime cost and technician cost. The optimum is where the marginal cost of another inspection equals the marginal expected saving from another catch. This turns a modelling argument into a business conversation, which is where it belongs. It is also the answer to “what’s your F1?” that will impress an interviewer.

Online. Run shadow mode first. Run for a full maintenance cycle, generate alerts into a log that nobody acts on, and compare against what actually happened. Then do a staged rollout by plant, with a control group of plants left on the fixed schedule so you can attribute the change. The business metric that actually matters is unplanned downtime hours per asset per year, with maintenance cost per asset as the guardrail metric. A system that eliminates unplanned downtime by triggering three times as much maintenance has not helped anyone.

Catching regressions. Monitor input feature distributions per asset class and alert on drift independently of any prediction, because input drift is the leading indicator and prediction degradation is the lagging one. Track precision on a rolling basis from the technician outcome feedback. Keep a frozen golden set of historical failure events and re-score it on every model change. A candidate that misses a failure the current model catches does not ship, regardless of aggregate metrics.

For the general machinery of eval harnesses, LLM-as-judge scoring for the explanation layer, and regression gating in CI, see the sibling agentic-ai-evaluation-guide rather than rebuilding it here. The part that is specific to this product is everything above about time-based splits, event-level metrics, and the cost curve.

Follow-ups they will ask

“You have eleven failures. Can you even train a model?” Not a supervised one, and I would say so rather than pretending. With eleven events I build an unsupervised anomaly detector per asset class. Then I use the first year in production to generate labels through the technician feedback loop. I would also mine the free-text maintenance history with an LLM to recover historical events that were never coded, which in practice can turn eleven into eighty. And I would be honest with the customer that year one is a labelling programme with a monitoring product attached, not a prediction product.

“How do you set the alert threshold?” From the cost asymmetry, not from the ROC curve. If a missed failure is $150k and an inspection is $400, the break-even is roughly a 1-in-375 chance of a real fault. So I should be willing to inspect on quite weak evidence. In practice the human attention budget binds before the economics do. So I take the top N assets per planner per week by expected cost saved, where N is what that planner will genuinely action. The threshold falls out of the ranking.

“Why not just use an LSTM or a transformer on the raw signal?” Because I have ninety positive examples. Deep sequence models need volume, and they need the failure modes to be represented in the training set. With this much data a GBDT over engineered window features will beat them, train in minutes, and give me SHAP values I can put in front of a reliability engineer. I would revisit that if we got to thousands of labelled events on a homogeneous fleet. I would also revisit it for the high-rate vibration channels specifically, where a learned spectral representation can genuinely beat hand-designed frequency bands.

“A technician inspects, finds nothing, and closes the work order. Was that a false positive?” Probably, but not certainly, and the distinction matters enough to build for. It could be a genuine early-stage fault below the threshold of visual inspection. Or the fault could be somewhere the technician did not look. So I capture structured outcomes: inspected and found, inspected and not found, or not inspected. Then I follow the not-found assets forward. If an asset flagged, was cleared, and failed three weeks later, that was a true positive with a bad inspection, and that is a workflow finding rather than a model finding. Failing to distinguish these is how teams end up retraining on wrong labels and degrading a working model.

“Concept drift — how do you actually detect it in production without labels?” There are three layers that do not need labels. The first is input drift: population stability index or a KS test on each feature’s distribution per asset class, week over week. The second is prediction drift: the distribution of scores across the fleet. If the mean score climbs steadily, either the fleet is genuinely degrading or something moved. The third is a physics-based residual. For many assets you can write an approximate expected relationship, such as power draw as a function of flow and pressure. Drift in the residual from that relationship is grounded in a way that a purely statistical detector is not. Labelled performance is the confirmation, and it arrives months later.

“Edge or cloud? Defend it.” Split by latency requirement and bandwidth. Anything protective, such as a hard vibration limit that trips the machine, is a deterministic rule on the PLC. That rule was already there before I arrived, and I do not touch it. Feature extraction from high-rate channels goes at the edge, because 2GB per channel per day cannot travel. The predictive models go in the cloud, because I need to retrain them monthly across the fleet, and I am not shipping firmware to four hundred gateways to do it. The edge keeps a simple threshold fallback, so a plant that loses connectivity for a day is not blind. The uncomfortable part of this answer is fleet management of edge software, and I would budget for it explicitly.

“How do you handle forty different machine models across twelve vendors?” By modelling the asset hierarchy properly, which is the real work. Every asset gets a class defined by physics and duty, such as centrifugal pump, screw compressor, or gearbox. The class is not defined by vendor. Sensor channels get mapped at ingest to a canonical semantic name with canonical units, per vendor, in a configuration that a domain engineer maintains rather than an ML engineer. Models are trained per class. Onboarding a new vendor then becomes a mapping exercise rather than a modelling project. That is the difference between a system that scales to forty plants and one that scales to two.

“The customer wants remaining useful life in days. Give it to them?” Carefully, and only where the data supports it. RUL requires run-to-failure curves for that specific asset class and failure mode. Without them, any number you produce is an extrapolation dressed as a measurement. Where I can produce it, I produce an interval with an explicit confidence, never a point estimate, because “14 days” gets read as a guarantee and an interval does not. Where I cannot, I give a horizon bucket such as “elevated risk in the next 1–2 weeks,” which is what the planner actually needs to schedule. Saying no to a false-precision number is part of the job.

“How do you stop alert fatigue?” Structurally, not by tuning. Set a fixed alert budget per planner per week, ranked by expected cost avoided rather than by model score. Apply hysteresis, so a transient spike never fires. Deduplicate against open work orders, so one degrading asset produces one alert rather than thirty. Suppress during startup, shutdown, and scheduled maintenance windows, which are the three biggest sources of legitimate but uninteresting anomalies. And show a visible precision number in the UI, per plant. When planners can see the tool is right 60% of the time, they treat it as a 60% tool, rather than deciding it is a 0% tool after two bad calls.

“Where does the LLM go, and how do you keep it from making things up?” Only in the explanation, retrieval, and history-mining layers. Never in the prediction. In the explanation layer it is strictly constrained. It receives the numeric score, the ranked feature attributions, the asset’s service history, and retrieved manual sections, and it is instructed to describe only what it was given. Every number in the output is a number that was passed in, not one it produced. I would validate that programmatically by checking generated figures against the source payload before display. If it cannot ground a claim, it says less. And the score is always displayed alongside the prose, straight from the model, so the human’s decision anchors on the number.

“A failure happens with no alert. What’s your incident process?” The same as any production incident, with a domain-specific twist. Pull the raw edge waveform buffer for the 48 hours before the event, which is exactly why you keep it. Then establish which of three things happened. The signal was absent, meaning the failure mode is not observable with the sensors installed, which is a sensing gap and possibly unfixable. Or the signal was present but the model did not score it, which is a model gap, so add it to the golden set and retrain. Or the model scored it and the alert engine suppressed it, which is a logic bug, and it is the most fixable and most embarrassing one. Those three have completely different remedies, and conflating them is how teams retrain models to fix bugs in suppression rules.

“Can this run on the machine vendor’s own telemetry instead of your sensors?” Sometimes, and it changes the project’s economics enormously if it can. Vendor telemetry is usually cheaper and already installed. However, it is designed for warranty and diagnostics rather than prognostics, it is often coarsely sampled, and you have no control over schema changes. The strategic risk is worse than the technical one, because you are building a product on a data feed a competitor controls and can withdraw. I would use it where it exists to bootstrap, while making the ingestion layer source-agnostic, so adding your own sensors on critical assets is a configuration change.

“What’s the first thing you’d ship?” Not a model. Ship a reliable ingestion pipeline, plus a fleet dashboard showing each asset against its own historical normal band, plus the outcome-capture workflow in the CMMS. That alone catches obvious problems and earns trust with the planners. It also starts producing the labelled events that make everything afterwards possible. The anomaly detector comes second, and the supervised model third. Shipping the model first on eleven labels is how you burn the customer’s patience before you have anything to show.

Say it in one breath

Predictive maintenance is a classical machine learning problem. It is gradient-boosted trees over rolling sensor-window features, plus an unsupervised anomaly detector for the asset classes with no failure labels. That core is wrapped in a very large amount of ordinary engineering: edge feature extraction, store-and-forward ingestion, a feature store that keeps training and serving identical, and a CMMS integration that captures technician outcomes so you have labels at all. The hard parts are labelling, meaning what counts as a failure and when, then the extreme class imbalance that makes accuracy and ROC-AUC lie to you, then concept drift as machines age and duty cycles change. The language model belongs in the explanation layer, and in mining decades of free-text maintenance notes into structured labels. It belongs nowhere near the prediction, because you cannot calibrate it, cannot run it at the edge, and cannot defend it after an incident.