The question that sorts people who have run models in production from people who have only read about it is short: your offline evaluation looked great, the model shipped, and three weeks later the predictions are quietly worse. Nobody filed a bug. What do you check first, and in what order?
Interviewers for ML platform and MLOps roles ask some version of this constantly, because the answer shows whether you treat a model as a pipeline or as a file. The failure is almost never the weights. It is the data feeding them, the way features get computed, or the gap between how you built the thing and how it actually gets called at serving time.
Drift is three separate problems wearing one name
When a candidate says “the model drifted,” a good interviewer will ask which kind, because the diagnoses and fixes share almost nothing.
Feature drift, sometimes called covariate shift, is when the input distribution moves. A signup form starts collecting a new country, a currency changes, an upstream team swaps miles for kilometers. You can catch this without any labels by comparing the live feature distribution against the training window. The two workhorse tests are the population stability index and the Kolmogorov-Smirnov test. PSI under 0.1 means no real change, 0.1 to 0.25 is a moderate shift worth watching, above 0.25 is a serious move that usually warrants action. The KS test gives you a p-value on whether two samples came from the same distribution, which is handy for continuous features but gets noisy on large samples where tiny, harmless differences turn statistically significant.
Concept drift is the harder one. Here the inputs may look identical, but the relationship between inputs and the target has changed. A fraud pattern evolves, user behavior shifts after a pricing change, and the same feature values now imply a different outcome. You cannot see this from input distributions alone. You need labels, and labels almost always lag, sometimes by days or weeks. That lag is the real trap in the interview: if you say “I’ll monitor accuracy in production,” the follow-up is “with what labels, and how late do they arrive?” A strong answer names a label pipeline and a plan for the window where you are flying without ground truth.
Prediction drift sits in the middle. You watch the distribution of the model’s own output scores over time. It is cheap, it needs no labels, and it often moves first. The catch is that a shift in output can be perfectly benign, driven by seasonality or a legitimate change in traffic mix, so it is an early warning to investigate rather than proof of a problem.
Naming the tools helps here. Evidently and NannyML are the two most common open-source monitoring libraries people mention. NannyML is worth knowing for one specific reason: it estimates model performance before labels land, using methods like confidence-based performance estimation, which directly addresses the label-lag gap interviewers probe. For data quality checks upstream of any of this, Great Expectations and schema validation catch nulls, range violations, and type changes that would otherwise poison the features silently.
| Monitoring signal | What it compares | Common test or method | Typical alert point | What it will not tell you |
|---|---|---|---|---|
| Feature drift (covariate shift) | Live input distribution vs training window | PSI, Kolmogorov-Smirnov | PSI above 0.25, or KS p below 0.05 | That the model actually got less accurate |
| Prediction drift | Distribution of output scores now vs baseline | PSI on scores | PSI above roughly 0.2 | Whether the shift is a real problem or seasonality |
| Concept drift | Relationship between inputs and target | Performance drop once labels arrive; DDM, ADWIN on the error stream | Accuracy or AUC below the agreed SLO | Anything until ground-truth labels show up |
| Training/serving skew | Same feature computed offline vs online | Value-level diff on logged features | Any mismatch above tolerance | Nothing visible in distributions; the model is just wrong |
| Data quality | Nulls, ranges, schema, freshness | Great Expectations, schema checks | Null rate or out-of-range above baseline | Whether semantically valid data still means the same thing |
The skew that never shows up on a dashboard
Training/serving skew deserves its own section because it is the bug that most often causes the “great offline, bad online” story, and it does not register as drift at all. The distributions look fine. The model is simply being fed a feature that was computed one way during training and a different way at serving time.
A concrete version: during training you compute a user’s seven-day average order value from a data warehouse, using a clean window with all events present. At serving time you compute the same feature from a real-time store that is missing the last few hours of events because of ingestion lag, or that uses a slightly different time boundary. Same feature name, different number. The model was trained on one and now sees the other, and it degrades in ways no distribution test will flag.
This is the problem a feature store is built to solve, and it is why “what is a feature store” is really two questions in disguise. The shallow answer is that it is a central place to store and reuse features. The answer an interviewer wants covers the split between the offline store and the online store, and how a feature store keeps the two in sync. The offline store holds historical feature values for training, usually in a warehouse like BigQuery or Snowflake. The online store holds the latest value per entity for low-latency lookups at serving time, backed by something like Redis or DynamoDB. Feast is the common open-source reference; Tecton is the managed option people reach for when they want the transformation layer handled for them alongside storage and serving.
Two ideas make or break the answer. The first is point-in-time correctness. When you build a training set, each row must join feature values as they existed at the moment of that event, not their current value, or you leak future information into the past and your offline metrics become fiction. Feature stores implement this with point-in-time joins for exactly that reason. The second is a single transformation path: the same code that materializes features into the offline store should populate the online store, so there is no second implementation to drift away from the first. If a candidate can explain point-in-time joins and a shared transformation path, they have understood the hard part.
What to do when drift fires
Once you have detection, the natural next question is retraining, and it is a trap worth walking into carefully. “I detect drift, so I retrain automatically” sounds responsible and is often wrong. Drift firing means the inputs moved, not that a fresh model will be better. Retrain blindly on the drifted data and you can promote a model that is worse, or one trained on a transient anomaly like a holiday spike or a logging outage.
The version that holds up under questioning uses drift as a trigger, not an action. Drift or a performance dip kicks off a retraining run, the candidate model gets evaluated against a held-out set and compared to the current production model in a champion-challenger setup, and it only ships if it clears a bar. Layered on top is a maximum staleness fallback, so a model that has gone too long without a refresh gets retrained on a schedule even if nothing tripped an alarm. That combination, a drift trigger plus a validation gate plus a time-based backstop, is the shape of a good answer.
Deployment mechanics come up in the same breath. Shadow deployment runs the new model on live traffic while logging its predictions and acting on none of them, which lets you compare against the incumbent with zero user risk. Canary sends a small slice of real traffic to the new model and watches the metrics before widening. Blue-green keeps two full environments so a rollback is a traffic switch rather than a redeploy. The point an interviewer listens for is that you always have a fast path back. A model rollback should be as boring as a config change.
How the loop is usually structured
An MLOps or ML platform loop tends to run four or five rounds. There is a system design round where you design a serving or training platform end to end, a coding round that is usually standard data-structures work or a pipeline-flavored problem, a round on ML fundamentals to confirm you actually understand the models you are shipping, and a behavioral or debugging round built around a production incident. The questions get phrased in ways like these:
- “Accuracy is fine offline but dropping in production. Walk me through your investigation, step by step.”
- “A feature has different values in training and at serving time. How does that happen, and how would you catch it before it hurts you?”
- “You see drift on an input feature. Do you retrain? What has to be true before you promote the new model?”
- “Design monitoring for a model whose labels arrive two weeks after prediction.”
Notice how little of that is about model architecture. The people who do well have usually been paged at 3am for a pipeline that broke in a boring way, and they answer from that memory rather than from a textbook. If you have not lived it, the fastest way to sound like you have is to build a tiny end-to-end project: a model behind an API, a feature computed in two places on purpose, and a drift check that catches the mismatch you planted. The moment you watch your own skew bug fool your own offline metrics, every question above stops being abstract.
