Here is another option for those lucky to have the labels.
If you face a major change, naive retraining is often not enough. We've
discussed this difference in our past blogs.
With retraining, you repeat the existing model training pipeline. You keep the feature engineering steps, model type, hyperparameters, and so on. You only change the training data.
You can add a new bunch of data to the existing dataset or re-run the pipeline on the new dataset only. Or do some combination, mixing old and new. Side note: you can
test it in advance to choose what's best for your use case.
Say that is what we did. We "replayed" the model training. But the updated model's performance does not look good on the validation set.
To rebuild or calibrate the model, you can make more changes to the training pipeline.
You can re-run the model experiments from scratch, testing new model architectures, parameters, or feature processing approaches.
You might also test some more specific ideas to combat drift:
- Reweigh samples in the training data, giving more importance to the recent ones. The goal is to make the model give priority to newer patterns.
- Identify new segments where the model fails, and create a different model for it. Consider using an ensemble of several models for different segments of the data.
- Change the prediction target. For example, switch from weekly to daily forecast or replace the regression model with classification into categories from "high" to "low."
- Pick a different model architecture to account for ongoing drift. You can consider incremental or online learning, where the model continuously adapts to new data.
- Apply domain adaptation strategies. There is a number of approaches to help the model better generalize to a new target domain.
As usual with data science, there are no guarantees.
Sometimes the major concept drift breaks the model entirely. Imagine you work on quality prediction on the manufacturing line, and it just got a revamp. That makes the model learnings obsolete.
You might need to wait for more new data to be collected until you can restore the machine learning system.
What other options do you have if machine learning does not work?
You can consider a non-ML solution. Something else robust: based on the understanding of the causal relationships, physics of the process, or expert knowledge.