If we face a significant drift, we might need to rethink our model.
Imagine a major external change, such as facing a completely new customer segment.
Maybe, we'll need to tune the model parameters or change its architecture? Review pre- or post-processing? Reweigh the data to give priority to the most recent examples? Build an ensemble to account for new segments?
Here, it becomes more of an art than science. The solution depends on the use case, the severity of changes, and the data scientist's judgment.
There is another side of the medal: we might be able to improve the model!
We could have started with a simpler, limited model. As more data is collected, we might be able to rebuild it and capture more complex patterns. Maybe, also add new features from other sources?
Word of caution: this new model might have a different decay profile!
To prepare for both, we need three things. First, keep the update option in mind.
If naive retraining is the only option considered, we might completely miss out on other ways to maintain the models. It makes sense to schedule some regular time to review existing models for the improvement potential. Second, build up a model analytics practice.
What do we mean by this? Get more visibility into your model behavior!
Is not just a singular performance metric like ROC AUC
. We can search for underperforming segments, changes in the data, feature correlations, patterns in the model errors, and more.