Yes, and no.
It can technically work. Meaning, you might be able to train a model that actually predicts something.
But if it does, this means you should just retrain the initial model instead!
Let us explain. Why can machine learning models be wrong?
Data quality aside, it is usually one of the two:
- There is not enough signal in the data the model trained on. Or not enough data. Overall, or for a specific segment where it fails. The model did not learn anything useful and now returns a weird response.
- Our model is not good enough. It is too simple to capture the signal from the data correctly. It does not know something it can potentially learn.
In the first case, model errors would have no pattern. So, any attempt to train the "watchdog" model would fail. There is nothing new to learn.
In the second case, you might be able to train a better model! A more complex one that better suits the data to capture all the patterns.
But if you can do so, why train the "watchdog"? Why not update the first model instead? It can learn from the same real-world feedback we got when applying it in the first place.