Model Management
Once a model change is capable of altering behaviour people rely on, which is immediately after the first production deployment.
Model management makes changing a model a controlled operation rather than an incident. That requires versioning, evaluation against a fixed suite, and a rollout path that can be reversed.
The evaluation suite is the part organizations skip and later wish they had, because without it there is no way to distinguish a better model from a differently-wrong one.
What the work includes
- Versioning of models and their configuration together
- Evaluation suites run before promotion between environments
- Staged rollout with the ability to roll back quickly
- Records of which version produced which output, for later review