Evidently 0.1.17: Meet JSON Profiles, an easy way to integrate Evidently in your prediction pipelines
June 4, 2021
We are excited to announce our latest release!
TL;DR: Now, you can use Evidently to generate JSON profiles. It makes it easy to integrate the calculated metrics and statistical test results into external pipelines and visualization tools.
TL;DR: Now, you can use Evidently to generate JSON profiles. It makes it easy to integrate the calculated metrics and statistical test results into external pipelines and visualization tools.
What is it?
Previously, you could use Evidently to generate interactive visual reports, displayed in the Jupyter notebook or available as an HTML file.
The reports look great—and help debug and explore the model behavior in detail. But many users asked for a more lightweight alternative, too.
Sometimes you do not want to look at the plots but instead, get a quick summary of the performance.
Now, you have this option!
Meet the JSON profiles.
The reports look great—and help debug and explore the model behavior in detail. But many users asked for a more lightweight alternative, too.
Sometimes you do not want to look at the plots but instead, get a quick summary of the performance.
Now, you have this option!
Meet the JSON profiles.
How does it work?
Once you prepare your data, you can calculate the results of the analysis and choose to generate the profile instead of the dashboard. You can select which sections to include.
Here is an example. That is what we need to get a summary of the Data and Categorical Target Drift on the Iris dataset:
Here is an example. That is what we need to get a summary of the Data and Categorical Target Drift on the Iris dataset:
iris_target_and_data_drift_profile = Profile(sections=[DataDriftProfileSection, CatTargetDriftProfileSection])
iris_target_and_data_drift_profile.calculate(iris_frame[:75], iris_frame[75:], column_mapping = None)
iris_target_and_data_drift_profile.json() You can include several analyses in a single JSON output. You specify each as a section in a profile: just like you choose tabs in the visual dashboards.
Right now, we can choose among the six sections:
The same rules apply to the data requirements as they do for the dashboards. Similarly, you can generate the profiles from the Jupyter notebook or directly from the Terminal.
The output you receive is a JSON summary of all metrics, statistical tests, and quick histogram previews.
Check out a notebook example to have a look!
Right now, we can choose among the six sections:
- data_drift to estimate the data drift,
- num_target_drift to estimate target drift for numerical target,
- cat_target_drift to estimate target drift for categorical target,
- classification_performance to explore the performance of a classification model,
- prob_classification_performance to explore the performance of a probabilistic classification model,
- regression_performance to explore the performance of a regression model.
The same rules apply to the data requirements as they do for the dashboards. Similarly, you can generate the profiles from the Jupyter notebook or directly from the Terminal.
The output you receive is a JSON summary of all metrics, statistical tests, and quick histogram previews.
Check out a notebook example to have a look!
When should I use the profiles?
Whenever you prefer to get your output in text instead of an interactive dashboard.
It can help you to:
1. Get a quick look at metrics.
Some users work with Evidently during the model development phase. You might want to check the model performance or historical data drift quickly. If you need just a couple of numbers, you might prefer a JSON profile output. No need to load the complete report.
2. Send the calculation results elsewhere.
You might be using some external visualization solution, like Grafana, but rely on Evidently to calculate complex metrics like drift. You can generate the JSON profiles (for example, as a scheduled batch job), select the metrics you need and then send them to be displayed in your preferred tool. If something goes wrong with metrics, you can get back to Evidently to generate an interactive report to troubleshoot.
3. Integrate Evidently into the prediction pipeline.
You might want to include certain analyses in your prediction pipeline. For example, to compare performance results with the past or to calculate and return a data drift check. Now you can flexibly do so. The combination of Python code and JSON output makes it easy to plug into a variety of environments.
It can help you to:
1. Get a quick look at metrics.
Some users work with Evidently during the model development phase. You might want to check the model performance or historical data drift quickly. If you need just a couple of numbers, you might prefer a JSON profile output. No need to load the complete report.
2. Send the calculation results elsewhere.
You might be using some external visualization solution, like Grafana, but rely on Evidently to calculate complex metrics like drift. You can generate the JSON profiles (for example, as a scheduled batch job), select the metrics you need and then send them to be displayed in your preferred tool. If something goes wrong with metrics, you can get back to Evidently to generate an interactive report to troubleshoot.
3. Integrate Evidently into the prediction pipeline.
You might want to include certain analyses in your prediction pipeline. For example, to compare performance results with the past or to calculate and return a data drift check. Now you can flexibly do so. The combination of Python code and JSON output makes it easy to plug into a variety of environments.
Did anything else change?
We introduced a few updates to make the tool work faster and more reliably.
In particular, now you need the "calculate" step before you choose to display a dashboard or a profile. Bear this in mind as you update to the new version!
In particular, now you need the "calculate" step before you choose to display a dashboard or a profile. Bear this in mind as you update to the new version!
regression_model_performance = Dashboard(tabs=[RegressionPerfomanceTab])
regression_model_performance.calculate(reference_data, current_data, column_mapping = column_mapping)Evidently also became faster, and we fixed a bunch of bugs!
Thanks to everyone who opens the issues on GitHub, or reports those in our Discord chat!
Thanks to everyone who opens the issues on GitHub, or reports those in our Discord chat!
Sounds good. How can I try it?
Go to Github, pip install evidently, and give it a spin! And a star, if you like it!
If you want more detailed documentation, here it is.
If you want more detailed documentation, here it is.
For any questions, contact us via hello@evidentlyai.com. That's an early release, so let us know of any bugs! You can also open an issue on Github.
Want to stay in the loop?
Want to stay in the loop?
- Sign up if you want to receive our news and product updates.
- Follow on Twitter and Linkedin for more content on ML in production.
- Join our Discord community to chat and connect.
Co-founder and CTO
Co-founder and CEO