Easiest way to Detect Data Drift in your dataset using Evidently in Python

Snapshot of our Final Report…

Step 1 — Importing required Packages

  • Importing Pandas to read our CSV dataset.
  • Importing Evidently library to create interactive Data Drift Dashboards.
import pandas as pd
from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab

Step 2 — Reading the Data

df = pd.read_csv('UCI_Credit_Card.csv')
print(df.columns)

Step 3 — Creating a Data Drift report

  • Create a Dashboard object and pass DataDriftTab as the parameter.
  • Then calculate the Data Drift using the calculate method which takes two data frames to compare data distributions.
  • Then we are simply just saving the Dashboard in the HTML format.
credit_data_drift_dashboard = Dashboard(tabs=[DataDriftTab(verbose_level=1)])
credit_data_drift_dashboard.calculate(df[:25000], df[25000:], column_mapping=None)
credit_data_drift_dashboard.save('DataDrift.html')
print('Data Drift saved')

Our Final Report

  • The image below shows the final view of our Dashboard.
  • Let’s observe the BILL_AMT_4 column.
  • The first column says that our BILL_AMT_4 is of numeric type.
  • Then the next two columns display both, the reference distribution and the current distribution. We can observe the difference between the two.
  • Then in the last column, we can see the p-value for the similarity test. You can set your own p-value threshold in the code above.

Let’s open the BILL_AMT_4 field

Let’s see the full code…

import pandas as pd
from evidently.dashboard.tabs import DataDriftTab
from evidently.dashboard import Dashboard
df = pd.read_csv('UCI_Credit_Card.csv')
print(df.columns)
credit_data_drift_dashboard = Dashboard(tabs=[DataDriftTab(verbose_level=1)])
credit_data_drift_dashboard.calculate(df[:25000], df[25000:], column_mapping=None)
credit_data_drift_dashboard.save('DataDrift.html')
print('Data Drift saved')

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abhishek Sharma

Abhishek Sharma

Data Scientist || Blogger || machinelearningprojects.net || Contact me for freelance projects on asharma70420@gmail.com