Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
Generating Comparison Reports
00:00 In the last lesson, we looked at how we can analyze time series data with this package. In this lesson, we will look at another feature, which is comparing two datasets together in the same report. This can be useful for a number of reasons. The first one might be that you have two datasets that are similar but from different sources.
00:18 A very typical example is a company that has multiple CRM systems. We export customer data from them and we want to compare them. And another one comes up in machine learning, where the data that you train the model on might not have the same characteristics as the data that it’s tested on in the future.
00:36
Generating a comparison report can help identify if that’s a problem. The process for generating a comparison report is as follows. You start by loading both datasets into pandas DataFrames, generate a profile report for each dataset separately, and then use the .compare() method to generate specifically a comparison report.
00:56
Let’s have a look at the code. Starting with the same code as we did before, it starts with importing the necessary libraries, creating a pandas DataFrame from the data, but here we’re going to need two different datasets.
01:09 One of the things we could do is take out some flight data for specific airports and see how those datasets compare to each other. So start by creating one for the airport with the code LAX.
01:22
We’ll save that into a variable called df_lax,
01:26
and we’re going to slice the pandas DataFrame, only take rows where the origin airport code is LAX. Save that into a DataFrame, and we’ll do the same for a second airport, which is ATL. So this is Los Angeles and Atlanta. So first we’re going to say df square bracket where df['origin'] equals LAX, and the second one is df square bracket where df['origin'] equals ATL.
01:54 We have our two DataFrames. The next step is to create two profile reports.
02:00
The first one I will call lax_profile. This will be a profile report using the df_lax DataFrame.
02:07
The second one is going to be atl_profile, which will equal the profile report of the df_atl DataFrame.
02:15
So now we have two profile report objects, and to compare them we use the .compare() method on one of them and pass in the other as the report to compare against.
02:25
So I’m going to save a variable called comparison and save it as the result of lax_profile.compare() and then pass in atl_profile.
02:36
So the first report is going to be compared against the second report, generating this comparison object, which is the comparison report from which we can call the .to_file() method and save an HTML report as in previous lessons.
02:50
I’ll call this one airport_comparison.html.
02:56 Over in the terminal, you’ll run the script again.
03:02 And this will start generating the comparison report. Now both of these datasets are slightly smaller than the raw dataset because we filtered them to specific airports, but the process does need to generate both profile reports individually and then generate the comparison report.
03:19 So it will take another minute or two to run. And now the comparison report is ready to view.
03:28 So this is now the comparison report that we have generated from the two different datasets. You can see that where we previously had a single number for some of these statistics, such as number of variables or observations, we have two columns, one for dataset A, one for dataset B.
03:46
for each variable for each column, we get the same information for both profiles separately. And we can see some of the specific differences between these datasets, such as with the origin column, which is what we filtered on.
03:59 In one of the datasets, the only value is LAX. In the second dataset, the only value is ATL, which is exactly how we filtered it in the first place.
04:09 So this lets you generate a comparison report and really dig into the differences between your datasets. In the next lesson, we will look at one more feature of the package, which is how to customize this report a little bit more.
Become a Member to join the conversation.
