Interpreting a Report

00:00 In the last lesson, you wrote a few lines of code to generate a profile report of a new dataset. In this lesson, we’ll walk through one of these reports and see exactly what kind of information it contains. What you’ll see is that the report shows an overview of your columns, a breakdown of the individual columns and the different values inside them, how the columns interact with each other, whether there are any correlations in your data, and things like missing values, and even a sample of some of your rows.

00:31 Let’s take a look. When you open one of these profile reports in your browser, it looks like this. It starts with an overview of some dataset-level statistics such as the number of columns, number of observations, that’s the number of rows, what percentage of rows are missing, and so on.

00:48 It also gives a summary of data types. Below that, you can also see a list of all the different variables, that means all the columns in your data.

00:58 Depending on the data type, it will give you different statistics. So the year column is categorical, which means there’s only a distinct set of values, so it even says how many distinct values there are, whether there’s any missing, and so on. As it happens, this dataset is all 2024.

01:18 Another column is the month, which is a real number. And because it’s a different data type, you get a different graph and you get different statistics, things like minimum and maximum, which we didn’t see for the categorical

01:31 column. As I scroll down, you’ll see that there are date columns, again, slightly different statistics, including whether any of the dates are invalid.

01:43 And what you’ll also see is for text columns, it generates a little word cloud you can look at with word frequencies.

01:51 In each of these columns, you can click the More details button and get even more information at a column level.

02:05 If we go further down, the next section of the report shows interactions between your variables. Specifically, that looks at a relationship between two variables exactly. And you can select which two variables you’d like to examine.

02:19 For example, you can put distance on the x-axis and the actual elapsed time of the flight on the y-axis. The chart on the right therefore changes.

02:31 And in this case, it shows that distance and the elapsed time of the flight are very strongly correlated, meaning the longer a flight goes, the longer it takes, which is pretty intuitive.

02:41 So this section deals specifically with the relationship between two variables. But if you wanted to look at all the relationships in your data at a glance, the next section, Correlations, shows all the columns against all the columns, showing a specific correlation value between minus one and plus one for all combinations of columns.

03:02 So you can immediately identify hotspots where things are either strongly positively correlated or negatively correlated.

03:12 There’s a section dedicated to missing values. For example, here we can see that the cancellation _code column has very little data, probably because most flights are not cancelled.

03:24 And finally, at the bottom, you’ll also get a sample of rows to remind you what the raw data looks like. At the top of the report is a navigation bar. I can click these sections to jump around in the report because it is quite long.

03:38 And at the top, there’s also a section called Alerts. Now the Alerts section gives you warning messages about your data. These aren’t really errors, but they are suggestions of where you might want to explore the data further.

03:52 So for example, there’s a lot of correlation between our data, which may or may not be of interest. It also tells us that the year column has just a single value in the entire column.

04:04 And you’ll see other potential data quality issues that you can investigate, such as missing data

04:12 or data that contains lots of zeros.

04:16 So as you can see, this profile report is very comprehensive and it saves you a lot of typing boilerplate code that you’d have to write to generate the same amount of analysis.

04:26 This HTML format is not the only way to look at the profile report. In the next lesson, we’ll look at how to view the same report directly in a Jupyter Notebook.