Profiling Time Series

00:00 In this lesson, we will see how fg-data-profiling can be used to explore specifically time series data. By time series data, I mean data which contains measurements repeated over time.

00:12 In the case of our flights data, the original dataset is one row per flight that took place, but our time series is going to be one row per day, which is going to contain the number of flights that happened on each day.

00:25 Some of the additional options that fg-data-profiling supports when looking at time series include patterns that happen over time, identifying gaps in the data, which is particularly important for time series, and also statistical tests that are specific to time series data.

00:43 fg-data-profiling supports all of these options, but we need to set some things up. So in order to be able to profile time series data with this package, first of all, we need at least one column to be a date type in the dataset.

00:57 When we create the profile report, we need to tell the package which of the columns should sort the data in temporal order, and depending on whether there are existing relationships in your data, we may have to explicitly say which columns we want to be treated as time series.

01:14 Some of the options we will need to set include the tsmode parameter, which enables the time series options, the sortby parameter, which tells the package which columns are dates, and we can also specify the type_schema parameter, where we can explicitly tell the package what data type certain columns should be, in this case telling it which columns should be treated as time series.

01:38 Let’s have a look at doing this in the code. We start with a script that contains the necessary imports and the creation of the pandas DataFrame.

01:47 Before we run the time series analysis, we need to make sure we have date columns in our DataFrame. So we’re going to take the fl_date column, so that’s df['fl_date'], and we’re going to overwrite it with the output of the pd.to_datetime() function, which converts that column, which would be text, into its date representation.

02:14 Now that we have a specifically date type column, you’re going to convert the raw data, which is one row per flight, to a time series, which is one row per day, counting up the number of flights that happened each day.

02:28 I’ll call this daily_flights, and it will be the flight dataset grouped by the date

02:36 .size(), which counts up the rows in each group, and then we’re going to call .reset_index(), which will convert the index of the DataFrame into a separate column.

02:47 I’m going to name that column num_flights.

02:52 Before we create the profile report, let’s have a look at the actual dataset, the time series that we’re going to give to the package. So we will print daily_flights, let’s say .head(), so printing daily_flights.head(), which will show us the first five rows.

03:08 Now you may have noticed when I saved that script, a few parts of it changed, such as single quotes have turned to double quotes. That’s because I wasn’t, strictly speaking, sticking to the PEP 8 guidelines, but I also have a function enabled in VS Code to automatically format my scripts every time I save them.

03:25 That’s why you saw a little jump and a few small changes being made to the script just as I saved it. Over in the terminal, you’ll run this script, uv run report.py,

03:39 and this shows the format of the DataFrame you’re going to use to generate the time series report. There’s one column that represents the date and one column that represents the number of flights that took place on that date.

03:53 So this is now a time series.

03:57 I’m going to jump back to the script,

04:00 delete this line with the print() function.

04:03 I’m then going to create a profile object using the ProfileReport class,

04:08 passing in daily_flights as our data. And here is where we set the time series specific options. The first one is tsmode=True. This will enable the time series specific sections in the report.

04:22 Then the sortby option, which is going to equal fl_date, which is the name of our date column. So this makes sure that our data is sorted correctly if it isn’t already.

04:33 And finally, the type_schema parameter, which is a dictionary where we can pass names of columns as keys, and the values are the specific data type we would like the report to interpret them as.

04:48 So there is a special one called time series, where we can explicitly tell the profile report to treat that column as a time series.

04:58 And finally, you’re going to save this to a file. Call this one flight_timeseries_report.html. Go ahead and save that. And then you can run the script again.

05:23 Because this is now a much smaller and less complex dataset, this finishes much more quickly. Now let’s have a look at the generated report.

05:33 The generated report for this time series data now looks like this. First of all, there is a new time series tab at the top where we can see our time series.

05:42 So this is number of flights per day over time. And when we scroll down to look at the data about that specific time series column, we see that the report highlights it as a time series and gives us some time-specific values.

05:57 But if we click More details, you can see a lot more time series specific information. So there’s some descriptive statistics.

06:08 There’s a tab that shows a histogram of the values. The time series itself.

06:15 An analysis of whether the data contains gaps, which is very important to know for a time series. And some other more time series specific tabs such as autocorrelation.

06:25 These are plots that are specific to time series and they analyze whether there is a temporal pattern in the data.

06:33 In the next lesson, we will see another feature of this package, which is how to compare two datasets in the same report.