Python Celery Basics

00:01 Celery basics.

00:03 Celery is a distributed task queue that can collect, record, schedule, and perform tasks outside of your main program. There are two main reasons why most developers want to start using Celery.

00:16 Firstly, offloading work from your app to distributed processes that can run independently of the app. Secondly, scheduling task execution at a specific time.

00:26 Sometimes as recurring events, Celery is an excellent choice for both of these use cases. It defines itself as a task queue with focus on real-time processing, while also supporting task scheduling.

00:41 Even though both of these functionalities are part of Celery, they’re often addressed separately. Celery workers are worker processes that run tasks independently from one another and outside the context of your main service.

00:54 Celery Beat is a scheduler that orchestrates when to run tasks. You can use it to schedule periodic tasks as well.

01:03 Celery workers are the backbone of Celery. Even if you aim to schedule recurring tasks using Celery Beat, a Celery worker will pick up the instructions and handle them at the scheduled time.

01:15 What Celery Beat adds to the mix is a time-based scheduler for Celery workers.

01:21 In this course, you’ll learn how to integrate Celery with Django to perform operations asynchronously from the main execution thread of the app using Celery workers.

01:31 You won’t tackle tasks scheduling with Celery Beat in this course, but once you understand the basics of Celery tasks, you’ll be well equipped to set up periodic tasks with Celery Beat.

01:42 You can see more about that in the documentation links seen on screen.

01:48 Celery isn’t only useful for web applications, but it’s certainly popular in that context. That’s because you can efficiently tackle some everyday situations in web development by using a distributed task queue such as Celery.

02:01 You may want to send an email verification, a password reset email, or a confirmation of a form submission. Sending emails can take a while and slow down the app, especially if you have many users.

02:14 You might want to resize avatar images that users upload or apply some encoding on all images that users can share on the platform. Image processing is often a resource-intensive task that can slow down the app, particularly if you’re serving a large community of users.

02:30 If you allow users to add data to your app, then you might want to monitor their input. For example, you may want to check for profanity in comments or translate user-submitted text to a different language. Handling all this work in the context of the web app can significantly impair performance.

02:47 If you need to make web requests to provide the service that your app offers, you can quickly run into unexpected wait times. This is true for rate-limited API requests just as much as other tasks such as web scraping.

02:59 It’s often better to hand off these requests to a different process.

03:04 Crunching data is notoriously resource-intensive. If your web app analyzes data for your users, you’ll quickly see your app become unresponsive. If you’re handling all the work within Django, just as with any other data analysis, waiting for the results of machine learning operations can take some time.

03:22 Instead of letting your users wait for the calculations to complete, you can offload that work to Celery so they can continue using the app until the results come back.

03:32 If you are serving an app that allows users to generate reports from day-to-day provided data, you’ll notice that building PDF files doesn’t happen instantly.

03:40 It will be a better user experience if you let Celery handle that in the background instead of freezing the web app until the report is ready for download.

03:49 The main setup for all these different use cases will be similar. As soon as you understand how to hand off compute or time-intensive processes to a distributed task queue, you’ll free up Django to handle the HTTP request-response cycle.

04:04 In this course, you’ll be handling the email sending scenario. You’ll start with a project in which Django handles the email sending synchronously, and you’ll test to see how that freezes your Django app.

04:15 Then you’ll learn how to offload the task to Celery so you can experience how that will make your web app respond much more quickly.

04:23 In this course, you’ll be focusing on using Celery on Unix systems. So if you’re trying to set up a distributed task queue on Windows, then this might not be the right course for you.

04:33 Celery dropped support for Windows in version four, so while you may still be able to get it to work for Windows, you’ll be better off using a different task queue such as Huey or Dramatiq.

04:43 Instead, to receive tasks from your program and send results to a backend, Celery requires a message broker for communication. Redis and RabbitMQ are two message brokers that developers often use together with Celery.

04:59 In this course, you’ll be using Redis as the message broker after completing the course. If you want to challenge yourself, you can try using RabbitMQ as a message broker instead.

05:11 If you want to keep track of the results of your task runs, then you’ll also need to set up a results backend database. Connecting Celery to a results backend is optional.

05:20 Once you instruct Celery to run a task, it will do its duty whether you keep track of the task result or not. But keeping a record of all task results is often helpful, especially if you’re distributing tasks to multiple queues.

05:34 You can use many different databases to keep track of Celery task results. In this course, you’ll work with Redis both as a message broker and as a results backend.

05:44 By using Redis, you limit the dependencies you need to install because it can take on both roles. You won’t be doing any work with the recorded task results in the scope of this course.

05:54 But as a next step, you could inspect results with the Redis command line interface or pull information into a dedicated page in your Django project.

06:04 So now you’ve seen in more detail how Celery can help you. In the next section of the course, you’ll get started on that journey by looking at how to integrate Celery with Django.