The following is a soup to nuts walkthrough of how to set up and deploy a Django application, powered by Python 3, and PostgreSQL to Amazon Web Services (AWS) all while remaining sane.
Check out the Python 2 version of this article here.
Updated 08/21/2016: Updated EB global configuration settings.
Elastic Beanstalk vs EC2
Elastic Beanstalk is a Platform As A Service (PaaS) that streamlines the setup, deployment, and maintenance of your app on Amazon AWS. It’s a managed service, coupling the server (EC2), database (RDS), and your static files (S3). You can quickly deploy and manage your application, which automatically scales as your site grows. Check out the official documentation for more information.
We’ll be using a simple “Image of the Day” app, which you can grab from this repository:
1 2 3
After you download the code, create a virtualenv and install the requirements via pip:
Next, with PostgreSQL running locally, set up a new database named
iotd. Also, depending upon your local Postgres configuration, you may need to update the
DATABASES configuration in settings.py. For example, I updated the config to:
1 2 3 4 5 6 7 8 9 10
Now you can set up the database schema, create a superuser, and run the app:
1 2 3
Navigate to the admin page in your browser at http://localhost:8000/admin and add a new image, which will then be displayed on the main page.
The application isn’t meant to be very exciting; we’re just using it for demo purposes. All it does is let you upload an image via the admin interface and display that image full screen on the main page. That said, although this is a relatively basic app, it will still allow us to explore a number of “gotchas” that exist when deploying to Amazon Beanstalk and RDS.
Now that we have the site up and running on our local machine, let’s start the Amazon deployment process.
CLI for AWS Elastic Beanstalk
To work with a Amazon Elastic Beanstalk, we can use a package called awsebcli. As of this writing the latest version of is 3.7.4 and the recommended way to install it is with pip:
Now test the installation to make sure it’s working:
This should give you a nice 3.x version number:
To actually start using Elastic Beanstalk you will need an account with AWS (surprise!). Sign up (or log in).
Configure EB – initialize your app
With the AWS Elastic Beanstalk CLI working, the first thing we want to do is create a Beanstalk environment to host the application on. Run this from the project directory (“image-of-the-day”):
This will prompt you with a number of questions to help you configure your environment.
Choosing the region closest to your end users will generally provide the best performance. Check out this map if you’re unsure which to choose.
Next, it’s going to ask for your AWS credentials.
Here, you will most likely want to set up an IAM User. See this guide for how to set one up. If you do set up a new user you will need to ensure the user has the appropriate permissions. The simplest way to do this is to just add “Administrator Access” to the User. (This is probably not a great choice for security reasons, though.) For the specific policies/roles that a user needs in order to create/manage an Elastic Beanstalk application, see the link here.
This will default to the directory name. Just go with that.
Next, the CLI should automagically detect that you are using Python and just ask for confirmation. Say yes. Then you need to select a platform version. You have 2 different options here for Python 3:
- Python 3.4
- Python 3.4 (Preconfigured – Docker)
If you’re a hipster, choose the ‘Preconfigured – Docker’ choice, otherwise go with the normal ‘Python 3.4’. No, only teasing; the basic difference is this:
This gives you an EC2 image running 64bit Amazon Linux with Python 3.4 pre-installed. The front end web server is apache, with mod_wsgi installed. This is the “standard” or “traditional” way that Beanstalk works. In other words, with this option Beanstalk will create EC2 images for you, and you can use the
ebextension files we will talk about later to customize the EC2 image.
Python 3.4 (Preconfigured – Docker)
This gives you an EC2 image running Docker, with a Docker image already setup for you. The Docker image runs 64bit Debian Jessie with Python 3.4, nginx 1.8 and uWSGI 2.0.8. Because you’re basically interacting with the Docker image directly, if you choose this route you would use standard Docker configuration techniques (i.e., a ‘Dockerfile’), and then you don’t have to do much that is AWS Beanstalk specific, as Beanstalk knows how to manage the Docker image for you.
For this article we will focus on the “standard” or “traditional” way of using an EC2 image, so choose the ‘Python 3.4’ option and let’s move on.
Say yes to setting up SSH for your instances.
Next, you need to generate an RSA keypair, which will be added to your ~/.ssh folder. This keypair will also be uploaded to the EC2 public key for the region you specified in step one. This will allow you to SSH into your EC2 instance later in this tutorial.
What did we accomplish?
eb init is finished, you will see a new hidden folder called .elasticbeanstalk in your project directory:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
Inside that directory is a
config.yml file, which is a configuration file that is used to define certain parameters for your newly minted Beanstalk application.
At this point, if you type
eb console it will open up your default browser and navigate to the Elastic Beanstalk console. On the page, you should see one application (called
image-of-the-day if you’re following along exactly), but no environments.
An application represents your code application and is what
eb init created for us. With Elastic Beanstalk, an application can have multiple environments (i.e., development, testing, staging, production). It is completely up to you how you want to configure/manage these environments. For simple Django applications I like to have the development environment on my laptop, then create a test and a production environment on Beanstalk.
Let’s get a test environment set up…
Configure EB – create an environment
Coming back to the terminal, in your project directory type:
eb init, this command will prompt you with a series of questions.
You should use a similar naming convention to what Amazon suggests – e.g., application_name-env_name – especially when/if you start hosting multiple applications with AWS. I used –
DNS CNAME prefix
When you deploy an app to Elastic Beanstalk you will automatically get a domain name like xxx.elasticbeanstalk.com.
DNS CNAME prefix is what you want to be used in place of
xxx. The default probably won’t work if you’re following along because somebody else has already used it (the names are global to AWS), so pick something unique and keep on going.
What happens now?
At this point
eb will actually create your environment for you. Be patient as this can take some time.
If you do get an error creating the environment, like –
aws.auth.client.error.ARCInstanceIdentityProfileNotFoundException– check that the credentials you are using have appropriate permissions to create the Beanstalk environment, as discussed earlier in this post.
Also, it may prompt you with a message about
Platform requires a service role. If it does, just say yes and let it create the role for you.
Immediately after the environment is created,
eb will attempt to deploy your application, by copying all the code in your project directory to the new EC2 instance, running
pip install -r requirements.txt in the process.
You should see a bunch of information about the environment being set up displayed to your screen, as well as information about
eb trying to deploy. You will also see some errors. In particular you should see these lines buried somewhere in the output:
Don’t worry – It’s not really invalid. Check the logs for details:
This will grab all the recent log files from the EC2 instance and output them to your terminal. It’s a lot of information, so you may want to redirect the output to a file (
eb logs -z). Looking through the logs, you’ll see one log file named eb-activity.log:
The problem is that we tried to install
psycopy2 (the Postgres Python bindings), but we need the Postgres client drivers to be installed as well. Since they are not installed by default, we need to install them first. Let’s fix that…
Customizing the Deployment Process
eb will read custom
.config files from a folder called “.ebextensions” at the root level of your project (“image-of-the-day” directory). These
.config files allow you to install packages, run arbitrary commands and/or set environment variables. Files in the “.ebextensions” directory should conform to either
YAML syntax and are executed in alphabetical order.
The first thing we need to do is install some packages so that our
pip install command will complete successfully. To do this, let’s first create a file called .ebextensions/01_packages.config:
1 2 3 4 5
EC2 instances run Amazon Linux, which is a Redhat flavor, so we can use yum to install the packages that we need. For now, we are just going to install three packages – git, the Postgres client, and libjpeg for Pillow.
After creating that file to redeploy the application, we need to do the following:
We have to commit the changes because the deployment command
eb deploy works off the latest commit, and will thus only be aware of our file changes after we commit them to git. (Do note though that we don’t have to push; we are working from our local copy…)
As you probably guessed, the next command is:
You should now see only one error:
1 2 3 4 5
Let’s find out what’s happening…
Configuring our Python environment
EC2 instances in Beanstalk run Apache, and Apache will find our Python app according to the WSGIPATH that we have set. By default
eb assumes our wsgi file is called application.py. There are two ways of correcting this-
Option 1: Using environment-specific configuration settings
This command will open your default editor, editing a configuration file called .elasticbeanstalk/iod-test.env.yml. This file doesn’t actually exist locally;
eb pulled it down from the AWS servers and presented it to you so that you can change settings in it. If you make any changes to this pseudo-file and then save and exit,
eb will update the corresponding settings in your Beanstalk environment.
If you search for the terms ‘WSGI’ in the file, and you should find a configuration section that looks like this:
1 2 3 4 5
Update the WSGIPath:
1 2 3 4 5
And then you will have you WSGIPath set up correctly. If you then save the file and exit,
eb will update the environment configuration automatically:
1 2 3 4 5
The advantage to using the
eb config method to change settings is that you can specify different settings per environment. But you can also update settings using the same
.config files that we were using before. This will use the same settings for each environment, as the
.config files will be applied on deployment (after the settings from
eb config have been applied).
Option 2: Using global configuration settings
To use the
.config file option, let’s create a new file called /.ebextensions/02_python.config:
1 2 3 4 5 6 7 8 9 10
DJANGO_SETTINGS_MODULE: "iotd.settings"– adds the path to the settings module.
"PYTHONPATH": "/opt/python/current/app/iotd:$PYTHONPATH"– updates our
PYTHONPATHso Python can find the modules in our application. This path may vary depending on your setup! See this comment for more details. (Note that the use of the full path is necessary.)
WSGIPath: iotd/iotd/wsgi.pysets our WSGI Path.
NumThreads: 20– updates the number of processes and threads used to run our WSGI application.
"/static/": "www/static/"sets our static files path.
Again, we can do a
git commit then an
eb deploy to update these settings.
Next let’s add a database.
Configuring a Database
Try to view the deployed website:
This command will show the deployed application in your default browser. You should see a connection refused error:
1 2 3 4
This is because we haven’t set up a database yet. At this point
eb will set up your Beanstalk environment, but it doesn’t set up RDS (the database tier). We have to set that up manually.
eb console to open up the Beanstalk configuration page.
From there, do the following:
- Click the “Configuration” link.
- Scroll all the way to the bottom of the page, and then under the “Data Tier” section, click the link “create a new RDS database”.
- On the RDS setup page change the “DB Engine” to “postgres”.
- Add a “Master Username” and “Master Password”.
- Save the changes.
Beanstalk will create the RDS for you. Now we need to get our Django app to connect to the RDS. Beanstalk will help us out here by exposing a number of environment variables on the EC2 instances for us that detail how to connect to the Postgres server. So all we need to do is update our settings.py file to take advantage of those environment variables. Confirm that the
DATABASES configuration parameter reflects the following in settings.py:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
This simply says, “use the environment variable settings if present, otherwise use our default development settings.” Simple.
Handling database migrations
With our database setup, we still need to make sure that migrations run so that the database table structure is correct. We can do that by modifying .ebextensions/02_python.config and adding the following lines at the top of the file:
1 2 3 4
container_commands allow you to run arbitrary commands after the application has been deployed on the EC2 instance. Because the EC2 instance is set up using a virtual environment, we must first activate that virtual environment before running our migrate command. Also the
leader_only: true setting means, “only run this command on the first instance when deploying to multiple instances.”
Don’t forget that our application makes use of Django’s admin, so we are going to need a superuser…
Create the Admin User
createsuperuser doesn’t allow you to specify a password when using the
--noinput option, so we will have to write our own command. Fortunately, Django makes it very easy to create custom commands.
Create the file iotd/images/management/commands/createsu.py:
1 2 3 4 5 6 7 8 9
Make sure you add the appropriate
__init__.py files as well:
1 2 3 4 5
This file will allow you to run
python manage.py createsu, and it will create a superuser without prompting for a password. Feel free to expand the command to use environment variables or another means to allow you to change the password.
Once you have the command created, we can just add another command to our
container_commands section in .ebextensions/02_python.config:
1 2 3
Before you test this out, let’s make sure our static files are all put in the correct place…
Add one more command under
So the entire file looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
We now need to ensure that the
STATIC_ROOT is set correctly in the settings.py file:
Make sure you commit the
www directory to git so the static dir can be created. Then run
eb deploy again, and you should now be in business:
1 2 3 4
At this point you should be able to go to http://your_app_url/admin, log in, add an image, and then see that image displayed on the main page of your application.
Using S3 for Media Storage
With this setup, each time we deploy again, we will lose all of our uploaded images. Why? Well, when you run
eb deploy, a fresh instance is spun up for you. This is not what we want since we will then have entries in the database for the images, but no associated images. The solution is to store the media files in Amazon Simple Storage Service (Amazon S3) instead of on the EC2 instance itself.
You will need to:
- Create a bucket
- Grab your user’s ARN (Amazon Resource Name)
- Add bucket permissions
- Configure your Django app to use S3 to serve your static files
Since there are good write ups on this already, I’ll just point you to my favorite: Using Amazon S3 to store you Django Static and Media Files
Since we are using Apache with Beanstalk, we probably want to set up Apache to (among other things) enable gzip compression so files are downloaded faster by the clients. That can be done with
container_commands. Create a new file .ebextensions/03_apache.config and add the following:
1 2 3
Then you need to create the file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Doing this will enable gzip compression, which should help with the size of the files you’re downloading. You could also use the same strategy to automatically minify and combine your CSS/JS and do any other preprocessing you need to do.
Don’t forget the very helpful
eb ssh command, which will get you into the EC2 instance so you can poke around and see what’s going on. When troubleshooting, there are a few directories you should be aware of:
/opt/python– Root of where you application will end up.
/opt/python/current/app– The current application that is hosted in the environment.
/opt/python/on-deck/app– The app is initially put in on-deck and then, after all the deployment is complete, it will be moved to
current. If you are getting failures in your
container_commands, check out out the
on-deckfolder and not the
/opt/python/current/env– All the env variables that
ebwill set up for you. If you are trying to reproduce an error, you may first need to
source /opt/python/current/envto get things set up as they would be when eb deploy is running.
opt/python/run/venv– The virtual env used by your application; you will also need to run
source /opt/python/run/venv/bin/activateif you are trying to reproduce an error.
Deploying to Elastic Beanstalk can be a bit daunting at first, but once you understand where all the parts are and how things work, it’s actually pretty easy and extremely flexible. It also gives you an environment that will scale automatically as your usage grows. Hopefully by now you have enough to be dangerous! Good luck on your next Beanstalk deployment. Cheers!
Did we miss anything? Have any other tips or tricks? Please comment below.