Please note: This is a collaboration piece between Michael Herman, from Real Python, and Sean Vieira, a Python developer from De Deo Designs.
Articles in this series:
- Part I: Application setup
- Part II: Setup user accounts, Templates, Static files <— CURRENT ARTICLE
- Part III: Testing (unit and integration), Debugging, and Error handling
Welcome back to the Flask-Tracking development series! For those of you who are just joining us, we are implementing a web analytics application that conforms to this napkin specification. For all those of you following along at home, you may check out today’s code with:
To quickly review, in our last article we set up a bare-bones application which enabled sites to be added and visits recorded against them via a simple web interface or over HTTP.
Today we will add users, access control, and enable users to add visits from their own websites using tracking beacons. We will also be diving into some best practices for writing templates, keeping our models and forms in sync, and handling static files.
From single to multi-package.
When last we left our application, the directory structure looked something like this:
1 2 3 4 5 6 7 8 9 10 11
To keep things clear, let’s move the existing
views into a
tracking sub-package and create another sub-package for our
User-specific functionality which we will call
1 2 3 4 5 6 7 8 9 10
This means that we will need to change our import in
from .views import tracking to
from .tracking.views import tracking.
Then there is the database setup in
tracking.models. This we will move out into the parent package (
flask_tracking) since the database manager will be shared between packages. Let’s call that module
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Then we can update
tracking.models to use
from flask_tracking.data import db and
tracking.views to use
from flask_tracking.data import db, query_to_list and we should now have a working multi-package application.
Now that we have split up our application into separate packages of related functionality, let’s start working on the
users package. Users need to be able to sign up for an account, manage their account, and log in and out. There could be more user-related functionality (especially around permissions) but to keep things clear we will stick with these basics.
We have a rule for taking on dependencies – each dependency we add must solve at least one difficult problem well. Maintaining user sessions has several interesting edge-cases which makes it an excellent candidate for a dependency. Fortunately, there is one readily available for this use case – Flask-Login. However, there is one thing that Flask-Login does not handle at all – authentication. We can use any authentication scheme we want to – from “just provide a username” to distributed authentication schemes like Persona. Let’s keep it simple and go with username and password. This means that we need to store a user’s password, which we will want to hash. Since properly hashing passwords is also a hard problem we will take on another dependency,
backports.pbkdf2 to ensure our passwords are securely hashed. (We picked pbdkdf2 because it is considered secure as of this writing and is included in Python 3.3+ – we only need it while we are running on Python 2.)
Let’s go ahead and add:
requirements.txt file and then (making sure our virtual environment is activated) we can run
pip install -r requirements.txt again to install them. (You may get some errors compiling the C speedups for pbkdf2 – you can ignore them). We will integrate it with our application in a moment – first we need to set up our Users so Flask-Login has something to work with.
We will set up our
User SQLAlchemy class in
users.models. We will only store a user’s name, email address, and (salted and hashed) password:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
Phew – almost half of this code is for the password! Even worse, by the time you are reading this our implementation of
_hash_password is likely to be considered imperfect (such is the ever-changing nature of cryptography) but it does cover all of the basic best practices:
- Always use a salt unique to each user.
- Use a key-stretching algorithm with a tunable unit of work.
- Compare hashes using a constant time algorithm.
In non-password related notes, we are making a one-to-many relationship between
sites = db.relationship('Site', backref='owner', lazy='dynamic')) so that we can have users who manage multiple sites.
In addition, we are subclassing Flask-Login’s
UserMixin class. Flask-Login requires that the
User class implement certain methods (
is_authenticated, etc.) so that it can do its work.
UserMixin provides default versions of those methods that work quite well for our purposes.
Now that we have a
User we can integrate with Flask-Login. In order to avoid circular imports we are going to setup the extension in its own top-level module named
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
@login_manager.user_loader registers our
load_user function with Flask-Login so that when a user returns after logging in Flask-Login can load the user from the user_id that it stores in Flask’s
Finally, we import
flask_tracking/__init__.py and register it with our application object:
1 2 3 4 5
Next let’s set up our view and controller functions for Users to enable register/log in/log out functionality. First, we will set up our forms:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
Again, a decent amount of code, this time mostly around validating user input. One thing to note is that for our login form, when the user is authenticated, we expose the
User instance on the form as
form.user (so we do not have to make the same query in two places – even though SQLAlchemy will do the right thing here and only hit the database once).
Finally, we can set up our views:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
And import and register them with our application object:
1 2 3 4 5 6
Notice the call to
load_user inside of our
Flask-Login requires us to call this function in order to activate our user’s session (which it will manage for us).
One last thing to look at is our
1 2 3 4 5 6 7 8 9 10
We will cover the
forms macros in a little while – the key thing to note is that for our form’s
action we are explicitly passing in the value of the
This ensures that when the user submits the form to
next parameter is available for our redirect code:
1 2 3
There’s a subtle security hole in this code, which we will be fixing in our next article (but points to you if you have already spotted it).
But wait! Did you see the pattern that we just repeated for the third time? (We actually repeated at least two patterns, but we are only going to remove the duplication from one of them today). This part of the
1 2 3 4
is also repeated multiple times in the
tracking code. Let’s pull that database session behavior out using a custom mixin which we can borrow from Flask-Kit. Open
flask_tracking/data and add the following code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
CRUDMixin provides us with an easier way of handling the four most common model operations (Create, Read, Update, and Delete):
1 2 3 4 5 6 7
Now, if we update our
User class to also subclass
1 2 3
we can then use the much clearer:
call in our views. This makes it easier to reason about what our code is doing and makes it much easier to refactor (since each piece of code deals with fewer concerns). We can also update our
tracking package’s code to make use of the same methods.
In Part I, we skipped reviewing our templates to save time. Let’s take a couple of minutes now and review the more interesting parts of what we are using to render our HTML.
That said, since we are focusing on Flask, we will use Jinja to serve up the page for now.
First, look at
layout.html (I am leaving the majority of the code out of this article to save space, but I am providing links to the full code):
1 2 3
This snippet showcases two of my favorite tricks – first, we have a block (
title) that contains a variable so we can set this value from our
render_template calls (so we don’t need to create a whole new template just to change a title). Second, we are re-using the contents of the block for our header with the special
self variable. This means, when we set
title (either in a child template or via a keyword argument to
render_template) the text we provide will show up both in the browser’s title bar and in the
The other piece of our templating structure that merits a look is our macros. For those of you coming from a Django background, Jinja’s macros are Django’s
tags on steroids. Our
form.render macro, for example, makes it incredibly easy to add a form to one of our templates:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Using it is as simple as:
1 2 3 4 5 6
Instead of writing the same form HTML over and over again we can just use
form.render to automatically generate the boilerplate HTML for each field in our forms. This way all of our forms will look and function in the same way and if we ever have to change them we only have to do it in once place. Don’t Repeat Yourself makes for very clean code.
Refactoring the tracking application
Now that we have all that set up properly, let’s go back and refactor the meat of the application: the request tracking.
In Part I, we built the skeleton of a request tracker. Sites were created on the index page and anyone could view all the available sites. As long as the end user sent all the information themselves, Flask-Tracking would store it happily. Now, we have users, so we want to filter the list of sites. Additionally, it would be good if our application could derive some of the data from the visitor, rather than asking the end user of our application to derive it all for themselves.
Let’s start with site list:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Starting from the top, the
@login_required decorator is provided by
Flask-Login. Anyone who isn’t logged in who tries to go to
/sites/ will be redirected to the login page. Next, we are checking to see if the user is currently adding a new site (
form.validate_on_submit checks to see if
request.method is POST and validates the form – if either of the preconditions fails, the method returns
False, otherwise it returns
True). If the user is creating a new site, we create a new site (using the method defined by our
CRUDMixin, so if you are making changes to the code yourself, you will want to make sure that
Visit both inherit from
CRUDMixin) and redirect back to the same page. We redirect back to ourselves after saving the new site to prevent a page refresh causing the user to attempt to add the site twice. (This is called the Post-Redirect-Get pattern).
If you are not sure what I mean by that, try commenting out the
return redirect(url_for(".view_sites")), then submit the “Add a Site” form and when the page reloads push
F5 to refresh your browser. Try that same exercise after restoring the redirect. (When the redirect is removed the browser will ask if you really want to submit the form data again – the last request that the browser made is the POST that created the new site. With the redirect, the last request that the browser made is the GET request that reloaded the
Continuing on, if the user is not creating a new site (or if the provided data has errors) we are querying our database to look up all of the sites that were created by the currently logged in user. We then slightly transform our list, turning the database ID into an HTML link for each of our non-header rows. This use of an “inline” template is good for fast prototyping, when you do not yet have a good idea of whether the template pattern is worth “macro-izing”. In our case, this is the only view we have with a table with an action link, so we use the inline template technique to demonstrate another way of doing things.
It is worth noting that we have elected to use
sites_view for both displaying sites and their visits and for registering sites. It’s really up to you how you want to break up your application. Having a
view_sites and an
add_site view, where the former is only accessible to GET requests and the latter to POST is also a valid technique. Whichever technique feels clearer to you is the one you should prefer – just make sure you are consistent.
Deriving data from visitors
add_visit, meanwhile, is now a bit more complex (although it is mostly mapping code):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
We have removed the ability for users to manually add visits from our website via a form (and so we have also removed the second route on
add_visit). We now do explicit mapping for data that we can derive on the server (the browser, the IP Address) and then we construct our
VisitForm passing in those mapped values directly. The IP address we pull from
access_route in case we are behind a proxy since then
remote_addr will contain the IP address of the last proxy, which is not what we want at all. We disable CSRF protection because we actually want users to be able to make requests to this endpoint from elsewhere. Finally, we know what site this request is for because of the
<int:site_id> parameter that we have set to the URL.
http://freegeoip.net/ so we can get a rough idea of where the requests are coming from:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Save this as
geodata.py in the
Return to the view, all this view is doing is copying info from the request down and storing it in the database. It responds to the request with an HTTP 204 (No Content) response. This tells the browser that the request succeeded, but we do not have to spend any extra time generating content that the end-user will not see.
Seeing the visits
We also add authentication to the Visits view for each individual site:
1 2 3 4 5 6 7 8 9 10
The only real change here is that if the user is logged in, but does not own the site, they will see an authorization error page, rather than being able to view the visits for the site.
Providing a means of tracking visitors
Finally, we want to provide users a snippet of code that they can place on their website that will automatically record visits:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Our snippet is very simple – when the page loads we create a new image and set its source to be our tracking URL. The browser will immediately load the image specified (which will be nothing at all) and we will record a tracking hit in our application. We also have a
Do Not Track header and only record the visit if the user has opted into tracking.)
That’s it for this post. We now have user accounts, and the beginnings of an easy to use client-side API for tracking. We still need to finalize our client-side API, style the application and add reports.
The code for the application can be found here.
Your app should now look like this:
In Part III we’ll explore writing tests for our application, logging, and debugging errors.
In Part IV we’ll do some Test Driven Development to enable our application to accept payments and display simple reports.
In Part V we will write a RESTful JSON API for others to consume.
In Part VI we will cover automating deployments (on Heroku) with Fabric and basic A/B Feature Testing.
Finally, in Part VII we will cover preserving your application for the future with documentation, code coverage and quality metric tools.