Remember: Here’s what we’re building – A Flask app that calculates word-frequency pairs based on the text from a given URL.
- Part One: Set up a local development environment and then deploy both a staging and a production environment on Heroku.
- Part Two: Set up a PostgreSQL database along with SQLAlchemy and Alembic to handle migrations.
- Part Three: Add in the back-end logic to scrape and then process the word counts from a webpage using the requests, BeautifulSoup, and Natural Language Toolkit (NLTK) libraries.
- Part Four: Implement a Redis task queue to handle the text processing.
- Part Five: Set up Angular on the front-end to continuously poll the back-end to see if the request is done processing.
- Part Six: Push to the staging server on Heroku – setting up Redis and detailing how to run two processes (web and worker) on a single Dyno.
- Part Seven: Update the front-end to make it more user-friendly.
Need the code? Grab it from the repo.
Let’s look at what we currently have…
Current User Interface
Start Redis in a terminal window:
Then get your process worker going in another window:
1 2 3 4 5
Finally, in a third window, fire up the app:
You should see your word counter working. Now we can add in a custom Angular Directive to display the results in a D3 chart.
Start by adding the D3 library (v3) to the index.html file:
1 2 3 4 5 6
Now let’s set up a new custom Directive.
Angular Directives are markers on a DOM element, which allow us to insert sections of HTML with specific events and attributes attached to it. Let’s build out the first part of our Directive by adding the following code just below the controller in main.js:
1 2 3 4 5 6 7 8
restrict: 'E' creates a Directive that is restricted to an HTML element.
replace: true simply replaces the HTML Directive with the HTML in the
link function gives us access to variables in the scope defined in the controller.
Next, add a
watch function to “watch” for any changes to the variables and respond appropriately. Add this to the
link function like so:
1 2 3 4 5
Finally, add the Directive just below the closing divider to
With the Directive set up, let’s turn our attention to the D3 library…
D3 Bar Chart
Step 1: Functional Logic
Add the following to the
watch function within the Angular Directive:
1 2 3 4 5 6 7 8 9 10 11
scope.wordcounts changes, this function is fired, which updates the DOM. Since an object is returned from the AJAX request, we iterate through it to add the specific data to the chart. Essentially, every word is appended to a new
div via a data join.
Try running the code.
What happens? Nothing shows up, right? Check out the DOM in Chrome’s Developer Tools, after you submit a new site. You should see a number of nested
divs. We just need to add styles…
Step 2: Styling the Bar Chart
Start with some simple CSS:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Make sure to include this at the top of HTML page, after the Bootstrap stylesheet:
Fire up the app in our browser. What’s happening now?
When you search for a website, you should now see a grey area with some thin blue bars on the left hand side. So you can see that we are generating a bar for each data element we’re getting back – 10 in total. However, we need to modify our D3 code in order to increase the width of each bar so they are readable.
Step 3: Making the Bar Chart more Interactive
We can chain this on to our existing code and use the D3 style function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Now we are dynamically creating a width based on the numeric value of how often a word shows up on a webpage:
1 2 3 4 5 6
The style is calculated by returning the value associated with each word, multiplying that number by 20, and then converting it into pixels. We can also add text to each bar element by inserting the string value of the word along with how often it shows up on the page.
Try this out. You should see something like:
There’s still one thing missing though. What happens when you search for a new website? Try it. The new chart is appended beneath the previous one. We need to clear out our chart div before a new one is created.
Step 4: Clean Up for the Next URL Search
link function in the Directive:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
d3.select('#chart').selectAll('*').remove(); simply clears out the chart each time the
$scope.watch function is fired. Now we have a chart that is cleared before each new use, and we have a fully functional word count application!!
Test it out!
Conclusion and Next Steps
That’s it. Push your changes to the staging and production servers. Let’s review what we tackled:
- We started with the configuration and workflow, setting up staging and production servers
- From there, we added the basic functionality – web scraping, data analsyis – and set up a task queue with Redis
- With the back-end functionality set up, attention turned to the front-end where we added Angular, built a custom Directive, and added D3 into the mix
We have an MVP, but there’s still much to be done:
- Refactor, refactor, refactor!
- Write tests
- Handle errors and exceptions
- Abstract out state in the Angular app to a Service
- Work on the UI and UX
Want to help? Add a feature, write part 9, get paid, and become Internet famous!