Today I’m speaking to Brett Slatkin, a principal software engineer at Google and the author of the Python programming book Effective Python. Join us as we discuss Brett’s experience working with Python at Google, refactoring, and the challenges he faced when writing the second edition of his book. Without any further ado, let’s get into it!
Ricky: Welcome to Real Python, Brett. I’m glad you could join me for this interview. Let’s start the same way we do with all our guests. How’d you get into programming, and when did you start using Python?
Brett: Thanks for having me! My path to programming was long and varied. Python was a surprising twist at the end of my education. It was something that I never expected, but it was a blessing. Once I learned the language, I fell in love with it.
When I was growing up, my family had computers in our house, which was an enormous privilege. I always loved taking things apart to see how they worked, and I don’t remember a time when I wasn’t using computers and trying to figure them out. One of my late grandfather’s favorite stories was about the time he was having a printing problem and I fixed it for him. (I was 3 years old.)
In the 5th grade, my whole class went to a computer lab one day and programmed with Logo on an Apple IIgs. I’m lucky that my mom encouraged my interest in programming after that by buying me books and enrolling me in nerdy summer camps. I used Lego Technic Control Center kits to program small machines. These were the precursor to today’s Lego Mindstorms.
I also used Borland Delphi to build simple GUI apps, like guess the number games. Then, the staff at my local internet cafe showed me Unix, Linux, and C, which blew my mind. That’s when I knew I wanted to be a programmer when I grew up.
During high school, I took free classes at my local community college and studied C, C++, x86 assembly, and game development. I read Scott Meyers’ second edition of Effective C++ and became a huge fan of C++ and all of its arcane features. One summer, I had an unpaid internship with a dot-com bubble startup where I wrote web apps in ASP and VBScript. I was most excited about network programming and tried to build my own versions of DikuMUD and Hotline.
In college, I majored in computer engineering because I wanted to understand how computers worked at the transistor level. Most of my coursework was electrical engineering, where I used programming tools like SPICE, VHDL, and MATLAB. For computer science classes, I learned LISP and wrote a lot of Java. I was a teaching assistant for a variety of classes and really enjoyed teaching others.
Finally, my first encounter with Python was in 2003 when I was looking into how the BitTorrent client worked. (It was open source back then.) I thought the code was ugly because of all the __dunder__ special methods, and I ignored the language for all of my projects!
I went to Google in 2005 after graduating from college because it seemed like the best place to learn more about network programming. But in those earlier days of the company, you didn’t get to pick what you worked on as a new hire. You were just assigned to a team.
On my first day, my boss gave me a copy of the book Python in a Nutshell and told me to go fix a ~25KLOC codebase that was broken. I thought that in the job I would be writing C++ for networked systems, but I ended up having to do something completely different. What made matters worse was that I was on my own because the original programmer had gone on mandatory vacation due to burnout!
Luckily, Alex Martelli was a colleague on the team I joined. He was the author of the book Python in a Nutshell and he helped me learn the language. I was able to rewrite most of the codebase I inherited, fix its fundamental issues, and scale it to Google’s entire production fleet of machines. Believe it or not, this code is still in use today, nearly 15 years later.
The productivity I gained from using Python was dramatic compared to my experience with C++ and Java. My perspective on programming and languages had completely changed. I was now excited about Python!
Ricky: You’ve already touched on using Python at Google, where you’re now a principal software engineer, but in your tenure, you launched Google’s first cloud product, the App Engine. Now you’re the technical lead on Google Surveys, among other projects. How instrumental was Python in helping you develop these, and what does the future of Python look like at Google?
Brett: There’s no doubt that I owe my career at Google to Python, but there’s more to the story. As background, it’s important for you to know that the five engineers who founded App Engine were originally using JavaScript, similar to Netscape’s Livewire server from 1995. However, this was before the release of Chrome and the large performance improvements of V8, which led to NodeJS.
The founders were concerned that server-side JavaScript would make the system seem too niche. They wanted to provide the LAMP stack, which was very popular at the time. The “P” in LAMP stood for Perl, PHP, or Python. Perl was generally on its way out of style, and PHP wasn’t a language Google used internally. But Python was, thanks to Peter Norvig, and so it was the natural choice.
I started on the App Engine team with a 20% project in late 2006 when I implemented the dev_appserver in Python. Before that, there was no way to locally run an app before deploying it. I was so excited by the product’s potential and its use of Python that I moved over to work on it full-time shortly after.
Guido van Rossum, who also worked at Google then, soon joined us. I had the exciting privilege of doing the live demo during the launch event in 2008, where I wrote a web app in Python from scratch and deployed it in just a few minutes. I collaborated with Guido on a variety of projects over the years, the most interesting of which was ndb (the precursor to asyncio) that I beta tested extensively.
What I enjoyed the most was writing full applications with Python in the App Engine environment to push the product’s limits. I would frequently break the system with my demo apps by hitting new barriers that we didn’t realize we had. This would lead us to build infrastructure improvements to ensure that the next app that needed to scale, such as Snapchat, wouldn’t hit the same issues.
My favorite demo app was the PubSubHubbub protocol and hub that Brad Fitzpatrick and I developed together. The protocol turned RSS feeds into real-time streams. It brought the liveliness of services like Twitter to the open web.
At its height, our hub app was processing millions of feeds and thousands of requests per second, all using Python on App Engine. Since then, the hub has been integrated into Google’s standard web crawling infrastructure. PubSubHubbub (now called WebSub) became part of a larger group of specifications that made up OStatus, which helped get Mastadon off the ground.
Google Surveys started as another Python demo app on App Engine. I prototyped the whole system end-to-end in just a few weeks. My co-founder and I had the chance to show it off to Larry Page, and he greenlit the project. We launched a year later with a lean team, thanks to the leverage that Python provides.
Our services now handle 500K+ requests per second and impact the majority of Google’s users and revenue. The codebase has grown from ~10KLOC to over 1MLOC, with the majority of it being in Python. We’ve had to migrate some services over to Go, C++, and Java as we’ve hit various CPU cost and latency constraints, but Python is still at the core of everything we do.
Python has found its place at Google primarily as the tool for data science, machine learning, and DevOps / SRE. Right now, engineers across the company are working towards deprecating Python 2 and moving the whole monorepo over to Python 3 to stay in line with the Python community and open source packages.
Ricky: On top of what you just talked about, you’re also an author of the popular Python programming book, Effective Python, which we previously reviewed in The Best Python Books. You’ve recently published a second edition, which has substantial updates. What do the updates include, and what’s changed since the first edition?
Brett: Thanks for considering my book one of the best! The response to the first edition has been wonderful. It was even translated into eight languages! There’s a lot of info about the new book on the official website, including the full table of contents with all of the item titles, a variety of samples, and the example code.
The second edition of Effective Python is nearly twice the length of the original and significantly revises all of the items of advice, in addition to providing 30+ new ones. The first edition of the book was written back in 2014 with Python 2.7 and Python 3.4 in mind. So much has been added in Python 3.5 through Python 3.8 over the past 5 years, so there was a lot to write about! The general style is the same and it still includes multi-color syntax highlighting.
The new book covers the walrus operator, f-strings, best practices for dictionaries, class decorators, and all of the important features of functions (including asynchronous execution). I get into the details of unpacking generalizations, customized sorting, testing facilities, and algorithmic performance in the standard library. I also explain how to leverage more advanced tools such as the typing module and the memoryview type.
In this new edition, my advice about metaclasses has completely changed, thanks to the addition of __init_subclass__ and __set_name__ to the language. My recommendation for advanced generator usage with send() and throw() is now the opposite of what it was in the first edition. (Now I say to avoid them.) I compare the benefits of asyncio to threads and processes, which was entirely missing before. I provide guidance on how to convert synchronous code to asynchronous code, as well as how to mix asyncio with threads.
Anyone out there who reads the book, please feel free to send me questions or feedback!
Ricky: At PyCon 2016, you gave a talk titled Refactoring Python: Why and How to Restructure Your Code, which made our Top 10 Must-Watch PyCon Talks list, too. I’m curious to know your motivations for giving the talk. What was it about refactoring (or the lack thereof) that inspired you to give a talk about it? Do people seem to struggle with refactoring? Are they falling for the same common pitfalls?
Brett: Thanks for including my talk in your top ten list!
When I put this talk together, I had been working in the same codebase for about six years. Any code of that age needs to be refactored constantly and updated to prevent it from rotting and becoming unusable.
The rot I’m talking about happens when dependencies change in backward-incompatible ways, when the underlying infrastructure shifts to have different performance characteristics, or when product decisions break a large number of prior assumptions. So, refactoring was an ongoing necessity and it was on my mind frequently. This experience gave me a new appreciation for how crucial the skill of refactoring is.
People often talk about how testing, tooling, and frameworks make them more effective programmers. But I believe refactoring is the big underappreciated fundamental skill that every programmer needs to focus on improving. Why? In my view, the best code is code that doesn’t exist and has been deleted or was never written. The best program you can have is an empty .py file. The question is how do you get from where you are in a project to closer to that ideal? By refactoring.
I wanted to help the other programmers on my team learn how to become better at refactoring, but the book Refactoring by Martin Fowler hadn’t been updated since 1999. So I was considering writing a Python-specific version of the book to modernize it into something my colleagues could read. However, I put that on hold when I learned that Fowler was working on the second edition of Refactoring, which has since been released and uses JavaScript as the implementation language.
Unfortunately, it still falls short of what I need. I would love to see a Python-specific refactoring book, or a set of workable examples, that take advantage of everything the language has to offer.
Ricky: Now for my last few questions. What else do you get up to in your spare time? What other hobbies and interests do you have, aside from Python and programming?
Brett: We have two very young children at home, so my wife and I spend a lot of time with them. Singing songs and playing music (piano primarily, but sometimes guitar, ukulele, xylophone, and harmonica) have been the best ways to enjoy our time together. I’m terrible at all of these instruments, and not a great singer either, but we still have a lot of nice times.
I like to be outside every day, either walking the hills of San Francisco or going for a run (usually while pushing a stroller). I love to surf when I get a chance. I enjoy reading, especially long-form journalism, but in reality, it’s mostly reddit and Lobsters. I’d like to be a better cook and find the right balance between laziness and deliciousness. I’m also trying to teach myself how to build statistical models using Bayesian methods. If I’ve learned anything over the years, it’s that nothing is for certain.
Thank you, Brett, for joining me this week! You can find Brett on Twitter or GitHub, and learn more about the second edition of Effective Python at effectivepython.com.
If there’s someone you’d like me to interview in the future, then reach out to me in the comments below, or send me a message on Twitter. Happy coding!
