Solving the Task in Your REPL

00:00 So the highest level interface that MechanicalSoup provides is an object called StatefulBrowser() and that allows you to do a couple of common actions such as finding and filling a form quite straightforwardly.

00:11 So I’ll work with that and I’ll make my browser a stateful browser browser = mechanicalsoup. StatefulBrowser(). That’s the object that I want to instantiate.

00:27 And then what else do I need? I need my login URL, and I have that copied already. So I will go ahead and say `login_url = a string points to 'http://olympus.realpython.org/login'.

00:43 And now I can do fun things using my browser object and saying browser.open (login_url).

00:52 And you can see as a response I get a response object that says Response [200]. And this is a requests response object, which let me actually show you that browser. open(login_url) I’m gonna pass it to the type() function so that you can see that it’s a requests.models.Response object, which shows you that one of the libraries that MechanicalSoup uses under the hood is the requests library.

01:19 And the second one is BeautifulSoup, as you will see in just a moment. But this also contributes to the quite straightforward interface that this library gives and allows you to interact with simple websites.

01:33 I opened it, so now my StatefulBrowser knows about the url /login, right? But more than that, it also already has access to the HTML.

01:44 So I can say browser.page. And you can see that I get on the HTML that you saw before when looking at the source of the page in the browser.

01:53 So this already got the page and saved the HTML somewhere where I can use it. Now maybe you’re wondering what type is that? Is that yet another long Python string?

02:03 But worry not, it isn’t. So if I pass this to type, I can show you that `browser. page` is in fact a BeautifulSoup object which helps in making it easier to parse and work with it, right?

02:17 But MechanicalSoup adds some more abstraction on top of BeautifulSoup, which makes it even easier to interact with. What I can do is say browser.select_form.

02:30 So that’s a method on the browser object rather than just give it the name of the form and it creates a MechanicalSoup form object that returns that. But the browser is also pointing to that, so I don’t even need, need to save this to a variable.

02:44 The browser is stateful and remembers what you’re doing internally to a degree that now you can do browser square bracket notation and just give it the name of the input field that you want to fill.

02:58 And that is “user” in this case. The first input names field is “user” so I’ll pass in “user” here in the square brackets and I will fill it with zeus’s username.

03:12 And then I can do basically the same thing again, but I want to use the password field. So the name of the second input field is “pwd” and then again, fill it with the right value ThunderDude.

03:27 Alright. And now I can go ahead and keep working with the browser object, say browser.submit_selected,

03:38 I got a response [200] and now you may be wondering which page are you on your browser navigated. So if you now called browser page, you’ll see that you’re on the All Profiles page.

03:51 So that’s pretty amazing, right? And now, well the last task was to print out just the title.

03:56 And since you’re working with a BeautifulSoup object, you can easily access that browser.page. It can even more easily access it using MechanicalSoup so I can do something like that where this gives me the BeautifulSoup object.

04:09 And then I just want the title element and it can see that it prints out the thing that I need.

04:16 This is a pretty fun way of interacting with a website in my opinion. At the end of this, I’ll give you a couple of gotchas and things that you need to be aware of where this works and where it doesn’t.

04:26 But for now there’s your choice and then move this information and this little practice round in the Python REPL into the script so that we can reliably reproduce it by just running the script.