Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
Infrastructure as Code
Most of the interactions you’ve had with S3 in this course had to do with objects. You didn’t see many bucket-related operations, such as adding policies to the bucket, adding a LifeCycle rule to transition your objects through the storage classes, archive them to Glacier or delete them altogether or enforcing that all objects be encrypted by configuring Bucket Encryption.
Manually managing the state of your buckets via Boto3’s clients or resources becomes increasingly difficult as your application starts adding other services and grows more complex. To monitor your infrastructure in concert with Boto3, consider using an Infrastructure as Code (IaC) tool such as CloudFormation or Terraform to manage your application’s infrastructure. Either one of these tools will maintain the state of your infrastructure and inform you of the changes that you’ve applied.
If you decide to go down this route, keep the following in mind:
- Any bucket related-operation that modifies the bucket in any way should be done via IaC.
- If you want all your objects to act in the same way (all encrypted, or all public, for example), usually there is a way to do this directly using IaC, by adding a Bucket Policy or a specific Bucket property.
- Bucket read operations, such as iterating through the contents of a bucket, should be done using Boto3.
- Object-related operations at an individual object level should be done using Boto3.
00:00 So far, most of the examples that we’ve worked on have had to do with objects. While you did learn to use Boto3 to set Access Control Lists or encryption to buckets, managing buckets through Boto3 can become very difficult as your project grows in size.
00:17
This is where Infrastructure as Code, or IaC, comes in. IaC is about managing your assets through code. While using boto3 in Python is also code, IaC is more geared towards setting up configuration files.
00:33 These files will set the rules for your assets as you’re setting them up and also if you need to make changes across the board. One of the goals of IaC is to reduce or eliminate config through “tools”.
00:47 When I say “tools” here, I’m talking about things like the graphic web interface on AWS’s website or anything that you have to point and click through. Trying to automate a process like that to make it fast and consistent can be very difficult, especially if you need to go back and make changes.
01:05 Some examples of IaC tools are things like CloudFormation or Terraform. These are services that will allow you to set up the state of your infrastructure and allow you to control changes to it.
01:18 So, when should you use Boto3 or IaC?
01:22 IaC is very good when you’re modifying or setting up your buckets. You’ll use your config files to make sure that all new buckets follow the policies that you define.
01:33 Another good use of IaC is if all of your objects need to follow the same rules.
01:38 Most IaC tools will allow you to set these policies and have them take effect across the board. Boto3, on the other hand, is good when you need to read individual buckets.
01:49 This can be as you’re looking for a specific object or a group of objects and any object-related operations, like uploading or accessing. A good rule of thumb is to use IaC to set up the rules and framework for your assets, and then Boto3 for actually interacting with your assets.
02:09 If you’re interested in IaC, take a look at CloudFormation and Terraform and see what kind of benefits they might be able to bring your project. And that’s all there is to learning how to use Boto3. In the next video, we’ll wrap up and review everything that you’ve learned.
Scott Pedersen on Aug. 20, 2023
Hi,
It seems the code_examples.zip file does not contain the files for this course.
Can the link please be updated to point to the correct file?
Bartosz Zaczyński RP Team on Aug. 21, 2023
@Scott Pedersen Thanks for flagging it. We’ll look into this.
Chris Bailey RP Team on Aug. 21, 2023
@Scott Pedersen - The original file has been restored. Thanks for letting us know.
Scott Pedersen on Aug. 21, 2023
Thank you for fixing so quickly :)
Become a Member to join the conversation.
Mark Walle on Aug. 24, 2020
There are AWS CDK (Cloud Development Kit) bindings for Python that allow people who prefer to scaffold their IAC in an imperative language rather than as declarative markup. After defining an IAC stack in Python using the CDK, it can then generate the necessary CloudFormation template to build, deploy, or destroy the infrastructure defined in your Python code.
It would be great to see a multipart series on using AWS CDK with Python on Real Python. But in absence of that, here is an AWS Developer blog that provides a good start with the tool: aws.amazon.com/blogs/developer/getting-started-with-the-aws-cloud-development-kit-and-python/