Join us and get access to thousands of tutorials and a community of expert Pythonistas.

This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Bucket and Object Traversals

Demystifying Python, Boto3, and AWS S3 Joe Tatusko 02:59

If you need to retrieve information from or apply an operation to all your S3 resources, Boto3 gives you several ways to iteratively traverse your buckets and your objects.

To traverse all the buckets in your account, you can use the resource’s buckets attribute alongside .all(), which gives you the complete list of Bucket instances.

You can use the client to retrieve the bucket information as well, but the code is more complex, as you need to extract it from the dictionary that the client returns.

00:00 Sometimes you need to know info on all of your buckets or objects in a bucket, or maybe you need to apply some action to everything. When this comes up, Boto3 gives you a way to iterate over your buckets and objects.

00:15 We’ll start with buckets. Since your resource interface is built off of your user credentials, it also has access to all of your buckets. You can return a list of all of your buckets to loop through by using the resource .buckets.all() method, like so. Let’s take a look at our s3_resource and then call .buckets off of that and .all().

00:41 You’ll see you end up with this bucketsCollection here. So if you wanted to loop through that, you could do a regular for loop and say for bucket in s3_resource.buckets.all():

00:59 go ahead and print the bucket.name.

01:06 And look at that! You can see those are both buckets that we’ve created. Cool! So now that you can loop through your buckets, what if you want to take a look at the objects within a bucket?

01:18 Let’s go ahead and define the first_bucket as being the s3_resource.Bucket() and pass in the name=first_bucket_name.

01:35 All right. And just like the resource, this Bucket contains all of the objects within it. So, you could make a loop that’s something like for obj in first_bucket.objects.all():

01:55 print(), and let’s just print the obj.key.

02:02 And there you go! We’ve got all three files there that printed out. Now, one thing to keep in mind is that the object that’s returned in this list is an ObjectSummary, and that’s just a lightweight version of a regular Object.

02:16 It doesn’t have quite all the attributes of the full Object, but it does have the .name at least, so if you need to use some of those attributes that aren’t available, you can always create the new Object instance as needed in your code.

02:29 Awesome! Now you have the ability to iterate over all of your buckets and all of the objects within those buckets. Hopefully, by now you’re starting to see how beneficial this Boto3 resource interface is.

02:42 The object-oriented representations of the data in your S3 become very straightforward and consistent to work with. In the next video, you’re going to learn how to delete buckets, which is not quite as intuitive as deleting an individual object.

Become a Member to join the conversation.