Getting Image Descriptions Using MarkItDown

00:00 So far, you’ve been converting different file formats into Markdown, but what if you wanted to make it more straightforward for LLMs to understand an image?

00:10 Before starting this lesson, make sure you’ve set up your OpenAI API key. If you want to learn more about that, check out either our tutorial or video course, Leverage OpenAI’s API in Your Python Projects.

00:25 MarkItDown can use LLMs to generate image descriptions and captions in an MD format that is straightforward to feed to LLMs. First, you need to have openai installed.

00:38 You can go ahead and do that. Inside of your virtual environment, in your terminal, go ahead and install openai using pip. So pip install openai.

00:50 As you can see here, I already have it installed. And then you also have to export your API key in your environment. You can do that by saying export and then OPENAI_API_KEY equals.

01:07 And then you can paste your API key right here. Okay, your environment is ready. Now your goal is to use MarkItDown to get a description for this specific Real Python image.

01:22 You can find this image in the data subfolder. It’s called real-python.png. Let’s go ahead and write the code. You have a file called image_description.py.

01:34 You need to start by importing both MarkItDown and OpenAI. So from markitdown import MarkItDown. Make sure M, I, and D are capitalized in the second MarkItDown.

01:51 And from openai import OpenAI. O, A, and I are capitalized. Now you need to create an OpenAI client object. The client uses your OpenAI API credentials that you stored in your environment just now.

02:08 So client equals OpenAI, open close parentheses. And then you want to create the MarkItDown converter again. But this time you want to give it access to an LLM.

02:22 How you can do that is saying md equals MarkItDown, open close parentheses. And inside of these parentheses you need to specify two things.

02:34 First, you need to specify your client, meaning which LLM exactly you’re going to be using. You already have a client above. And then you want to specify the model you will be using.

02:46 Let’s go ahead and do that. So llm_client equals client, comma. And then llm_model, you can use whichever model that you’d prefer.

02:58 Here you’re going to see gpt-4o and close parentheses. Now you want to use the .convert() method from before. So result equals, using the converter, md.convert().

03:15 And inside of the parentheses of .convert(), you want to specify the path to the real-python.png. Here it’s in my subdirectory data,

03:29 real-python.png. Make sure the path is inside of quotes. Now you’re ready to print the Markdown content inside of result. So result.markdown.

03:44 Okay, you’re ready to run this. Let’s go ahead and say python image_description.py. And there you go. You get a detailed paragraph explaining everything in the image.

04:05 In this vibrant illustration, three animated characters enjoy an online Python training session, and so on. You just used MarkItDown and OpenAI to get a very detailed description of an image.

04:18 So far, MarkItDown has been amazing. You’ve used it to convert different file formats into Markdown. You’ve used its MCP within Claude Desktop. And also right now, you just learned how to get image descriptions from an image.

04:34 But what’s the catch? What are the bad sides? Keep watching to find out.