Understanding the Importance of the Markdown Format

00:00 You’re about to start setting up the MarkItDown library on your laptop or PC, but first, you might be wondering why the Markdown format? Why do you need to turn formats like PDF or CSV into Markdown?

00:14 It’s because Markdown, or the .md file format, comes with its own advantages. The Markdown format keeps text straightforward while preserving document structure.

00:26 It represents headings, lists, links, tables, code blocks, and emphasis using lightweight syntax instead of complex formatting instructions. Markdown format matches the format LLMs already understand well, and this is because LLMs are commonly trained on Markdown-heavy content such as documentation, README files from GitHub, forum posts, and in general technical articles.

00:54 So structures like heading or list items and fenced code blocks are straightforward for them to interpret. And finally, the Markdown format uses fewer tokens than many other structured formats.

01:09 Markdown usually requires less markup than formats like HTML or XML. To grasp this better, look at this table. You can see three columns. You have Markdown on the left, then you have HTML and XML on the right.

01:25 If you were to specify a heading, you just need a hash symbol in Markdown, and in HTML, you’d need to open and close the heading tag. And it’s the same with many other things like bold text, lists, or creating a linked text.

01:41 Markdown is generally shorter, and because of that, it uses fewer tokens when it comes to LLMs processing information. Now you understand that Markdown is a convenient way to feed information to LLMs.

01:54 Next, you need to set up MarkItDown so you can convert formats like PDF or DOCX into Markdown.