Discussing Output Reliability

00:00 Let’s discuss output reliability for a second. Pydantic AI is going to take your prompt and the desired output type, and it’s going to send a request to the LLM, and then it will try to build the model you specified, but this whole thing might fail.

00:16 Step by step, what happens is you define the model, you write your prompt, and you specify the output type to be that model, and then you pass that to Pydantic AI.

00:25 Pydantic AI will then send a request to the LLM with your prompt and the expected output type so that the LLM tries to reply or give an answer with a format that Pydantic AI can work with.

00:38 And once Pydantic AI has the response, it tries to parse it, it tries to do the type conversion that it needs, and then it gives you the answer. Now the process of parsing the information for the Pydantic model from the LLM response, that is the thing that might fail, or that specific step might fail, and in order to improve the reliability of your output, what you can do is automatically set a loop in which if building the Pydantic model fails, then Pydantic AI automatically retries and asks the LLM for the same information again.

01:18 Now it might sound silly to retry the exact same thing, but remember that LLMs are not deterministic, so sending the exact same prompt again can actually yield a different response that Pydantic AI might be able to parse.

01:33 So you can set this number of retries to be an integer that you want, 1, 2, 3, 4, 5, whatever you want, and then once Pydantic AI succeeds in parsing the output, it’s going to give you your response. In order to set this automatic retry in your code, you’re going to want to set the argument output_retries in the agent. By default it’s 1, you can set it to another integer.

02:02 There’s one more thing to be said about these automatic retries.

02:07 Retrying spends tokens. It’s going to either cost you money, or it’s going to eat up on your quota for your plan, because each retry, each Pydantic AI retry means that a prompt is being sent to the LLM and the response is coming back. So be aware of the potential increase in cost if you set your output_retries to a large number. Now obviously the need for the retries will also depend on the complexity of your prompts, the complexity of the expected output type, and the model itself. Different models have different capabilities,

02:48 and in general, with time, models will improve, but all of these things are variable, so you need to test with your provider, with your models, and with your prompts and outputs to find the balance that works best for you.

03:04 Now that you’ve learned how to automatically retry failed prompts, in the next lesson you’re going to learn how to enhance your agents by using tools.