What Are AI Evals? How to Tell If Your Prompt Change Helped

You tweaked your summary prompt and this answer looks better, but is it actually better than before? If you judge by a handful of outputs, it's easy to miss that the new prompt now fails more often on other meetings or other requests.

AI evals are repeatable tests that run an AI on the same cases with the same grading criteria, so you can see the real effect of changing a prompt or model. OpenAI describes evals as structured tests for measuring a model's performance. It sounds like engineering work, but marketers and product planners can use the same idea. This post draws on OpenAI's official documentation to explain what evals are, what they're made of, and how to run a before-and-after comparison.

What are AI evals? Meaning and three building blocks

An eval works like an exam: a question sheet, an answer key, and a grader. Because you grade the same sheet the same way each time, repeated runs help distinguish a prompt effect from ordinary output variation.

Don't only collect easy cases. OpenAI recommends including typical cases, tricky edge cases, and deliberately adversarial inputs. A high pass rate means the system handled the cases you prepared, not that it's right in every situation.

Five steps to compare before and after with the same test set

To measure a prompt change, change one thing, keep everything else fixed, and run the same test set before and after. If several things change at once, you can't tell what made the difference.

1. Collect representative cases: Pick 10 to 20 requests you make often, and mix in a few tricky ones.
2. Define criteria: Turn "good" into checkable statements, like "zero missed decisions, under 300 words."
3. Save a baseline: Record the current prompt's outputs and scores first.
4. Change one thing and rerun: Edit only the prompt and run the same cases. Don't switch models or source material at the same time.
5. Compare by case type: Don't stop at the average. Check each type of case for anything that got worse. When something that used to work breaks, that's called a regression.

If you keep tuning against the same cases, you'll end up with a prompt that only fits those cases. Hold back a few cases you never look at while editing, and add real failures to the test set as they appear.

How to grade: rules, AI judges, and people

There are three ways to grade. Use rules when the answer is clear-cut, an AI judge when meaning matters, and people for the most important calls. OpenAI recommends calibrating automated scoring against human judgment.

When you use an AI judge, don't just ask "Is this good?" Give it a clear rubric with specific criteria, examples for each score, and a pass threshold. Asking it to pick the better of two answers, or to give a pass or fail, tends to be more reliable. Source: OpenAI, Evaluation best practices

OpenAI Evals is shutting down: moving to Promptfoo

"Evals" can mean the practice of evaluation or the name of OpenAI's eval dashboard and API. The OpenAI Evals platform is scheduled to shut down on November 30, 2026.

According to OpenAI's deprecations page, the shutdown was announced on June 3, 2026, existing evals became read-only on October 31, and the dashboard and API are scheduled to shut down on November 30. OpenAI points to a migration path using Promptfoo, an open-source tool for evaluating AI apps and agents. You recreate your prompts, provider, test cases, and assertions in a config file and run them there.

The tool changes, but the practice doesn't. If your test set and criteria are well organized, you can keep running the same comparisons anywhere. After migrating, confirm on a few cases that the new setup grades the same way as the old one. Sources: OpenAI, Deprecations, OpenAI Cookbook, Moving from OpenAI Evals to Promptfoo

Testing changes to your daglo My Templates the same way

The same principle applies when you change how your meetings get summarized. To know whether new instructions really help, compare the before and after on a few recordings whose content you already know.

daglo's My Templates lets you save the summary prompts you use often, and daglo remembers the format and summarizes every recording that way automatically. When you edit a template's instructions, try this:

• Pick three to five past recordings as your test set. Include a meeting full of decisions and one with lots of small talk.
• Select the edited template, upload the same recordings again, and put the new summaries next to the old ones.
• Check against a short list: Are any decisions missing? Are owners mentioned in the source represented accurately? Because the transcript sits right next to the summary, gaps are easy to spot.

Getting owners' names right is a key criterion too. Speaker ID remembers the voices of the people in your meeting and fills in their real names instead of "Speaker 1" from the next recording on. daglo is the first in Korea to offer this to every user. With names in the transcript, you can quickly confirm that each action item in the summary belongs to the right person. Learn more about Speaker ID

Read next: why to design evals before building an agent, with real examples of rule, model, and human grading, in 6 AI Agent Design Patterns and Why You Should Design Evals First.

#AIEvals #PromptTesting #PromptEngineering #LLMasJudge #Rubric #Promptfoo #OpenAI #daglo

👉 Build your summary template