04.7.7 · Skill in AI-native engineering
Evals and datasets
Measuring AI behavior with test sets and repeatable evaluation runs.
Evals replace "it looked fine" with numbers. A dataset of real cases, a grading method, which may use code, people or another model, and repeatable runs let you compare prompts and models. They are the AI version of automated tests.
Resources
4- articleStart hereYour AI product needs evals ↗Hamel Husain · 2024 · 18 min
The practical case for evals, built from a real product, from simple assertions to human review of traces.
- articleGo deeperA field guide to rapidly improving AI products ↗Hamel Husain · 2025 · 28 min
Error analysis first: read real traces, find what fails, then measure it.
- articleGo deeperUsing LLM-as-a-judge for evaluation: a complete guide ↗Hamel Husain · 2024 · 35 min
How to build a model-based grader and check it against human judgment before trusting it.
- articleGo deeperAI evals: everything you need to know ↗Hamel Husain and Shreya Shankar · 2026 · 80 min
Answers to the questions teams ask most about evals, from two people who teach it.
Tools
2- ToolOpen sourcepromptfoo
Open-source tool for testing prompts and models against test cases, and for red-teaming LLM apps.
- ToolOpen source