Ten of the forty LCAE questions come from here, and most of them are architecture questions wearing an API costume. This course covers the five official focus areas: code-based evaluators versus LLM-as-judge, online versus offline evaluators, running and interpreting experiments, evaluator alignment, and adding examples to a dataset. It leans hard on the distinctions the exam actually tests (the perform_eval signature seam, Align Evals versus few-shot self-improvement, what the comparison view does and does not tell you) and it is honest about the places where the documented tooling gives you less statistical rigour than you would like. The exam is semi-open-book, so this is not a memorisation course. It is a knowing-what-to-look-up course.
You have a judge. You do not yet know whether it is any good, which is the entire problem with judges.
The capstone is one loop, run for real, in the LangSmith org provisioned for you. Take a tracing project with actual runs in it. Pull twenty of those runs into a dataset using at least three different routes from Module 3, deliberately picking a mix that will land roughly half good and half bad, because a dataset that is all passes teaches an evaluator nothing. Label all twenty by hand in an annotation queue, which is the part nobody wants to do and the part that makes everything downstream real.
Now write an LLM-as-judge for the thing you actually care about, and run Align Evals against your human labels. Read the alignment score. It will probably be worse than you expected, and that is the point of the exercise: before this number existed, you were shipping a judge on vibes. Iterate the prompt in the Evaluator Playground, re-run alignment, and stop when the score stops moving rather than when it hits a number you like.
Then run the same judge as an offline evaluator over the dataset, twice, once with repetitions set to 1 and once higher, and look at the standard deviation LangSmith reports. Finally, attach a code-based evaluator to a tracing project as an online evaluator, and notice that you had to delete the example parameter to make it work. If you can explain to somebody else why you had to do that, you can pass this domain.