What testing an agent actually means

Everything you know about testing assumes the function returns the same thing twice. Agents do not do that, and pretending otherwise is how teams end up with a test suite everyone has learned to ignore.

The assertion problem

A pure function has one right answer, so expect(f(x)).toBe(y) is a complete test. A model has a distribution of acceptable answers, and the good ones differ from each other. Assert equality and you get red on output that was genuinely fine, every single run. A test that cries wolf that reliably is worse than no test, because it teaches the whole team that red means nothing.

So the assertion gets replaced by a score. You stop asking "is this the right string" and start asking "does this output have the property I care about", then you watch that score across versions. That scoring function is an evaluator, and the entire rest of this course is about evaluators, what you run them against, and how you know whether to trust them.

The loop

The shape you are being tested on is a cycle, not a phase gate.

You build. You test offline, before deploying, against a curated set of inputs where you wrote down what good looks like. You deploy. You monitor online, against real production traffic, where nobody wrote down what good looks like because the user just typed a thing and left. You harvest the interesting production runs back into your curated set. Then you build again, and now your offline tests include the failure that embarrassed you last Tuesday.

That harvest step is where the value compounds. Your dataset is not something you author once at the start. It is a slowly growing museum of every way your agent has ever been wrong, and Module 3 is entirely about the six routes for getting exhibits into it.

Why this exam rewards architecture over memory

The LCAE is semi-open-book. During the exam you can consult docs.langchain.com and smith.langchain.com. You cannot use general web search and you cannot use an AI.

Think about what that does to the question design. Nobody writes a 40-question paper where a third of the answers are one search away in a permitted tab. So the questions that survive are the ones the docs cannot answer for you quickly: which evaluator type fits this situation, why does this signature differ, what does this number actually license you to conclude. Look-up-able facts get asked as traps rather than as recall, phrased so that you need to already know which page to open.

There is also a LangSmith org provisioned per candidate, and some items are answered by doing something in the UI. So when this course teaches a UI workflow, learn it as clicks, not as prose.

What to actually study

Spend your time on distinctions, not signatures. The five official focus areas for this domain are code-based evaluators versus LLM-as-judge, online versus offline evaluators, running and interpreting experiments, evaluator alignment, and adding examples to a dataset. Four of those five are versus or interpreting. That is your hint.