Offline vs online, the two worlds

Offline evals run against a dataset you built, so the right answer is sitting there to compare against. Online evals run against live production traffic, where there is no right answer, because nobody wrote one down.

That is the whole lesson. Everything below is consequences.

Offline: pre-deployment testing

Offline evaluation is what you do before you ship. You have a dataset of examples, each one an input paired with a reference output, meaning the answer you decided was correct. You run your application over those inputs, and evaluators score the outputs, often by comparing them to the reference.

This is the mode that answers "is version B better than version A", because you are holding the inputs fixed and varying the code. It is a regression suite that speaks in scores instead of booleans. Offline evaluators attach to a dataset.

Online: production monitoring

Online evaluation is what you do after you ship. It runs against real runs and threads flowing through a tracing project, continuously, on traffic you did not choose.

You configure it with a filter, deciding which runs qualify, and a sampling rate between 0 and 1, deciding what fraction of the qualifying ones actually get scored. You do not evaluate everything, because at production volume an LLM-as-judge on every run is a genuinely large model bill.

And critically, there is no reference output anywhere in this picture. You can ask "is this output valid JSON", "did it call a tool it was not supposed to", "does a judge think this was helpful". You cannot ask "is this correct", because correct compared to what.

The side effect nobody expects

Here is a thing that has nothing to do with evaluation and everything to do with your invoice. Attaching an online evaluator automatically upgrades matching traces to extended data retention.

Read that again with your finance hat on. You wire up a cheap-looking heuristic evaluator on your highest-volume project, set the sampling rate generously because sampling more sounds more rigorous, and you have quietly changed the retention tier on a large slice of your traffic. It is a completely reasonable design decision by LangSmith (an evaluated trace is one you will want to go back and look at) and it is a beautiful exam question precisely because it is a side effect rather than a feature.

Choosing between them, out loud

You do not pick one. They cover different failure modes and they feed each other.

Offline catches "the change I just made broke the thing that used to work". Online catches "reality contains inputs I never imagined". The bridge between them is Module 3's harvest: a production run scores badly online, you add it to a dataset, and now it is an offline test forever.

If an exam question gives you a scenario, the tell is the word "production". Production traffic, live users, monitoring, drift, alerting: online. Regression, before deploying, comparing two versions, prompt A versus prompt B: offline.