Module 8, Lesson 225 min

Datasets and evaluate(): a real accuracy score

Datasets are labelled examples. A dataset is a set of { inputs, outputs } pairs. Build one from a real run: chronicler's cached day results are already input (messages) plus expected output (A/B/C decisions). Upload those as a dataset and you have ground truth.

evaluate() runs an experiment. You give it a target function, a dataset, and evaluators, functions that score one output against its expected value.

Running an evaluation experiment

import { evaluate } from "langsmith/evaluation";

await evaluate(
  (input) => runClassifier(input),          // target
  {
    data: "chronicler-day-2026-07-01",       // dataset name
    evaluators: [
      ({ run, example }) => ({
        key: "category_match",
        score: run.outputs.category === example.outputs.category ? 1 : 0,
      }),
    ],
  },
);

Each evaluator returns { key, score }. LangSmith aggregates them into an experiment with a real number: classification accuracy across the dataset. Verify the exact evaluate() TS signature against the LangSmith reference at build time, the evaluator argument shape in particular is worth confirming.

Practice

Try it yourself

Recall

Why chronicler's cached day is a good dataset source

This has two halves, and the second is easy to skip. A dataset needs inputs and a trusted expected output.

Why chronicler's cached day specifically, and where does "expected output" come from?

Quiz

Evaluator vs target function

In evaluate(target, { data, evaluators }), what is the difference between the target function and an evaluator?

Trace, dataset, and eval: classifier plus vault-tidy

Instrument the module 1 and 2 builds, then extend the same lens to vault-tidy.

Set LANGSMITH_TRACING=true and LANGSMITH_API_KEY, and confirm the m1 classifier spike and m2 todoist agent both appear as traces in LangSmith.
Wrap one raw-SDK call (or a small standalone script using @anthropic-ai/sdk) with wrapAnthropic and confirm it also traces.
Build a dataset from chronicler's cached day results: messages as inputs, cached A/B/C decisions as expected outputs.
Run evaluate() with a category_match evaluator and get a real accuracy score for the m1 classifier.
Point the same lens at vault-tidy: take a sample of notes, define what a 'correct' frontmatter fix looks like, and score vault-tidy's actual fixes against that sample.
Write up the vault-tidy baseline as a real measured number, not a vibe.

Tick every step to confirm you did it.

Check

vault-tidy quality is measured, not vibed

Confirm the vault-tidy eval produced a real baseline.

You should see

A concrete accuracy or quality score exists for vault-tidy's frontmatter fixes on a real sample of notes, with a written explanation of how expected outputs were constructed for a task that has no obvious ground truth.