Running experiments, and reading them honestly

An experiment is one run of your application over one dataset, scored by one or more evaluators, recorded so you can compare it to the next one. The API is small. The interesting parts are the parameter almost nobody sets and the statistics that are not there.

The call

You give evaluate() three things: the thing under test, the data to test it on, and the evaluators to score with.

from langsmith import evaluate

results = evaluate(
    my_app,
    data="my-dataset",
    evaluators=[correctness, conciseness],
    num_repetitions=3,
)

import { evaluate } from "langsmith/evaluation";

const results = await evaluate(myApp, {
  data: "my-dataset",
  evaluators: [correctness, conciseness],
  numRepetitions: 3,
});

Note what changed beyond the obvious. num_repetitions became numRepetitions, and data went from optional in Python to required in TypeScript. Both are documented differences and both are exactly the shape of question that tests whether you read the language toggle.

Python also has aevaluate() for async evaluation and TypeScript has no equivalent: no aEvaluate, no evaluateAsync, no flag. Worth naming explicitly, because the natural response to not finding something in the docs is to assume you searched badly.

The parameter almost nobody sets

Repetitions defaults to 1.

Sit with that. Every experiment you have run without thinking about it ran each example exactly once. You drew one sample per example from a nondeterministic system, compared it against another single sample, and concluded which version was better.

Raise it and each example runs multiple times. LangSmith then shows the average and the standard deviation across repetitions, which is frequently humbling: a two-point improvement looks less exciting once you can see the same code varies by five points between runs.

Reading the comparison view

You pick a baseline experiment and everything is coloured relative to it. Red means regression, green means improvement, per example and in aggregate, which lets you scan a long dataset and land on the handful of rows where your change did damage.

That drill-down is the actual value. An aggregate moving from 0.82 to 0.85 tells you almost nothing about what to do next. Six red rows that turn out to be the same category of question tells you where to go.

Colour is relative, always. A red cell is not a bad score, it is a score that went down. You can be red at 0.9 because your baseline was 0.95. The comparison view answers whether this change helped, never whether this is good. Which also makes the baseline a load-bearing and trivially abusable choice: compare against your worst historical run and the screen turns satisfyingly green. Say which baseline you used whenever you report a result.

What you do not get

No confidence intervals. No significance test. No minimum sample size guidance. Nothing warns you that twelve examples is too few, and the tool will happily draw you a lovely green comparison across all twelve.

I am naming the absence as carefully as the presence, because the gap is where people get hurt. A team that believes there is a significance test somewhere in the interface will assume any comparison they are shown has been vetted by one. There is no such vetting.

So decide with judgement and the spread in front of you: raise repetitions before you care about a result, compare the gap against the standard deviation, read the rows that moved, and grow the dataset, which is the crudest and most reliable lever there is.

In the exam, be suspicious of the most rigorous-sounding option. References to p-values, confidence intervals, statistical significance or recommended sample sizes describe capabilities the documentation does not contain, and the paper is semi-open-book against that documentation.