Monitoring Agents in Production

You have one agent running in production and no idea whether it is fine. Fix that, in one sitting, using only what this course covered.

The build

Take any agent you already have, even a two node graph that calls one model. Then wire the whole monitoring surface onto it in order:

1. Trace it. Wrap the entry point with traceable() and confirm a root run appears in a project you named yourself. Expand the tree. Name every run type you can see, and note which of the seven never show up in your app.
2. Thread it. Pass a stable thread_id in metadata across at least three separate invocations, then find all three grouped in the threads view. Now do it a fourth time with session_id instead and confirm it still groups.
3. Price it. Check all three cost preconditions by hand: token counts present, ls_provider and ls_model_name present, and a matching entry in the model price map. Break exactly one of them on purpose and watch the cost column go blank. That is the most instructive thirty seconds in this course.
4. Judge it. Configure one online evaluator over a filter that matches your traffic, with a sampling rate below 1, and find the feedback it wrote back onto the sampled runs.
5. Discover. If your plan allows it, schedule one Insights job and read the category tree it produces. Notice that it told you what to look at, and told you nothing about whether you are getting better.
6. Alarm. Create one threshold rule on error count over a 5 minute window, pointed at a webhook you control. Then deliberately break the agent and confirm the webhook fires.

The bar

You are done when you can point at your own project and answer, without opening the docs: which runs belong to which conversation, what last night cost, what the most common failure category was, whether your quality metric moved, and who gets woken up when it breaks.

If any one of those has no answer, the gap is a real gap, and it is the same gap the exam is testing.