Monitor agents
The Monitor domain is LangSmith observability: the trace data model, grouping traces into threads, reading traces, cost and feedback, alerting, dashboards, and Online Evals versus Insights. Ten questions.
Try it yourself
Run as a span
In the LangSmith trace model, a Run is analogous to what in OpenTelemetry?
Show answer
Correct answer: B — A span within a trace
A Run is analogous to a span; a Trace is the collection of runs for one top-level operation. Docs: docs.langchain.com/langsmith (Monitor).
Grouping into threads
How are multiple traces grouped into a single multi-turn session?
Show answer
Correct answer: C — By a thread_id metadata key (forming a Thread)
Traces group into a Thread via the thread_id metadata key. Docs: docs.langchain.com/langsmith (Monitor).
What a trajectory is
A Trajectory is:
Show answer
Correct answer: D — A flat, ordered list of messages showing the path an agent took start to finish
A trajectory is a flat, ordered list of messages showing the agent's full path, rendered in the Messages view. Docs: docs.langchain.com/langsmith (Monitor).
Manual instrumentation
Which is the correct way to manually instrument a function for tracing?
Show answer
Correct answer: A — The @traceable decorator (or the trace context manager / RunTree API)
@traceable, the trace context manager, and the RunTree API are the manual instrumentation tools. Docs: docs.langchain.com/langsmith (Monitor).
Online Evals vs Insights
The exam frames Online Evals vs Insights as:
Show answer
Correct answer: B — Online Evals score individual live runs; Insights surface aggregate trends/patterns across many traces
Online Evals score individual live runs; Insights find aggregate trends across many traces. The Study Pack marks the exact Insights mechanics [unverified] at source; the tested distinction is the concept stated here. Docs: docs.langchain.com/langsmith (Monitor).
Reading a trace
You open one trace to debug why a single operation ran slow. What are you reading?
Show answer
Correct answer: A — A tree of runs: the runs recorded for that one operation
A trace is the collection of runs for a single operation, shaped as a tree of runs, and it is the surface you reach for to debug why one operation failed or ran slow. Docs: docs.langchain.com/langsmith/observability-concepts (Monitor).
How per-run cost is computed
LangSmith shows token cost per run automatically. What has to be in place for those costs to be calculated?
Show answer
Correct answer: D — A model pricing map from model names to per-token prices
Costs are computed from token counts using the model pricing map, which maps model names to per-token prices; the UI splits spend into input, output, and other. Docs: docs.langchain.com/langsmith/cost-tracking (Monitor).
Capturing user sentiment
You want to record an end user's thumbs-up or thumbs-down against the run that produced each answer. In LangSmith that is captured as:
Show answer
Correct answer: B — Feedback: a tag and a score bound to the run by its run ID
Each feedback entry is a tag and a score, bound to a run by its run ID, and can be continuous or discrete (categorical); end-user sentiment such as thumbs up or down is recorded this way. Docs: docs.langchain.com/langsmith/observability-concepts (Monitor).
What alerts can watch
Which set describes metrics LangSmith can raise a threshold alert on?
Show answer
Correct answer: C — Run count, error rate, latency, feedback score, and cost
Alerts fire when a threshold is breached on run count, errors (count or rate), latency, feedback score, or cost, over a 5 or 15 minute window, and deliver to Slack, PagerDuty, or a webhook. Docs: docs.langchain.com/langsmith/alerts (Monitor).
Prebuilt vs custom dashboards
What does LangSmith give you for monitoring one project's production performance over time?
Show answer
Correct answer: D — A prebuilt dashboard per project, and custom dashboards too
Every tracing project gets a prebuilt dashboard covering trace count, error rates, and token usage, and you can also assemble custom dashboards from configurable charts. Docs: docs.langchain.com/langsmith/dashboards (Monitor).