Observability

LangChain ecosystem projects trace through LangSmith by default. Tracing turns on automatically when LANGSMITH_TRACING=true is set — no code changes required. Every agent.invoke(...) becomes one trace; tool calls, sub-agent delegations, and middleware hooks are nested spans.

For production, you'll often want OTEL on top of (or instead of) LangSmith.

Required environment variables

LANGSMITH_API_KEY=ls_...        # required
LANGSMITH_TRACING=true          # enables capture
LANGSMITH_PROJECT=my-agent      # bucket name in the LangSmith UI

If LANGSMITH_TRACING is unset or false, the agent runs but no traces are captured. The agent does not fail — silent observability gap. Always set it explicitly.

Where traces live

LangSmith UI → the project named in LANGSMITH_PROJECT. Filter by experiment, time range, or status. Each trace is a tree:

• Top-level: agent.invoke(...).
• Children: middleware wrap_* calls, model calls, tool calls, sub-agent task invocations.
• Leaves: token streams, tool inputs/outputs, errors with stack traces.

What middleware looks like in traces

Each middleware that wraps a model or tool call appears as its own span. Useful for diagnosing "why did this take so long" — you can see exactly which retry or fallback fired.

Manual span instrumentation

from langsmith import traceable

@traceable
def my_helper(x: int) -> int:
    return x * 2

@traceable works on any callable; it shows up as its own span under the parent trace. Use it for application-specific operations not auto-captured (DB queries, external API calls outside of LangChain tools).

OpenTelemetry integration

LangSmith can emit OTLP traces alongside its native ones, so existing observability backends (Jaeger, Tempo, Honeycomb, Datadog) get the data:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=$HONEYCOMB_API_KEY"
export LANGSMITH_OTEL_ENABLED=true

When LANGSMITH_OTEL_ENABLED=true, traces are dual-emitted: to LangSmith AND to your OTLP collector. Spans use OpenTelemetry semantic conventions for AI / LLM workflows.

Distributed tracing across services

If your agent calls another service (e.g. a separate retriever microservice), propagate the trace context so spans link up:

from opentelemetry import propagate, trace

# When making an outbound call:
headers = {}
propagate.inject(headers)   # adds traceparent header
requests.post("https://retriever.internal/", headers=headers, json={...})

The downstream service should call propagate.extract(request.headers) and create a span as a child.

Cloud Run-specific tracing

Cloud Run automatically adds X-Cloud-Trace-Context to incoming requests. To correlate Cloud Run access logs with LangSmith spans, log the trace ID at the start of each request handler:

import os, structlog
log = structlog.get_logger()

@app.post("/invoke")
def invoke(req, request: Request):
    trace_header = request.headers.get("X-Cloud-Trace-Context", "")
    cloud_trace_id = trace_header.split("/")[0]
    log.bind(cloud_trace_id=cloud_trace_id, thread_id=req.thread_id).info("invoke")
    ...

Common failure-mode diagnostics

Quick local verification (no LangSmith needed)

python -c "
from agent.agent import agent
result = agent.invoke({'messages': [{'role': 'user', 'content': 'hello'}]})
print(result)
"

If this fails, the agent is broken regardless of tracing. Fix it first.

Production alerting

LangSmith UI supports alerts on:

• p95 latency exceeding a threshold.
• Error rate exceeding a threshold (% of runs that throw).
• Specific evaluator scores dropping below a threshold (eval-as-monitor pattern).

Configure these per project in the LangSmith UI. For escalation to PagerDuty / Slack, use LangSmith's webhook integrations or pipe traces through OTEL into your existing alert stack.

Trace sampling for high-volume services

LangSmith captures every trace by default. For very high traffic (>100 RPS sustained), sampling is supported via the LANGSMITH_SAMPLING_RATE env var (e.g. 0.1 = 10%). Trace integrity is preserved for sampled traces (full tree), so this is safe — you just see fewer of them.

Trace URLs in code

When LANGSMITH_TRACING=true, the SDK prints a trace URL to stderr after each agent.invoke(...). Capture it in production logs for debugging:

import sys, contextlib, io

buf = io.StringIO()
with contextlib.redirect_stderr(buf):
    result = agent.invoke({"messages": [...]})
trace_url = next((line for line in buf.getvalue().splitlines() if "smith.langchain.com" in line), "")
log.info("agent_invoked", trace_url=trace_url)

Source: cwijayasundara/agent_cli_langchain — distributed by TomeVault.