Skip to main content

Observe traffic

Capture LLM metrics, logs, and traces, including request and response bodies and token usage, through OpenTelemetry.

Get prompt logging, cost tracking, and a full audit trail: review LLM-specific metrics, logs, and traces (request/response capture and token usage) via OpenTelemetry.

Before you begin​

Complete an LLM guide, such as the virtual key management guide. This guide sends a request to the LLM and receives a response. You can use this request and response example to verify metrics, logs, and traces.

View LLM metrics​

You can access the agentgateway metrics endpoint to view LLM-specific metrics, such as the number of tokens that you used during a request or response.

  1. Open the agentgateway metrics endpoint.
  2. Look for the agentgateway_gen_ai_client_token_usage metric. This metric is a histogram and includes important information about the request and the response from the LLM, such as:
    • gen_ai_token_type: Whether this metric is about a request (input) or response (output).
    • gen_ai_operation_name: The name of the operation that was performed.
    • gen_ai_system: The LLM provider that was used for the request/response.
    • gen_ai_request_model: The model that was used for the request.
    • gen_ai_response_model: The model that was used for the response.

For more information, see the Semantic conventions for generative AI metrics in the OpenTelemetry docs.

View realized costs​

When you configure a model cost catalog, agentgateway computes the realized USD cost of each LLM request and exposes it across the observability surface:

  • Logs: each LLM request log line includes agw.ai.usage.cost.total. Add the cost breakdown or applied rates with CEL llm.cost and llm.costRates fields.
  • Metrics: the agentgateway_cost_catalog_lookups_total counter tracks lookups by status (Exact, Unpriced, Missing, or NoCatalog) and by provider and model, so you can confirm that your catalog prices your traffic.
  • Traces: cost attributes are attached to the request span.

For catalog configuration and the full list of cost fields, see Model costs.

View traces​

  1. Use docker compose to spin up a Jaeger instance with the following components:

    • An OpenTelemetry collector that receives traces from the agentgateway and forwards them to Jaeger. The collector is exposed on http://localhost:4317.
    • A Jaeger agent that receives the collected traces. The agent is exposed on http://localhost:14268.
    • A Jaeger UI that is exposed on http://localhost:16686.

    Steps to create a Jaeger instance:

    1. Create the OpenTelemetry collector configuration file in your current directory. The Compose file in the next step mounts this file into the collector container. Both files are sourced from the examples/mcp-telemetry directory in the agentgateway repository.

      cat > otel-collector-config.yaml <<'EOF'
      receivers:
           otlp:
             protocols:
               grpc:
                 endpoint: 0.0.0.0:4317

         processors:
           batch: {}

         exporters:
           debug:
             verbosity: detailed
           otlp/jaeger:
             endpoint: jaeger:4317
             tls:
               insecure: true

         service:
           pipelines:
             logs:
               receivers: [otlp]
               processors: [batch]
               exporters: [debug]
             traces:
               receivers: [otlp]
               processors: [batch]
               exporters: [otlp/jaeger]

      EOF

    2. Spin up the Jaeger and collector containers.

      docker compose -f - up -d <<EOF
      services:
           jaeger:
             container_name: jaeger
             restart: unless-stopped
             image: jaegertracing/all-in-one:latest
             ports:
             - "127.0.0.1:16686:16686"
             - "127.0.0.1:14268:14268"
             environment:
             - COLLECTOR_OTLP_ENABLED=true

           otel-collector:
             container_name: otel-collector
             image: otel/opentelemetry-collector-contrib:0.146.0
             volumes:
               - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
             ports:
               - "127.0.0.1:4317:4317"
             depends_on:
               - jaeger

      EOF

  2. Configure your agentgateway proxy to emit traces and send them to the built-in OpenTelemetry collector agent.

    cat <<'EOF' > config.yaml
    # yaml-language-server: $schema=https://agentgateway.dev/schema/config
    config:
      tracing:
        otlpEndpoint: http://localhost:4317
        randomSampling: true
    llm:
      models:
      - name: "*"
        provider: openAI
        params:
          apiKey: "$OPENAI_API_KEY"
    EOF

  3. Run your agentgateway proxy.

    agentgateway -f config.yaml

  4. Send a request to the OpenAI provider. In simplified LLM configuration mode, the LLM listener uses port 4000 by default (port 3000 is reserved for MCP), and requests use the OpenAI-compatible /v1/chat/completions path.

    curl http://localhost:4000/v1/chat/completions \
    --header 'Content-Type: application/json' \
    --data '{
      "model": "gpt-4o",
      "messages": [
        {
          "role": "user",
          "content": "Tell me a short story"
        }
      ]
    }'

  5. Open the Jaeger UI and verify that you can see traces for your LLM request.

View logs​

Agentgateway automatically logs information to stdout. When you run agentgateway on your local machine, you can view a log entry for each request that is sent to agentgateway in your CLI output.

Example for a successful request to the OpenAI LLM:

2025-12-12T21:56:02.809082Z	info	request gateway=agentgateway listener=http route=openai endpoint=api.openai.com:443
src.addr=127.0.0.1:60862 http.method=POST http.host=localhost http.path=/openai http.version=HTTP/1.1
http.status=200 protocol=llm gen_ai.operation.name=chat gen_ai.provider.name=openai
gen_ai.request.model=gpt-4o gen_ai.response.model=gpt-4o-2024-08-06
gen_ai.usage.input_tokens=68 gen_ai.usage.output_tokens=298 duration=2488ms

Example for a rate limited request:

2025-12-12T21:40:18.687849Z	info	request gateway=agentgateway listener=http route=openai endpoint=api.openai.com:443
src.addr=127.0.0.1:51794 http.method=POST http.host=localhost http.path=/openai http.version=HTTP/1.1
http.status=429 protocol=llm gen_ai.operation.name=chat gen_ai.provider.name=openai
gen_ai.request.model=gpt-4o error=rate limit exceeded duration=206ms