Trace requests with agctl
Verified Code examples on this page have been automatically tested and verified.
Capture a per-request trace as a standalone agentgateway instance handles the request.
This feature is experimental. Try it out, see how it helps, and provide us feedback in GitHub or Discord. But keep in mind that it is subject to change and not supported for production.
Capture a per-request trace as a standalone agentgateway instance handles the request, by using
agctl proxy trace.
About
agctl proxy trace taps the agentgateway admin endpoint at /debug/trace and streams a
step-by-step record of how the proxy processes the next request that arrives. The trace shows you
the matched route, the policies that were applied, the backend that was chosen, and the response
status. Tracing helps you understand why a request matched or did not match a route, why a policy
was or was not applied, or why a request returned an unexpected status.
flowchart LR
A[client] -->|:3000| B[agentgateway]
B -->|:8000| C[httpbin]
B -.->|/debug/trace SSE| D[agctl proxy trace --local]
You can run agctl proxy trace in two modes:
-
Inject mode: Enables tracing and sends the request itself through the same port-forward. The host portion of the URL sets the
Hostheader but is not used for DNS resolution.agctl proxy trace gateway/<name> --port <listener> -- <url> -
Watch mode: Waits for the next request that arrives at the proxy and traces it. Send the request from any client.
agctl proxy trace gateway/<name>
Before you begin
- Install agctl.
- Install the agentgateway binary and have a
config.yamlready. The examples in this guide use the non-agentic HTTP quickstart setup, with agentgateway listening on:3000and forwarding to httpbin on:8000.
If you do not already have a setup, the following minimal config.yaml works for the examples in
this guide.
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
gateways:
default:
port: 3000
protocol: HTTP
routes:
- name: httpbin
backends:
- host: 127.0.0.1:8000
Start agentgateway
In one terminal, run agentgateway with your config file.
agentgateway -f config.yaml
Notice a log line that confirms the admin server is listening on port 15000.
info app serving UI at http://localhost:15000/ui
info proxy::gateway started bind bind="bind/3000"
Trace a request from another client (watch mode)
After starting agentgateway, trace a request from another client.
-
In a second terminal from where agentgateway runs, start a watch.
agctl proxy trace --local --raw -
In a third terminal, send a request to agentgateway.
curl http://127.0.0.1:3000/headers -
In the watch terminal, review the JSON trace output of the request, such as the following.
{"eventStart":null,"eventEnd":3,"severity":"info","message":{"type":"requestStarted"}}{"eventStart":null,"eventEnd":462,"severity":"info","message":{"type":"requestSnapshot","stage":"initial request","requestState":{"request":{"method":"GET","uri":"http://127.0.0.1:3000/headers","path":"/headers","host":"127.0.0.1:3000",...},"env":{},"source":{"address":"::1","port":50946,...}}}}{"eventStart":null,"eventEnd":484,"severity":"info","message":{"type":"policySelection","phase":"gateway","effectivePolicy":{...}}}{"eventStart":null,"eventEnd":499,"severity":"info","message":{"type":"requestSnapshot","stage":"gateway policies","requestState":{...}}}{"eventStart":null,"eventEnd":507,"severity":"info","message":{"type":"routeSelection","selectedRoute":"default/default/bind/3000/listener0/default/httpbin","evaluatedRoutes":["default/default/bind/3000/listener0/default/httpbin"]}}{"eventStart":null,"eventEnd":522,"severity":"info","message":{"type":"policySelection","phase":"route","effectivePolicy":{...}}}{"eventStart":null,"eventEnd":537,"severity":"info","message":{"type":"requestSnapshot","stage":"route policies","requestState":{...}}}{"eventStart":null,"eventEnd":550,"severity":"info","message":{"type":"policySelection","phase":"backend","effectivePolicy":{...}}}{"eventStart":null,"eventEnd":1440,"severity":"info","message":{"type":"requestSnapshot","stage":"final request","requestState":{...,"backend":{"name":"default/default/bind/3000/listener0/default/httpbin/backend0","type":"static","protocol":"http"}}}}{"eventStart":null,"eventEnd":1457,"severity":"info","message":{"type":"backendCallStart","target":"127.0.0.1:8000"}}{"eventStart":null,"eventEnd":3080,"severity":"info","message":{"type":"bodySnapshot","stage":"final request","body":""}}{"eventStart":1470,"eventEnd":4890,"severity":"info","message":{"type":"backendCallResult","status":200,"error":null}}{"eventStart":null,"eventEnd":4905,"severity":"info","message":{"type":"responseSnapshot","stage":"raw response","requestState":{"response":{"code":200,...},"proxy":{"requestProcessingDuration":"0.001454916s","upstreamDuration":"0.003417333s"},"env":{}}}}{"eventStart":null,"eventEnd":4921,"severity":"info","message":{"type":"responseSnapshot","stage":"backend response ready","requestState":{"response":{"code":200,...}}}}{"eventStart":null,"eventEnd":4933,"severity":"info","message":{"type":"responseSnapshot","stage":"final response","requestState":{"response":{"code":200,...}}}}{"eventStart":null,"eventEnd":5300,"severity":"info","message":{"type":"bodySnapshot","stage":"response","body":"PCFET0NUWVBFIEhUTUw+Cjxod...(base64, truncated)"}}{"eventStart":null,"eventEnd":5305,"severity":"info","message":{"type":"requestFinished"}}
The trace records the following events for each request. Some events depend on the build and the policies that you configure, so a given trace might not include every row.
| Event type | Stage | What it tells you |
|---|---|---|
requestStarted | — | The proxy accepted a new request. |
requestSnapshot | initial request | The request as it arrived, before any processing. |
policySelection | — | The merged effective policy that applies to the request. Newer builds report this once per phase (gateway, route, and backend). |
requestSnapshot | gateway policies | The request after gateway-level policies ran. |
routeSelection | — | The route that matched the request, and the routes that were evaluated. |
requestSnapshot | route policies | The request after route-level policies ran. |
requestSnapshot | final request | The request as it is sent to the backend, including the resolved backend. |
backendCallStart | — | The proxy began the upstream call. |
bodySnapshot | final request | The request body sent to the backend, base64-encoded. Emitted when a body is present. |
backendCallResult | — | The upstream returned a status. The error field carries any transport error. |
responseSnapshot | raw response | The response exactly as the backend returned it. Includes a proxy block with requestProcessingDuration and upstreamDuration timings. |
responseSnapshot | backend response ready | The response from the backend, before any response processing. |
responseSnapshot | final response | The response as it is returned to the client. |
bodySnapshot | response | The response body, base64-encoded. Emitted when a body is present. |
requestFinished | — | The proxy completed the request. |
Trace a request that agctl sends (inject mode)
Run agctl proxy trace with --port and a request URL after --. agctl enables tracing and
sends the request itself, so you do not need a separate client.
agctl proxy trace --local --raw --port 3000 -- http://example.com/headers
The host portion of the URL (example.com) sets the Host header on the request but is not used
for DNS resolution. agctl always sends the request to 127.0.0.1:<port>.
You can pass extra curl arguments after the URL, such as headers or a request body.
agctl proxy trace --local --raw --port 3000 -- http://example.com/post \
-X POST \
-H "Content-Type: application/json" \
-d '{"key":"value"}'
Open the interactive TUI
Omit --raw to render the trace in an interactive, text-based terminal user interface (TUI) that
lets you step through each event and drill into the request and response state.
agctl proxy trace --local --port 3000 -- http://example.com/headers
Example TUI:
╔═════════════════════Events gateway/agentgateway-proxy═════════════════════╗┌──────────────────── Raw Event ───────────────────┐
║# Type Summary ║│eventEnd: 1112 │
║1 Request Start request started ║│eventStart: null │
║2 Snapshot initial request ║│message: │
║3 Snapshot gateway policies ║│ type: requestFinished │
║4 Route selected "httpbin/httpbin.00.http" (2 evaluated) ║│severity: info │
║5 Policies effective policies: apiKey, authorization, basicAuth, co…║│ │
║6 Snapshot route policies ║│ │
║7 Snapshot final request ║│ │
║8 Backend Start 10.244.0.7:8080 ║│ │
║9 Backend Result status=200 ║│ │
║10 Snapshot backend response ready ║│ │
║11 Snapshot final response ║│ │
║12 Request Done request finished ║│ │
║ ║│ │
║ ║│ │
║ ║│ │
║ ║│ │
║ ║│ │
╚═══════════════════════════════════════════════════════════════════════════╝└──────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────Help──────────────────────────────────────────────────────────────┐
│stream complete, press q to exit │
│tab: switch pane (events) arrows: scroll selected pane e/s/d: detail mode q: quit │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Press q to quit the TUI.