Skip to main content
OpenSRE uses Grafana Tempo to investigate trace-related alerts — searching spans by service, fetching full traces by ID, listing instrumented services, and filtering by error status or latency. This integration talks to Tempo directly via its HTTP API. It does not require a Grafana instance or datasource proxy. If you run the full Grafana stack, the Grafana integration already surfaces Tempo through the datasource proxy — use this integration when you run Tempo standalone.

​
Prerequisites

  • Grafana Tempo 1.4+
  • Network access from the OpenSRE environment to your Tempo instance
  • Auth credentials only if your deployment requires them (many run without auth behind a gateway)

​
Setup

​
Option 1: Interactive CLI

opensre integrations setup tempo
You will be prompted for the Tempo URL and optional auth. Leave auth fields blank if your Tempo runs without authentication.

​
Option 2: Environment variables

Add to your .env: 
TEMPO_URL=http://localhost:3200
TEMPO_API_KEY=<bearer-token>       # optional
TEMPO_USERNAME=<username>          # optional (basic auth)
TEMPO_PASSWORD=<token>             # optional (basic auth)
TEMPO_ORG_ID=<tenant-id>           # optional (X-Scope-OrgID for multi-tenant)
VariableDefaultDescription
TEMPO_URL—Required. Tempo HTTP API base URL
TEMPO_API_KEY(empty)Bearer token for auth
TEMPO_USERNAME(empty)Username for basic auth
TEMPO_PASSWORD(empty)Password for basic auth
TEMPO_ORG_ID(empty)Tenant ID sent as X-Scope-OrgID for multi-tenant deployments

​
Option 3: Persistent store

Integrations are automatically persisted to ~/.opensre/integrations.json: 
{
  "version": 1,
  "integrations": [
    {
      "id": "tempo-prod",
      "service": "tempo",
      "status": "active",
      "credentials": {
        "url": "http://localhost:3200",
        "api_key": ""
      }
    }
  ]
}

​
Investigation tools

OpenSRE exposes a single query_tempo tool with an action parameter: Searches traces by service, span name, duration, and tags using TraceQL. Returns one summary row per trace. 
query_tempo(action="search", service="checkout-service", min_duration_ms=500)
query_tempo(action="search", tags={"http.status_code": "500"})
ParameterDescription
serviceFilter by resource.service.name
span_nameFilter by span name
min_duration_msMinimum trace duration
max_duration_msMaximum trace duration
tagsKey/value span attributes
time_range_minutesLookback window (default 60)
limitMax traces to return (default 20)

​
get_trace

Fetches a full trace by ID and flattens its spans. 
query_tempo(action="get_trace", trace_id="4f8c...e21")

​
list_services

Lists all services registered in Tempo. 
query_tempo(action="list_services")

​
list_span_names

Lists all span names registered in Tempo. 
query_tempo(action="list_span_names")

​
Verify

opensre integrations verify tempo
Expected output: 
Service: tempo
Status:  passed
Detail:  Connected to Grafana Tempo HTTP API (/api/search, /api/traces).

​
Troubleshooting

SymptomFix
Connection refusedVerify the URL and port. Default Tempo HTTP port is 3200.
HTTP 401Set TEMPO_API_KEY or TEMPO_USERNAME/TEMPO_PASSWORD if your deployment requires auth.
HTTP 404 on verifyCheck your Tempo version — the /api/search/tags endpoint requires Tempo 1.4+.
No traces returnedConfirm your services are sending traces to Tempo and the time range covers the period of interest.
Multi-tenant 400Set TEMPO_ORG_ID to the correct tenant ID.

​
Security best practices

  • Use a read-only token or service account if your Tempo deployment supports auth.
  • Store credentials in .env, never in code.
  • Restrict network access to Tempo — OpenSRE only needs the HTTP API port (3200 by default).