Grafana Tempo

OpenSRE uses Grafana Tempo to investigate trace-related alerts — searching spans by service, fetching full traces by ID, listing instrumented services, and filtering by error status or latency. This integration talks to Tempo directly via its HTTP API. It does not require a Grafana instance or datasource proxy. If you run the full Grafana stack, the Grafana integration already surfaces Tempo through the datasource proxy — use this integration when you run Tempo standalone.

Prerequisites

Grafana Tempo 1.4+
Network access from the OpenSRE environment to your Tempo instance
Auth credentials only if your deployment requires them (many run without auth behind a gateway)

Setup

Option 1: Interactive CLI

opensre integrations setup tempo

You will be prompted for the Tempo URL and optional auth. Leave auth fields blank if your Tempo runs without authentication.

Option 2: Environment variables

Add to your .env:

TEMPO_URL=http://localhost:3200
TEMPO_API_KEY=<bearer-token>       # optional
TEMPO_USERNAME=<username>          # optional (basic auth)
TEMPO_PASSWORD=<token>             # optional (basic auth)
TEMPO_ORG_ID=<tenant-id>           # optional (X-Scope-OrgID for multi-tenant)

Variable	Default	Description
`TEMPO_URL`	—	Required. Tempo HTTP API base URL
`TEMPO_API_KEY`	(empty)	Bearer token for auth
`TEMPO_USERNAME`	(empty)	Username for basic auth
`TEMPO_PASSWORD`	(empty)	Password for basic auth
`TEMPO_ORG_ID`	(empty)	Tenant ID sent as `X-Scope-OrgID` for multi-tenant deployments

Option 3: Persistent store

Integrations are automatically persisted to ~/.opensre/integrations.json:

{
  "version": 1,
  "integrations": [
    {
      "id": "tempo-prod",
      "service": "tempo",
      "status": "active",
      "credentials": {
        "url": "http://localhost:3200",
        "api_key": ""
      }
    }
  ]
}

Investigation tools

OpenSRE exposes a single query_tempo tool with an action parameter:

search

Searches traces by service, span name, duration, and tags using TraceQL. Returns one summary row per trace.

query_tempo(action="search", service="checkout-service", min_duration_ms=500)
query_tempo(action="search", tags={"http.status_code": "500"})

Parameter	Description
`service`	Filter by `resource.service.name`
`span_name`	Filter by span name
`min_duration_ms`	Minimum trace duration
`max_duration_ms`	Maximum trace duration
`tags`	Key/value span attributes
`time_range_minutes`	Lookback window (default 60)
`limit`	Max traces to return (default 20)

get_trace

Fetches a full trace by ID and flattens its spans.

query_tempo(action="get_trace", trace_id="4f8c...e21")

list_services

Lists all services registered in Tempo.

query_tempo(action="list_services")

list_span_names

Lists all span names registered in Tempo.

query_tempo(action="list_span_names")

Verify

opensre integrations verify tempo

Expected output:

Service: tempo
Status:  passed
Detail:  Connected to Grafana Tempo HTTP API (/api/search, /api/traces).

Troubleshooting

Symptom	Fix
Connection refused	Verify the URL and port. Default Tempo HTTP port is `3200`.
HTTP 401	Set `TEMPO_API_KEY` or `TEMPO_USERNAME`/`TEMPO_PASSWORD` if your deployment requires auth.
HTTP 404 on verify	Check your Tempo version — the `/api/search/tags` endpoint requires Tempo 1.4+.
No traces returned	Confirm your services are sending traces to Tempo and the time range covers the period of interest.
Multi-tenant 400	Set `TEMPO_ORG_ID` to the correct tenant ID.

Security best practices

Use a read-only token or service account if your Tempo deployment supports auth.
Store credentials in .env, never in code.
Restrict network access to Tempo — OpenSRE only needs the HTTP API port (3200 by default).

Overview

LLM providers

Observability and incidents

Cloud, code, and collaboration

Messaging

Data and workflow systems

Prerequisites

Setup

Option 1: Interactive CLI

Option 2: Environment variables

Option 3: Persistent store

Investigation tools

search

get_trace

list_services

list_span_names

Verify

Troubleshooting

Security best practices

​Prerequisites

​Setup

​Option 1: Interactive CLI

​Option 2: Environment variables

​Option 3: Persistent store

​Investigation tools

​search

​get_trace

​list_services

​list_span_names

​Verify

​Troubleshooting

​Security best practices

Prerequisites

Setup

Option 1: Interactive CLI

Option 2: Environment variables

Option 3: Persistent store

Investigation tools

search

get_trace

list_services

list_span_names

Verify

Troubleshooting

Security best practices