OTel payment auto-triage pilot

This opt-in pilot runs the official OpenTelemetry Demo payment service, one ClickStack Collector, and ClickHouse. OpenSRE checks the payment error rate, investigates a new incident, and reports its recovery. It does not change your application or require a dashboard. Existing vendor integrations are unchanged.

How the pieces fit together

Payment service (standard OpenTelemetry SDK)
  → Collector: receives telemetry through OTLP
  → ClickHouse: stores metrics, traces, and logs

OpenSRE triage watch
  → reads recent payment metrics from ClickHouse every 30 seconds
  → on a new incident, reads a small set of related traces and logs
  → asks the configured API model to summarize that evidence
  → saves the report locally and sends it to Telegram, if enabled

OpenSRE reads ClickHouse, not the Collector. The Collector receives and forwards telemetry; it is not the history store. You configure these separately:

File	What you configure
`examples/telemetry-payment/collector.yaml`	How the Collector receives telemetry, derives request counts, and writes to ClickHouse.
`examples/telemetry-payment/watch.yaml`	Where OpenSRE reads the stored data, which service to watch, and when to open or recover an incident.

Raw telemetry stays in ClickHouse. The local watch.db SQLite file stores incident state, reports, and pending deliveries—not the telemetry stream. The Collector’s exporter creates and fills the telemetry tables; OpenSRE’s read-only account does not create them or manage their schema.

Start the demo

Use a source checkout with Docker Compose and a configured OpenSRE API model. A tool-enabled model CLI is not supported for this bounded investigation. For Telegram, first run uv run opensre integrations setup telegram and choose your own chat or a group the bot can access—not another bot’s account. Omit --telegram for terminal-only reports. From the repository root:

uv run python examples/telemetry-payment/setup.py init
docker compose -f examples/telemetry-payment/compose.yaml up -d

init creates random demo credentials in an owner-readable .env and starts with the fault disabled. It refuses to overwrite an existing credential file. Keep the checkout in a directory shared with Docker Desktop. Port 18123 must be free. Only ClickHouse HTTP is published, bound to 127.0.0.1; OTLP, payment, and flagd stay inside the Compose network. Load the demo reader credential in a Bash-compatible terminal, then start the watch:

set -a
source examples/telemetry-payment/.env
set +a
uv run opensre triage watch \
  --config examples/telemetry-payment/watch.yaml \
  --state examples/telemetry-payment/watch.db \
  --telegram

The foreground watch polls every 30 seconds. --once processes one completed window and exits. --chat-id optionally overrides the configured Telegram destination. Stop with Ctrl-C; retained reports and pending deliveries survive restart. Run the same command with the same state file to resume. Changing source, rule, bot identity, or destination requires a new state file.

Understand the watch configuration

The included watch.yaml is ready for this demo:

source:
  backend: clickhouse
  host: 127.0.0.1
  port: 18123
  database: default
  username: opensre_reader
watch:
  service: payment
  environment: demo
  # Explicitly map the collector's unsampled, delta SERVER-span counter.
  metric:
    name: traces.span.metrics.calls
    status_attribute: status.code
    error_value: STATUS_CODE_ERROR
    kind_attribute: span.kind
    server_value: SPAN_KIND_SERVER
  window_seconds: 30
  ingestion_delay_seconds: 15
  minimum_requests: 5
  error_ratio: 0.2
  healthy_windows: 2

Read this as: “Check payment in demo every 30 seconds. With at least five requests, open an incident if 20% or more fail. Recover only after two consecutive healthy windows.”

Setting	Meaning
`source`	The ClickHouse connection, using a read-only account. Port `18123` is ClickHouse HTTP, not an OTLP port.
`service`, `environment`	Limit all metric, trace, and log reads to this service and environment.
`metric`	Identify the request counter, its error label, and the label selecting incoming server requests. Keep this mapping unchanged for the demo.
`window_seconds`	How much data each check covers and how often the watch checks.
`ingestion_delay_seconds`	Wait before checking a completed window so telemetry has time to arrive. Late data can still be missed.
`minimum_requests`	Below this request count, the result is unknown—not healthy.
`error_ratio`	The fraction of requests that must fail to open an incident: `0.2` means 20%.
`healthy_windows`	How many consecutive below-threshold windows, each with enough requests, are needed to recover.

Keep passwords out of YAML. The setup command creates the demo password; loading .env above supplies OPENSRE_TELEMETRY_CLICKHOUSE_PASSWORD to OpenSRE.

What happens on each check?

For example, a check at 12:01:15 examines 12:00:30–12:01:00: a completed 30-second window with a 15-second allowance for ingestion.

Result in that window	What OpenSRE does
30 requests, no errors, no open incident	Prints `Telemetry: healthy`; no model call or Telegram report.
30 requests, 30 errors, no open incident	Opens one incident, reads up to ten span rows and ten logs, and produces an opening report.
Errors stay at or above 20% during that incident	Keeps the incident open; does not repeat model analysis or create another opening report.
Two consecutive windows below 20%, each with at least five requests	Marks the incident recovered and produces a recovery report, without another model call.
Missing data, fewer than five requests, or a failed query	Prints `Telemetry: unknown`; does not declare recovery. An unknown window resets the recovery streak.

This watches application telemetry, not Collector health. An unknown result does not tell you whether the Collector stopped, the application stopped, or there was simply no traffic. Previously stored data may remain eligible briefly after collection stops; missing data is not detected instantly. Keep the watch terminal running: closing it pauses detection even if the Collector keeps storing data. Restarting resumes recent checks, but does not backfill missed windows. A slow investigation can also skip windows.

Reproduce failure and recovery

In another terminal:

uv run python examples/telemetry-payment/setup.py fail
# Wait for an OPENED report (normally one or two completed metric windows).
uv run python examples/telemetry-payment/setup.py recover
# Wait for two consecutive healthy windows and a RECOVERED report.

If a Docker Desktop bind mount does not propagate the flag change, run docker compose -f examples/telemetry-payment/compose.yaml restart flagd. The traffic generator sends synthetic requests to the unmodified upstream payment service. The official paymentFailure flag rejects these requests; emitRawPii stays disabled. No real payment provider is contacted. View complete retained reports without contacting ClickHouse or Telegram:

uv run opensre triage reports --state examples/telemetry-payment/watch.db

Inside the REPL, use /triage reports --state <file>. The continuous watch runs in a separate terminal, not inside the REPL. Telegram delivery retries the saved report, not the completed model analysis. Delivery is at-least-once: a crash after Telegram accepts a message but before OpenSRE saves its acknowledgment can produce a duplicate. Notifications carry an incident ID. Keep the local state file private; it contains investigation reports.

Measurement and investigation limits

The demo’s application counter counts successful payments only. The Collector therefore derives a delta request counter from unsampled SERVER spans. watch.yaml explicitly maps that metric and its status/kind dimensions. Do not use this rule on sampled traces or substitute a cumulative counter: the reader rejects non-delta data. An application-native all-requests/error counter is a better production source when available, but requires an appropriate mapping. The model sees only the selected evidence from the incident’s service, environment, and time window, with a 90-second analysis deadline. It cannot run tools, execute SQL, or make changes to the application. If evidence or the model is unavailable, OpenSRE saves a partial report with the cause marked unknown. A report is an evidence-based explanation, not a guaranteed root cause. The span sample is not a collection of complete distributed traces.

Connect your own application

Use standard OpenTelemetry instrumentation and OTLP. Supply service.name and deployment.environment.name. Your SDK or existing Collector can send to a secured Collector configured to write the same tables. This demo’s private, unauthenticated OTLP listeners must not be exposed publicly. The included Collector configuration is a complete, pinned ClickStack Map-schema configuration, not a HyperDX SDK requirement. OpenSRE is not an OTLP receiver: ClickHouse supplies the queryable history. Another backend needs an implementation of the evidence-reader contract; OTLP alone does not standardize historical queries. The current adapter supports the included Map-schema tables, not arbitrary ClickHouse or ClickStack JSON schemas. For another ClickHouse source, copy watch.yaml, set its connection and rule, and provide OPENSRE_TELEMETRY_CLICKHOUSE_PASSWORD through the environment or OpenSRE’s credential store. Use a dedicated readonly=2 account with SELECT only on otel_metrics_sum, otel_traces, and otel_logs; never use collector or administrator credentials. Use TLS for remote connections. Known credentials, card-like numbers, and emails are redacted before model synthesis. This is not comprehensive DLP: remove sensitive content at collection time before using real telemetry with a model or Telegram. Resource/attribute maps are not exported into the report. Treat this as a single-owner reference pilot, not a hosted multi-tenant ingestion service.

Stop and remove demo data

docker compose -f examples/telemetry-payment/compose.yaml down
# Optional, destructive: remove this demo's telemetry volume too.
docker compose -f examples/telemetry-payment/compose.yaml down -v

Telemetry has a 24-hour TTL; expiry is asynchronous. Local reports persist until you remove the chosen SQLite state file and its sidecars. Preserve that state if you intend to resume the same watch.

Welcome

Get started

CI/CD

Using the shell

Guides

Run it anywhere

Platform

Integrations

Community

How the pieces fit together

Start the demo

Understand the watch configuration

What happens on each check?

Reproduce failure and recovery

Measurement and investigation limits

Connect your own application

Stop and remove demo data

​How the pieces fit together

​Start the demo

​Understand the watch configuration

​What happens on each check?

​Reproduce failure and recovery

​Measurement and investigation limits

​Connect your own application

​Stop and remove demo data

How the pieces fit together

Start the demo

Understand the watch configuration

What happens on each check?

Reproduce failure and recovery

Measurement and investigation limits

Connect your own application

Stop and remove demo data