> ## Documentation Index > Fetch the complete documentation index at: https://opensre.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # OTel payment auto-triage pilot > Run the payment demo, understand what OpenSRE watches, and receive failure and recovery reports. This opt-in pilot runs the official OpenTelemetry Demo payment service, one ClickStack Collector, and ClickHouse. OpenSRE checks the payment error rate, investigates a new incident, and reports its recovery. It does not change your application or require a dashboard. Existing vendor integrations are unchanged. ## How the pieces fit together ```text theme={null} Payment service (standard OpenTelemetry SDK) → Collector: receives telemetry through OTLP → ClickHouse: stores metrics, traces, and logs OpenSRE triage watch → reads recent payment metrics from ClickHouse every 30 seconds → on a new incident, reads a small set of related traces and logs → asks the configured API model to summarize that evidence → saves the report locally and sends it to Telegram, if enabled ``` **OpenSRE reads ClickHouse, not the Collector.** The Collector receives and forwards telemetry; it is not the history store. You configure these separately: | File | What you configure | | - | - | | `examples/telemetry-payment/collector.yaml` | How the Collector receives telemetry, derives request counts, and writes to ClickHouse. | | `examples/telemetry-payment/watch.yaml` | Where OpenSRE reads the stored data, which service to watch, and when to open or recover an incident. | Raw telemetry stays in ClickHouse. The local `watch.db` SQLite file stores incident state, reports, and pending deliveries—not the telemetry stream. The Collector's exporter creates and fills the telemetry tables; OpenSRE's read-only account does not create them or manage their schema. ## Start the demo Use a source checkout with Docker Compose and a configured OpenSRE API model. A tool-enabled model CLI is not supported for this bounded investigation. For Telegram, first run `uv run opensre integrations setup telegram` and choose your own chat or a group the bot can access—not another bot's account. Omit `--telegram` for terminal-only reports. From the repository root: ```bash theme={null} uv run python examples/telemetry-payment/setup.py init docker compose -f examples/telemetry-payment/compose.yaml up -d ``` `init` creates random demo credentials in an owner-readable `.env` and starts with the fault disabled. It refuses to overwrite an existing credential file. Keep the checkout in a directory shared with Docker Desktop. Port `18123` must be free. Only ClickHouse HTTP is published, bound to `127.0.0.1`; OTLP, payment, and flagd stay inside the Compose network. Load the demo reader credential in a Bash-compatible terminal, then start the watch: ```bash theme={null} set -a source examples/telemetry-payment/.env set +a uv run opensre triage watch \ --config examples/telemetry-payment/watch.yaml \ --state examples/telemetry-payment/watch.db \ --telegram ``` The foreground watch polls every 30 seconds. `--once` processes one completed window and exits. `--chat-id` optionally overrides the configured Telegram destination. Stop with Ctrl-C; retained reports and pending deliveries survive restart. Run the same command with the same state file to resume. Changing source, rule, bot identity, or destination requires a new state file. ## Understand the watch configuration The included `watch.yaml` is ready for this demo: ```yaml theme={null} source: backend: clickhouse host: 127.0.0.1 port: 18123 database: default username: opensre_reader watch: service: payment environment: demo # Explicitly map the collector's unsampled, delta SERVER-span counter. metric: name: traces.span.metrics.calls status_attribute: status.code error_value: STATUS_CODE_ERROR kind_attribute: span.kind server_value: SPAN_KIND_SERVER window_seconds: 30 ingestion_delay_seconds: 15 minimum_requests: 5 error_ratio: 0.2 healthy_windows: 2 ``` Read this as: **“Check payment in demo every 30 seconds. With at least five requests, open an incident if 20% or more fail. Recover only after two consecutive healthy windows.”** | Setting | Meaning | | - | - | | `source` | The ClickHouse connection, using a read-only account. Port `18123` is ClickHouse HTTP, not an OTLP port. | | `service`, `environment` | Limit all metric, trace, and log reads to this service and environment. | | `metric` | Identify the request counter, its error label, and the label selecting incoming server requests. Keep this mapping unchanged for the demo. | | `window_seconds` | How much data each check covers and how often the watch checks. | | `ingestion_delay_seconds` | Wait before checking a completed window so telemetry has time to arrive. Late data can still be missed. | | `minimum_requests` | Below this request count, the result is unknown—not healthy. | | `error_ratio` | The fraction of requests that must fail to open an incident: `0.2` means 20%. | | `healthy_windows` | How many consecutive below-threshold windows, each with enough requests, are needed to recover. | Keep passwords out of YAML. The setup command creates the demo password; loading `.env` above supplies `OPENSRE_TELEMETRY_CLICKHOUSE_PASSWORD` to OpenSRE. ### What happens on each check? For example, a check at **12:01:15** examines **12:00:30–12:01:00**: a completed 30-second window with a 15-second allowance for ingestion. | Result in that window | What OpenSRE does | | - | - | | 30 requests, no errors, no open incident | Prints `Telemetry: healthy`; no model call or Telegram report. | | 30 requests, 30 errors, no open incident | Opens one incident, reads up to ten span rows and ten logs, and produces an opening report. | | Errors stay at or above 20% during that incident | Keeps the incident open; does not repeat model analysis or create another opening report. | | Two consecutive windows below 20%, each with at least five requests | Marks the incident recovered and produces a recovery report, without another model call. | | Missing data, fewer than five requests, or a failed query | Prints `Telemetry: unknown`; does not declare recovery. An unknown window resets the recovery streak. | This watches **application telemetry, not Collector health**. An unknown result does not tell you whether the Collector stopped, the application stopped, or there was simply no traffic. Previously stored data may remain eligible briefly after collection stops; missing data is not detected instantly. Keep the watch terminal running: closing it pauses detection even if the Collector keeps storing data. Restarting resumes recent checks, but does not backfill missed windows. A slow investigation can also skip windows. ## Reproduce failure and recovery In another terminal: ```bash theme={null} uv run python examples/telemetry-payment/setup.py fail # Wait for an OPENED report (normally one or two completed metric windows). uv run python examples/telemetry-payment/setup.py recover # Wait for two consecutive healthy windows and a RECOVERED report. ``` If a Docker Desktop bind mount does not propagate the flag change, run `docker compose -f examples/telemetry-payment/compose.yaml restart flagd`. The traffic generator sends synthetic requests to the unmodified upstream payment service. The official `paymentFailure` flag rejects these requests; `emitRawPii` stays disabled. No real payment provider is contacted. View complete retained reports without contacting ClickHouse or Telegram: ```bash theme={null} uv run opensre triage reports --state examples/telemetry-payment/watch.db ``` Inside the REPL, use `/triage reports --state `. The continuous watch runs in a separate terminal, not inside the REPL. Telegram delivery retries the saved report, not the completed model analysis. Delivery is at-least-once: a crash after Telegram accepts a message but before OpenSRE saves its acknowledgment can produce a duplicate. Notifications carry an incident ID. Keep the local state file private; it contains investigation reports. ## Measurement and investigation limits The demo's application counter counts successful payments only. The Collector therefore derives a delta request counter from **unsampled SERVER spans**. `watch.yaml` explicitly maps that metric and its status/kind dimensions. Do not use this rule on sampled traces or substitute a cumulative counter: the reader rejects non-delta data. An application-native all-requests/error counter is a better production source when available, but requires an appropriate mapping. The model sees only the selected evidence from the incident's service, environment, and time window, with a 90-second analysis deadline. It cannot run tools, execute SQL, or make changes to the application. If evidence or the model is unavailable, OpenSRE saves a partial report with the cause marked unknown. A report is an evidence-based explanation, not a guaranteed root cause. The span sample is not a collection of complete distributed traces. ## Connect your own application Use standard OpenTelemetry instrumentation and OTLP. Supply `service.name` and `deployment.environment.name`. Your SDK or existing Collector can send to a secured Collector configured to write the same tables. This demo's private, unauthenticated OTLP listeners must not be exposed publicly. The included Collector configuration is a complete, pinned ClickStack Map-schema configuration, not a HyperDX SDK requirement. OpenSRE is not an OTLP receiver: ClickHouse supplies the queryable history. Another backend needs an implementation of the evidence-reader contract; OTLP alone does not standardize historical queries. The current adapter supports the included Map-schema tables, not arbitrary ClickHouse or ClickStack JSON schemas. For another ClickHouse source, copy `watch.yaml`, set its connection and rule, and provide `OPENSRE_TELEMETRY_CLICKHOUSE_PASSWORD` through the environment or OpenSRE's credential store. Use a dedicated `readonly=2` account with SELECT only on `otel_metrics_sum`, `otel_traces`, and `otel_logs`; never use collector or administrator credentials. Use TLS for remote connections. Known credentials, card-like numbers, and emails are redacted before model synthesis. This is not comprehensive DLP: remove sensitive content at collection time before using real telemetry with a model or Telegram. Resource/attribute maps are not exported into the report. Treat this as a single-owner reference pilot, not a hosted multi-tenant ingestion service. ## Stop and remove demo data ```bash theme={null} docker compose -f examples/telemetry-payment/compose.yaml down # Optional, destructive: remove this demo's telemetry volume too. docker compose -f examples/telemetry-payment/compose.yaml down -v ``` Telemetry has a 24-hour TTL; expiry is asynchronous. Local reports persist until you remove the chosen SQLite state file and its sidecars. Preserve that state if you intend to resume the same watch. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.