# ARCHITECTURE Source: https://opensre.com/docs/ARCHITECTURE # OpenSRE architecture How the OpenSRE codebase is structured: the eight first-party packages, what each is responsible for, and which may depend on which. These dependency rules are CI-enforced (`make check-imports`), so they are real invariants rather than aspirations. ## The layer stack The packages sit in five tiers. **Higher tiers may import lower tiers; a lower tier may never import a higher one.** Packages on the same tier are peers — the last column says whether peers may import each other. | Tier | Packages | May import | Must never import | Peer rule | | ---------- | --------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | 1 (top) | `surfaces`, `gateway` | `bootstrap`, `tools`, `integrations`, `core`, `platform`, `config` | — | Independent: must not import each other. | | 2 | `bootstrap` | `tools`, `integrations`, `core`, `platform`, `config` | `surfaces`, `gateway` | Composition root. The only package that may import `tools` **and** `integrations` together, because wiring needs both. | | 3 | `tools` | `core`, `platform`, `config` | `surfaces`, `gateway`, `bootstrap` | Peer of `integrations`: must not import it. Existing edges are listed as debt in `.importlinter.strict`. | | 3 | `integrations` | `core`, `platform`, `config` | `tools`, `surfaces`, `gateway`, `bootstrap` | Peer of `tools`: must not import it. | | 4 | `core`, `platform` | `config` | `surfaces`, `gateway`, `bootstrap`, `tools`, `integrations` | Siblings: **may** cross-import each other. | | 5 (bottom) | `config` | — (nothing first-party) | everything above | Independent — imports no other first-party package. | The shortcut: **dependencies point downward only.** A surface can reach all the way down; `config` can reach nothing. The single deliberate exception is `core ⟷ platform`, a mutually-dependent pair by design (see below). ```mermaid theme={null} flowchart TD subgraph T1["Tier 1 — hosts"] SURFACES[surfaces] GATEWAY[gateway] end subgraph T2["Tier 2 — composition root"] BOOT[bootstrap] end subgraph T3["Tier 3 — capability"] TOOLS[tools] INTEGRATIONS[integrations] end subgraph T4["Tier 4 — runtime + platform"] CORE[core] PLATFORM[platform] end subgraph T5["Tier 5 — config"] CONFIG[config] end SURFACES --> BOOT GATEWAY --> BOOT SURFACES --> TOOLS SURFACES --> INTEGRATIONS GATEWAY --> TOOLS GATEWAY --> INTEGRATIONS BOOT --> TOOLS BOOT --> INTEGRATIONS TOOLS --> CORE TOOLS --> PLATFORM INTEGRATIONS --> CORE INTEGRATIONS --> PLATFORM CORE <--> PLATFORM CORE --> CONFIG PLATFORM --> CONFIG ``` The arrows show edges between **adjacent** tiers to keep the diagram readable. The actual rule is broader: a tier may import **any** tier below it, not only the one directly beneath — so a surface may import `config` directly, and a tool may import `platform`. Refer to the "May import" column above for the complete set of allowed edges. ## The layers in detail ### Tier 1 — `surfaces` and `gateway` The entry points a human or an external system talks to. Nothing first-party may import from here, so a surface can be added or removed without touching the layers below it. * **`surfaces/`** — one folder per UI/client: `surfaces/cli` (the stateless `opensre ` runner), `surfaces/interactive_shell` (the stateful REPL), and `surfaces/shared` for code two or more surfaces use. A surface owns its own I/O, prompts, and presentation, and composes lower layers to do the actual work. Slack is not a surface: its inbound transport lives in `gateway/transports/slack`, outbound delivery in `integrations/slack`. * **`gateway/`** — the standalone messaging gateway for inbound chat platforms (`gateway/transports/telegram`, `gateway/transports/slack`, `gateway/core/session`, `gateway/core/storage`). A peer of `surfaces`, not a child: the two never import each other. ### Tier 2 — `bootstrap` The **composition root** for process and harness wiring. Hosts (`surfaces`, `gateway`) cannot import each other, and `tools` / `integrations` are peers that must not import each other — yet registering harness adapters needs both capability packages. `bootstrap/` is the only package allowed to import `tools` **and** `integrations` together. Every entrypoint boots through `configure_process()` from `bootstrap/process.py`: a profile (CLI, gateway, web, scheduler worker, embedded) selects which boot steps run, and the step order is fixed by this package — profiles cannot invent a different sequence. Adapter and scheduler-runner registration lives in `bootstrap/adapters.py` and nowhere else; surfaces and gateway used to keep peer copies — do not reintroduce them. Host-owned concerns stay out of this package: CLI/gateway logging, Rich product ports, CLI’s update-tolerant Sentry init. ### Tier 3 — `tools` and `integrations` The capability layer — "do a thing against the outside world" — split by responsibility: * **`integrations/`** — the boundary for **user config and external clients**: per-vendor config normalization, verification (`verifier.py`), API clients (`client.py`), the store/catalog that resolves credentials, and integration-local helpers. One folder per vendor (`integrations/datadog`, `integrations/grafana`, `integrations/github`, …) plus cross-cutting pieces like `integrations/hermes` and `integrations/llm_cli`. * **`tools/`** — the **agent-callable** boundary: every `@tool(...)` function and `BaseTool` subclass, the tool registry, framework subsystems (`tools/investigation`, `tools/interactive_shell`), `tools/system/` for tools with no vendor in their domain purpose (`fleet_monitoring`, `python_execution_tool`, `sre_guidance_tool`, `watch_dog`), and `tools/cross_vendor/` for tools whose logic spans 2+ vendor integrations (`fix_sentry_issue`). See [tool-placement-policy.md](tool-placement-policy.md) for the full decision rule, including when a tool belongs under `integrations//tools/` instead. A tool is what the planner selects and the runtime executes. The import rule between them is one-directional: `integrations` must never import `tools` (or `surfaces`), so a vendor client never depends on the agent layer and stays reusable on its own. The reverse edge is allowed and common — a tool reaches an integration's client for external data — so `integrations` effectively sits one step below `tools` in the dependency graph. Do **not** reintroduce top-level `vendors/` or `services/` packages — external-system code belongs in `integrations/`, agent-callable code in `tools/`. ### Tier 4 — `core` and `platform` The shared runtime and cross-cutting services the capability layer is built on. * **`core/`** — the provider-agnostic agent runtime: the think → call tools → observe loop (`core.agent.Agent`), agent/investigation state (`core/state`) and context-budget enforcement (`core/context_budget.py`), the tool framework primitives (`core/tool_framework`), shared LLM clients (`core/llm`), agent-harness session handling (`core/agent_harness`), and pure domain rules (`core/domain`). * **`platform/`** — cross-cutting services with no investigation logic of their own: guardrails, masking, sandbox, analytics, auth, notifications, observability, scheduler, and deployment. It deliberately shadows the stdlib `platform` name and re-exposes it, so `import platform` still works. These two are the one bidirectional pair by design: `core` reaches `platform` for guardrails, masking, observability, and evidence/log compaction, while `platform` reaches back into `core` for the shared state and session types (`core.state`, `core.agent_harness.session`). Splitting them into separate tiers would forbid that edge, so they share a tier as siblings. ### Tier 5 — `config` The floor: shared constants, prompts, and UI theme. Everything above may read from `config`, but `config` imports no other first-party package — keeping it a leaf means constants can be imported anywhere without dragging runtime along. ## Cross-layer flows Two worked examples showing how control descends the stack and results flow back up. Arrows only ever cross a boundary downward. ### An investigation from the CLI ```mermaid theme={null} flowchart LR A["surfaces/cli\n opensre investigate"] --> B["tools/investigation\n capability + lifecycle"] B --> C["core\n Agent runtime, context budget, LLM"] B --> D["integrations\n vendor clients + credentials"] C --> E["platform\n guardrails, masking, sandbox, observability"] B --> F["config\n prompts + constants"] ``` 1. `surfaces/cli` parses the command and hands off to the investigation capability in `tools/investigation` — the surface never runs pipeline logic itself. 2. `tools/investigation` drives the six-stage pipeline (see [`investigation-pipeline-architecture.md`](investigation-pipeline-architecture.md)), asking `core` to run the ReAct loop and select/execute tools. 3. Evidence-gathering tools reach `integrations` for vendor clients and resolved credentials; `core` and `platform` supply the runtime, guardrails, and masking around every call. 4. The structured diagnosis flows back up to the surface, which owns how it is presented or delivered. ### An inbound gateway message ```mermaid theme={null} flowchart LR A["gateway/transports\n inbound chat message"] --> B["gateway/core session + storage\n resolve conversation state"] B --> C["tools + core\n run the requested capability"] C --> D["platform\n notifications, observability"] ``` `gateway` receives a message, resolves session state from its own storage, then composes the same tier-3 capability code a surface would (after shared `bootstrap` process boot) — without ever importing `surfaces`, since the two are independent tier-1 peers. ## Related docs * [`AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/AGENTS.md) — repo map and per-area "files to touch" guides. * [`investigation-pipeline-architecture.md`](investigation-pipeline-architecture.md) — how a single investigation runs end-to-end within the `tools` + `core` layers. # DEVELOPMENT Source: https://opensre.com/docs/DEVELOPMENT # Development guide Contributor-focused workflows: local setup details stay in [SETUP.md](https://github.com/Tracer-Cloud/opensre/blob/main/SETUP.md) at the repo root (Windows, troubleshooting, MCP/OpenClaw). ## Clone and install ```bash theme={null} git clone https://github.com/Tracer-Cloud/opensre.git cd opensre make install ``` [`make install`](https://github.com/Tracer-Cloud/opensre/blob/main/Makefile) runs `uv sync --frozen --extra dev` and the analytics install helper. Use **`uv run opensre …`** from the repo root so you always hit this checkout’s `.venv`, not another `opensre` on your `PATH`. ```bash theme={null} opensre onboard opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` ## Quality gates (same as CI) From the repo root: ```bash theme={null} make lint # ruff check make format-check # ruff format --check (CI-enforced) make typecheck # mypy config core gateway integrations platform surfaces tools make test-cov # pytest + coverage (default unit suite) ``` One-shot (includes heavier `test-full`): `make check`. Before a PR, run at least `make lint`, `make format-check`, `make typecheck`, and `make test-cov` (see [CONTRIBUTING.md](https://github.com/Tracer-Cloud/opensre/blob/main/CONTRIBUTING.md)). ## Interactive shell action policy Action-planner behavior, postprocessing transforms, compatibility seams, and the rule-extension checklist are documented in [`docs/interactive-shell-action-policy.md`](https://github.com/Tracer-Cloud/opensre/blob/main/docs/interactive-shell-action-policy.md). ## Package architecture The eight first-party packages, the five-tier layering (which package may import which), the folder diagram, per-layer responsibilities, and cross-layer flows are documented in [`docs/ARCHITECTURE.md`](ARCHITECTURE.md). ## Tool registry — surface-scoped, lazy loading Loading every vendor tool at startup was slow. A static index (`tools/registry_index.py`) reads tool metadata by scanning the source, without importing executors, so a turn loads only the tools it needs. * `get_registered_tools(surface)` imports only that surface's tool modules. * `get_tool_descriptors(surface)` returns metadata with no executor import. * `load_tool(descriptor)` imports the executor, only when a tool runs. Adding a vendor tool is a `@tool`/`BaseTool` module; the index finds it and no other vendor is imported. `tests/tools/test_registry_index.py` checks the index matches the imported registry exactly, so they cannot drift. ## Investigation pipeline architecture The six-stage investigation pipeline (resolve integrations → extract alert → plan → ReAct evidence loop → diagnose → deliver), the loop's guardrails (tool cap, stagnation breaker, context budget, duplicate detection), and diagrams are documented in [`docs/investigation-pipeline-architecture.md`](investigation-pipeline-architecture.md). ## Investigation tool calling Tool schemas, provider adapters (`transports/sdk/agent_clients.py`), and investigation message shapes are documented in [`docs/investigation-tool-calling.md`](investigation-tool-calling.md) (all LLM providers, not vendor-specific). ## Interactive shell: REPL watchdog demo PR reviewers expect a **visible demo** (terminal log or screenshot) in the PR under **Demo/Screenshot**, not only tests. Copy the exact steps from this section into your PR description, then attach your terminal output or recording. 1. `uv run opensre` (TTY). 2. `/trust on` (or confirm the elevated-action prompt when running `/watch`). 3. `/watch --max-cpu 80` — expect `task … started.` (use a real PID, e.g. the shell’s Python process). 4. `/watches` — table columns include id, pid, kind, status, thresholds, last sample. 5. `/unwatch ` or `/cancel ` — then `/watches` again; status should show **cancelled**. 6. Optional: lower `--max-cpu` so a threshold trips; after delivery, the REPL prints one line: `[task …] alarm fired: … (telegram delivered)`. Add `--provider rocketchat --chat-id "#channel"` to `/watch` to alarm via Rocket.Chat instead (`… (rocketchat delivered)`). Automated equivalent (runs in `make test-cov`):\ `uv run pytest tests/interactive_shell/test_watchdog_repl_e2e_demo.py -v --tb=short` Longer transcript (optional): [tests/interactive\_shell/repl\_watchdog\_demo.md](https://github.com/Tracer-Cloud/opensre/blob/main/tests/interactive_shell/repl_watchdog_demo.md). ## VS Code dev container The dev container is defined under [`.devcontainer/`](https://github.com/Tracer-Cloud/opensre/tree/main/.devcontainer). It builds from [`.devcontainer/Dockerfile`](https://github.com/Tracer-Cloud/opensre/blob/main/.devcontainer/Dockerfile) (Python **3.13**), then **`postCreateCommand`** creates `.venv-devcontainer` and runs **`pip install -e '.[dev]'`** (not `uv`). Docker Desktop, OrbStack, Colima, or another compatible runtime must be available on the host. ## Benchmark ```bash theme={null} make benchmark ``` To refresh README benchmark copy from cached results (no LLM calls): `make benchmark-update-readme`. ## Deployment Full deployment instructions, prerequisites, and environment variable reference: **[DEPLOYMENT.md](../DEPLOYMENT.md)** Quick reference: | Path | Commands | | -------------------------------------- | --------------------------------------------------------------------------- | | Gateway (AMI + systemd — gateway only) | `make build-gateway-image` → `make deploy-gateway` / `make destroy-gateway` | | Hosted (Railway / ECS / Vercel) | Deploy with repo `Dockerfile`; set `LLM_PROVIDER` + API key | ### Hosted runtime (Railway / ECS / Vercel) 1. Deploy this repository as a standard Python/FastAPI app using the repo `Dockerfile` or your host's native Python workflow. 2. Set `LLM_PROVIDER` and the matching API key (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` — see [`.env.example`](https://github.com/Tracer-Cloud/opensre/blob/main/.env.example)). 3. Add `DATABASE_URI` and `REDIS_URI` for hosted layouts that need persistence. 4. Add integration and storage env vars your deployment needs. Minimal LLM env: ```bash theme={null} export LLM_PROVIDER=anthropic export ANTHROPIC_API_KEY=... ``` For Railway: ensure the project has Postgres and Redis services and that the OpenSRE service has `DATABASE_URI` and `REDIS_URI` set before deploying. Set `OPENSRE_DEPLOYMENT_METHOD=railway` for telemetry labeling. ## Telemetry and privacy `opensre` ships with two telemetry stacks, both opt-out: * **PostHog** — anonymous product analytics (commands used, success/failure, rough runtime, CLI/Python/OS/arch, and limited command metadata). * **Sentry** — crashes and errors (stack traces, environment, release). Events are tagged with `entrypoint`, `opensre.runtime`, and `deployment_method`. Sensitive headers, paths, and secret-shaped keys are scrubbed before send. PostHog product events also carry `execution_environment` (`local`, `ci`, `container`, or `ci_container`), `is_ci`, `is_container`, and `container_runtime`. Use these first-party fields to exclude automated environments from product funnels; PostHog's virtual traffic classification intentionally treats CLI HTTP clients as automation. A random install ID is stored under `~/.opensre/anonymous_id`. PostHog `distinct_id` is scoped to that ID. Telemetry is off in GitHub Actions and pytest. ### First-launch GitHub login On the first interactive launch (all platforms, except CI/CD and test harnesses), OpenSRE runs a GitHub device-flow sign-in gate before the REPL prompt. Installs are split offline into an A/B experiment via sticky bucketing on `~/.opensre/anonymous_id`: | Variant | Behavior | How to force locally | | --------- | ------------------------------------------------ | ------------------------------------- | | `control` | Skip allowed (menu + Escape defer the gate) | `OPENSRE_GITHUB_GATE_VARIANT=control` | | `forced` | Skip removed; abandoning the gate aborts startup | `OPENSRE_GITHUB_GATE_VARIANT=forced` | Every gate exposure emits `github_login_prompted` with `github_gate_variant`. Outcomes are `github_login_completed`, `github_login_skipped` (control only), or `github_login_abandoned` (forced drop-off). The variant is also stamped as a persistent event property so later REPL events break down by cohort. On success OpenSRE sets `github_username` as a PostHog **person property** (via `$identify`/`$set`, which forces `$process_person_profile: True` for that one event — this is the only intentional PII OpenSRE sends). A configured GitHub integration suppresses re-prompting on later launches. Because bucketing is local and does not call PostHog feature flags, PostHog's built-in experiment exposure chart will not populate automatically. Configure `github_login_prompted` as the custom exposure event and use `github_gate_variant` as the breakdown/filter for funnels and trends. The existing kill-switches still apply: `OPENSRE_NO_TELEMETRY` / `DO_NOT_TRACK` make analytics calls no-ops, but the login itself still runs. Set `OPENSRE_SKIP_GITHUB_LOGIN=1` to bypass the login gate entirely (also auto-bypassed in CI — `CI=true`, `GITHUB_ACTIONS=true` — and in pytest). ### Kill-switch matrix | Env var | PostHog | Sentry | | ----------------------------------- | ---------- | ----------------------------------------------------------------------------------------------- | | `OPENSRE_NO_TELEMETRY=1` | disabled | disabled | | `DO_NOT_TRACK=1` | disabled | disabled | | `OPENSRE_ANALYTICS_DISABLED=1` | disabled | unaffected | | `OPENSRE_SENTRY_DISABLED=1` | unaffected | disabled | | `OPENSRE_SENTRY_LOGGING_DISABLED=1` | unaffected | disables `logger.error`/`logger.exception` forwarding to Sentry; `capture_exception` unaffected | Full opt-out: ```bash theme={null} export OPENSRE_NO_TELEMETRY=1 ``` ### Sentry DSN Self-hosted users can set `SENTRY_DSN` to their project; unset uses the bundled default. `SENTRY_DSN=` (empty) drops events in `before_send`. ### Deployment tagging Set `OPENSRE_DEPLOYMENT_METHOD` to `railway`, `ec2`, `vercel`, or `local` (default `local`) to label Sentry events. ### Local PostHog event log By default, outbound PostHog payloads are also appended to `~/.opensre/posthog_events.txt` (rotates at 1000 lines). Disable: ```bash theme={null} export OPENSRE_ANALYTICS_LOG_EVENTS=0 ``` We do not collect alert contents, file contents, hostnames, credentials, raw CLI arguments, or PII by design. # NAMING Source: https://opensre.com/docs/NAMING # Naming conventions for `core/` A small, enforceable vocabulary so file and type names say what they are. The goal is that a reader can tell a data type from a process, a mutable state from a frozen view, and a package's purpose from its name alone. ## Glossary (one meaning per term) | Term | Means | Example | | ---------------------------- | ------------------------------------------------------------ | --------------------------------------------------------------- | | **State** | Mutable investigation/session facts that evolve during a run | `AgentState`, `InvestigationState`, harness port `SessionState` | | **Storage** / **Repo** | Durable persistence backends (not the in-memory turn port) | `SessionStore`, `SessionRepo`, `JsonlSessionStore` | | **Snapshot** | A frozen view captured at a boundary (turn start, run start) | `TurnSnapshot` | | **RunInput** / **RunResult** | The input to and output from one `Agent.run()` boundary | `AgentRunInput`, `AgentRunResult` | | **Slice** | A typed segment of a state dict | `DiagnosisSlice`, `AlertInputSlice` | | **Resources** | Handles passed into tool executors for one call | `ToolCallResources` | | **Budget** | An LLM token/window policy — not application state | `enforce_token_budget` | | **Host** | The callback contract an algorithm drives (a `Protocol`) | `LoopHost` | ## Module naming: `{domain}_{role}.py` Name a file for the concept it holds, not with a generic bucket word. ``` core/agent/ agent.py # the Agent facade react_loop.py # ReactLoop + run_react_loop (the algorithm) loop_host.py # LoopHost (the callback contract) run_io.py # AgentRunInput, AgentRunResult (the run boundary's I/O) mixins.py # the reusable *Mixin behaviors provider_hooks.py # ProviderHookDelegate ``` ## Type naming * **Mixins** carry a `Mixin` suffix — they cannot stand alone (they assume fields/methods the host provides). `EventEmitterMixin`, `ToolFilterMixin`, `SteeringMixin`. * **Protocols** are named by their role, not with a `Protocol` suffix — matches the stdlib (`Iterable`, `SupportsRead`) and `agent_harness/ports.py` (`OutputSink`, `SessionState`). `LoopHost`, not `LoopHostProtocol`. * **Do not prefix a type with its own package name.** Inside `core/agent/`, a class is `EventEmitterMixin`, not `AgentEventEmitter` — the namespace already says "agent." ## Anti-patterns (do not add in new code) * `context.py` at `core/` or `core/agent/` root — "context" is overloaded across the repo. Name the concept (`run_io.py`, `turn_snapshot.py`). * `models.py` when the file holds only run I/O — too vague. Say what the models are (`run_io.py`). * `*Context` without a domain prefix when another `*Context` already exists. * A package whose only child is a single sub-package — collapse the wrapper. * Calling the mutable harness session port `SessionStore` — that word means durable persistence. The port is `SessionState` (`InMemorySessionState` for headless); JSONL durability stays `SessionStore` / `SessionRepo`. * Module-level mutable globals for “current session” — pass `SessionState` explicitly (see local notes: no-globals design). ## Imports Use fully qualified paths in code; keep short mental labels for docs. | Mental label | Import | | ---------------------- | ----------------------------------------------------------------- | | ReAct run I/O | `from core.agent.run_io import AgentRunInput, AgentRunResult` | | ReAct loop | `from core.agent.react_loop import run_react_loop` | | Loop callback contract | `from core.agent.loop_host import LoopHost` | | The agent primitive | `from core.agent import Agent` | | Harness turn snapshot | `from core.agent_harness.turns.turn_snapshot import TurnSnapshot` | Re-export from a package `__init__.py` only for its single canonical symbol (`Agent`), not everything — avoid `from core.agent import *`-style ambiguity. # Adding tools and integrations Source: https://opensre.com/docs/adding-tools-and-integrations # Adding Tools & Integrations — Definition of Done Use this checklist whenever you add or materially change: * a tool — under `integrations//tools/` for a single-vendor tool, or `tools/system/` / `tools/cross_vendor/` for a cross-cutting one (see [tool-placement-policy.md](tool-placement-policy.md)) * an integration under `integrations//` — its config, client, verifier, and tools * investigation source wiring for an existing tool or integration This is the detailed definition of done; use it with [AGENTS.md](../AGENTS.md) and [CI.md](../CI.md). ## 1. Tool checklist ### Files usually involved * `integrations//tools/_tool/__init__.py` — the tool package (most common path: the tool belongs to a vendor integration) * `tools/system//` or `tools/cross_vendor//` — only when the tool is not vendor-specific (e.g. `tools/system/sre_guidance_tool/`) * `integrations//client.py` — reuse a dedicated integration API client instead of inlining requests * `core/tool_framework/utils/` — shared helper code reused across vendors * `docs/.mdx` — user-facing usage, parameters, examples * `tests/tools/test_.py` — behavior and regression coverage Tools are registry-discovered from **both** `tools/` and `integrations//tools/`, so placement is about ownership, not discovery — see [tool-placement-policy.md](tool-placement-policy.md). Wherever a tool lives, it calls integration-local clients/helpers rather than inlining transport, and never lives in a top-level `vendors/` or `services/` package. Tool packages must be substantive production modules — no empty or discovery-only `__init__.py`, no thin wrapper that only satisfies registry import. Any tool with validation, credential/parameter resolution, transport/client calls, output normalization, or error handling should split those concerns into focused sibling files (`tool.py`, `models.py`, `validation.py`, `delivery.py`/`client.py`, `results.py`), leaving `__init__.py` as a small registry entrypoint that imports the public tool object. ### Contract and implementation * [ ] Pick the simplest shape that fits (`@tool(...)` for lightweight tools, a richer class only when needed) * [ ] `__init__.py` is a small registry entrypoint; non-trivial tools use sibling modules for implementation concerns * [ ] Metadata is complete and accurate: `name`, `description`, `source`, `surfaces`, `requires`, and any `use_cases` / `outputs` / `retrieval_controls` * [ ] `input_schema` matches the actual runtime arguments and required fields * [ ] `is_available` returns `True` only when the tool can genuinely run * [ ] `extract_params` maps resolved integration state into tool args correctly * [ ] Validation, credential/parameter resolution, transport/client calls, and result formatting are separated so each can be tested independently * [ ] Reusable transport or integration-specific parsing lives in `integrations//` or `core/tool_framework/utils/`, not copied into the tool body * [ ] Failure responses have a stable, investigation-friendly shape; expected external failures (missing config, auth, rate limit, upstream 4xx/5xx) return structured errors rather than raising — unexpected exceptions use the global `BaseTool` wrapper intentionally or are migrated with telemetry coverage * [ ] Output is normalized enough for the planner/LLM to consume reliably * [ ] Secrets never leak through `extract_params`, return values, logs, or traceable tool-call kwargs; secret/PII output is run through `platform/masking/` before return * [ ] External side effects declare `side_effect_level`, `requires_approval`, and `approval_reason` where appropriate * [ ] To appear in both investigation and chat, set `surfaces=(ToolSurface.INVESTIGATION, ToolSurface.CHAT)` ### Live payload parsing If the tool parses API, MCP, log, or webhook payloads: * [ ] Validate against the real or documented upstream response shape, not only idealized mocks * [ ] Handle alternate field names used in live payloads * [ ] Handle missing or partial fields without returning unusable output * [ ] Preserve important context when truncating, tailing, paginating, or flattening data * [ ] Upstream 429 / 5xx responses return a clear, investigation-friendly error rather than raising * [ ] Add at least one regression test using a realistic fixture payload Common failure modes to consider: grouped + ungrouped log content; nested/foldered resources; paginated responses; `hasMore` / cursor mismatches; content-vs-pointer shapes (`logs_content` vs `logs_url`-style payloads). ## 2. Integration checklist ### Files usually involved * `integrations//__init__.py` — config builders, validators, selectors, normalization helpers * `integrations//client.py` — a dedicated API client, when the integration makes direct remote calls * `integrations//verifier.py` — local verification logic * `integrations//tools/_tool/` — the vendor's agent-callable tools (see §1) * `integrations/catalog.py` — resolve the integration into the shared runtime config * `integrations/verify.py` — wire the local verification path * `docs/.mdx` — user-facing setup, usage, verification * `tests/integrations/test_.py`, plus `tests/tools/`, `tests/e2e/`, or `tests/synthetic/` where tools or scenarios exercise it `integrations//` owns everything about one vendor — config, resolution, clients, verifiers, helpers, **and its tools**. Only vendor-less (`tools/system/`) and cross-vendor (`tools/cross_vendor/`) tools live under top-level `tools/`. ### Examples from the repo * Datadog: `integrations/datadog/` (with `integrations/datadog/tools/`), `integrations/catalog.py`, tests under `tests/integrations/datadog/` and `tests/tools/test_datadog_*.py`. * Grafana: `integrations/grafana/` (with `integrations/grafana/tools/`), `integrations/catalog.py`, `surfaces/cli/wizard/local_grafana_stack/`, tests under `tests/integrations/grafana/` and `tests/tools/test_grafana_*.py`. * Hermes: `integrations/hermes/` (with `integrations/hermes/tools/hermes_logs_tool/` and `.../hermes_session_evidence_tool/`), `surfaces/cli/commands/hermes.py`, `tests/hermes/`, `tests/synthetic/hermes/`. ### Core completeness * [ ] Config, normalization, and validators are in place under `integrations//__init__.py` * [ ] Catalog resolution / env loading is wired correctly * [ ] Verification path is wired in `integrations/verify.py` and adapters/registry as needed * [ ] Integration-local client added under `integrations//client.py` (only if it makes direct remote calls) * [ ] Tool layer is wired and stable * [ ] CLI setup flow is updated if the integration is user-configurable locally * [ ] `opensre onboard` parity is added, or intentionally documented as out of scope * [ ] New required env vars / credentials are added to `.env.example` (never `.env`) * [ ] Sensitive credentials follow the [Credential resolution](#credential-resolution) contract below * [ ] `make verify-integrations` passes ### Credential resolution Keyring-eligible secrets and non-keyring config follow different write/read paths. Keep this contract when adding or changing an integration. | Surface | Write (wizard / setup) | Read (runtime) | | -------------------------------------------------- | --------------------------------------------------- | ---------------------------------------------- | | Integration store (`~/.opensre/integrations.json`) | Always on setup | First (preferred) | | OS keyring | Keyring-eligible secrets via `sync_env_secret` | Via `resolve_env_credential` when env is unset | | `.env` / process env | Non-keyring config (`*_URL`, user ids, channels, …) | Plain `os.getenv` for that tier | **Hard rules for new code** * Never use bare `os.getenv` for a keyring-eligible secret env name (`*_TOKEN`, `*_KEY`, `*_PASSWORD`, `*_SECRET`, connection strings, and similar). Use `resolve_env_credential` from `config.llm_credentials` (env first, then keyring). * Webhook / `*_URL` values are **never** keyring-backed (wizard routes them to store/`.env`, not `sync_env_secret`). Read with store → plain `os.getenv` only. Webhook URLs often **embed** a secret token — treat them like passwords for logging/masking, even though they are not keyring-eligible. * Leave `load_env_integration_services` plain-env-only (startup-safe; no keyring at boot). * Store still wins in `resolve_effective` / merge — env/keyring is the fallback tier only. * Tools receive credentials through `extract_params` (resolved integration state), never their own env reads. At execution, keys listed in the tool's `injected_params` override model-supplied values, so the verified source wins even when the model passes a token. A tool's resolver may read env only as the final fallback when nothing was injected — `integrations/github/tools/github_cli/credentials.py` is the reference: explicit/injected token first, then `GITHUB_MCP_AUTH_TOKEN`, then `GITHUB_TOKEN`/`GH_TOKEN`. * Set `OPENSRE_DISABLE_KEYRING=1` to skip keyring reads/writes (env and store still work). Canonical helpers: `resolve_env_credential` (env → keyring), `sync_env_secret` / `save_keyring_secret` (keyring-eligible writes), `sync_env_values` (non-keyring `.env` keys only). ## 3. Investigation wiring If the tool/integration is relevant to investigations: * [ ] Review alert-source seeding in `core/domain/alerts/alert_source.py` * [ ] Review source-priority/prompt mapping in `tools/investigation/stages/gather_evidence/prompt.py` * [ ] Review evidence/source registration in `core/domain/types/` or related state models * [ ] Add scenario coverage proving the tool surfaces useful RCA evidence If the integration is first-class for an `alert_source`, review the source-to-tool maps explicitly. ## 4. Discovery and edge cases For tools that list, search, or inspect resources: * [ ] Folder/nested resource layouts are considered where the upstream supports them * [ ] Large result sets are capped or paginated intentionally * [ ] Partial fetches are surfaced clearly (`truncated`, `fetch_error`, etc.) * [ ] Time/order-sensitive results preserve causal ordering where it matters ## 5. Docs and tests ### Docs * [ ] Ship or update a `docs/` page/section in the same PR (new tool, CLI command, pipeline behavior, or integration; and whenever a tool's API/schema or an integration's setup changes) * [ ] Any new `docs/` page is registered in `docs/docs.json` (without the `.mdx` suffix) so Mintlify navigation shows it * [ ] Investigation LLM tool-calling changes follow [investigation-tool-calling.md](investigation-tool-calling.md) ### Tests * [ ] Unit tests for config/normalization * [ ] Tool contract tests, or equivalent schema/metadata coverage * [ ] A registry/discovery test proves the tool is visible on the expected surface(s) * [ ] Runtime behavior tests for success and failure paths * [ ] At least one realistic fixture for live-payload parsing when external payloads are involved * [ ] If investigation-relevant, a test proves the planner/agent can discover or invoke the tool through the normal runtime path (plus synthetic/scenario coverage when the loop depends on it) * [ ] `tests/integrations/` updated when integration wiring changes Green tests are not enough if they only cover idealized mocks. ### Final gate (new integrations) Everything above is complete, **and**: * [ ] Screenshot or demo GIF showing the integration working end-to-end * [ ] E2E or synthetic test added * [ ] CI checks pass (see [CI.md](../CI.md)) ## 6. Reviewer focus Before opening or approving the PR, confirm the items most often missed are handled **explicitly**: tool placement (§1), live-payload robustness (§1), alert-source maps (§3), onboarding/setup/docs parity (§2 and §5), pagination/truncation/partial-response behavior (§4), and tests that cover realistic payloads and investigation usefulness — not only happy-path mocks (§5). Follow [CI.md](../CI.md) for the mandatory pre-push commands. # Airflow Source: https://opensre.com/docs/airflow Investigate DAG failures and extract execution context from Apache Airflow. ## Overview When a DAG failure alert fires, OpenSRE queries your Airflow REST API for the failing DAG run, task instances, and logs — then folds that evidence into the RCA pipeline alongside metrics and logs from your observability integrations. It supports: * DAG run inspection * Task instance retrieval * Failure detection * Evidence collection for RCA generation This integration is designed for **incident-driven workflows**, where an alert referencing a DAG triggers an investigation. *** ## Configuration ### Required Environment Variables ```bash theme={null} AIRFLOW_BASE_URL=http://localhost:8080 # Authentication (choose one) # Basic Auth AIRFLOW_USERNAME=your_username AIRFLOW_PASSWORD=your_password # Token-based (if supported) AIRFLOW_AUTH_TOKEN=your_token # Optional AIRFLOW_TIMEOUT_SECONDS=15 AIRFLOW_VERIFY_SSL=true AIRFLOW_MAX_RESULTS=50 ``` ### Setup Example Start Airflow locally: ```bash theme={null} docker run -p 8080:8080 apache/airflow:2.8.1 standalone ``` Create a failing DAG: ```python theme={null} from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime def fail_task(): raise Exception("Intentional failure") with DAG( dag_id="test_fail_dag", start_date=datetime(2024, 1, 1), schedule=None, catchup=False, ) as dag: PythonOperator( task_id="fail_task", python_callable=fail_task, ) ``` Trigger the DAG: ```bash theme={null} airflow dags trigger test_fail_dag ``` *** ## Investigation Flow Run the investigation CLI: ```bash theme={null} python -m cli investigate ``` Provide the alert payload: ```json theme={null} { "source": "airflow", "message": "Airflow DAG test_fail_dag failed", "metadata": { "dag_id": "test_fail_dag" } } ``` *** ## Capabilities | Capability | Description | | ------------------ | --------------------------------------------------- | | List DAG runs | Fetch execution history | | Get task instances | Inspect task-level failures | | Detect failures | Identify recent failing runs | | RCA support | Provide structured evidence for root cause analysis | *** ## Planner Behavior When `source = airflow`, the planner: * Prioritizes Airflow-related actions * Seeds Airflow tools into the action space However: * Tool selection is LLM-driven * Exact ordering may vary between runs This design avoids hard-coded routing and keeps the system extensible. *** ## Error Handling * Per-run failures are isolated — one failing request does not break the loop * Network/API errors are handled defensively * Partial evidence is preserved whenever possible *** ## Testing ### E2E Tests ```bash theme={null} python -m pytest tests/e2e/airflow/test_orchestrator.py -v ``` Expected output: ``` test_airflow_investigation_e2e PASSED ``` *** ## Limitations * Planner routing is probabilistic (LLM-based) * Requires a reachable Airflow instance * No CI-backed Airflow instance by default (local validation required) *** ## Design Notes * Integration follows the same contract as other sources (Datadog, Grafana, etc.) * Uses env-based configuration for simplicity * Avoids introducing hard overrides in planning logic * Focuses on evidence-driven investigation, not static rules *** ## Future Work * Stronger tool routing guarantees * CI-backed disposable Airflow instance for e2e tests * Deeper DAG dependency analysis * Richer RCA explanations # Alertmanager Source: https://opensre.com/docs/alertmanager Connect Alertmanager so OpenSRE can surface firing alerts and active silences during investigations OpenSRE queries Alertmanager to retrieve firing, silenced, and inhibited alerts — correlating the triggering alert with concurrent signals to narrow root-cause hypotheses faster. ## Prerequisites * Alertmanager v0.20+ reachable from the machine running OpenSRE * The Alertmanager URL (e.g. `http://alertmanager.monitoring.svc:9093`) * Credentials if your instance sits behind authentication (bearer token or basic auth) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup alertmanager ``` The wizard will ask for: 1. **Alertmanager URL** — the base URL of your Alertmanager instance 2. **Authentication method** — choose one of: * **None** — for unauthenticated instances on an internal network * **Bearer token** — for instances behind a reverse proxy that accepts a token * **Basic auth** — username and password ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} ALERTMANAGER_URL=http://alertmanager.monitoring.svc:9093 # Bearer token auth (optional) ALERTMANAGER_BEARER_TOKEN=your-token # Basic auth (optional — use instead of bearer token) ALERTMANAGER_USERNAME=admin ALERTMANAGER_PASSWORD=secret ``` | Variable | Default | Description | | --------------------------- | ------- | ---------------------------------------------------- | | `ALERTMANAGER_URL` | — | **Required.** Base URL of your Alertmanager instance | | `ALERTMANAGER_BEARER_TOKEN` | — | Bearer token for reverse-proxy auth | | `ALERTMANAGER_USERNAME` | — | Basic auth username | | `ALERTMANAGER_PASSWORD` | — | Basic auth password | Only one auth method is used at a time. If `ALERTMANAGER_BEARER_TOKEN` is set it takes precedence over basic auth. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "alertmanager-prod", "service": "alertmanager", "status": "active", "credentials": { "base_url": "http://alertmanager.monitoring.svc:9093", "bearer_token": "", "username": "", "password": "" } } ] } ``` ## Verify ```bash theme={null} opensre integrations verify alertmanager ``` Expected output: ``` Service: alertmanager Status: passed Detail: Connected to Alertmanager at http://alertmanager.monitoring.svc:9093; cluster status: ready. ``` ## How it works in investigations When an Alertmanager integration is configured, OpenSRE automatically includes it as an evidence source during every investigation. Two tools become available to the investigation agent: ### `alertmanager_alerts` Queries `/api/v2/alerts` for firing, silenced, and inhibited alerts. The agent uses this to: * Discover other alerts firing at the same time as the triggering alert * Check whether the triggering alert is already silenced or inhibited * Understand the blast radius of an infrastructure change by inspecting active alert labels * Correlate Prometheus alerts (OOM, latency spikes, error-rate increases) into a single timeline The `alertname` label from the incoming alert is automatically used as a filter so results are scoped to the incident under investigation by default. ### `alertmanager_silences` Queries `/api/v2/silences` for active silences. The agent uses this to: * Determine whether a known noisy alert has been silenced intentionally * Surface maintenance windows that overlap with the incident timeline * Avoid false root-cause conclusions caused by suppressed alerts ## Troubleshooting | Symptom | Fix | | ---------------------- | ----------------------------------------------------------------------------- | | **Status: missing** | Set `ALERTMANAGER_URL` or run `opensre integrations setup alertmanager` | | **Connection refused** | Verify the URL is reachable from this host; check firewall rules | | **401 Unauthorized** | Supply a bearer token or basic auth credentials | | **SSL error** | Ensure the CA cert is trusted, or use an `http://` URL for internal instances | | **Empty alert list** | Normal when no alerts are firing — check Alertmanager UI directly to confirm | ## Security best practices * Use a **read-only** reverse-proxy token when possible — OpenSRE only reads alerts and silences and never writes to Alertmanager during investigations. * Store credentials in `.env`, not in source code. * For internal Kubernetes deployments, prefer no auth over exposing credentials — restrict access at the network level instead. # HTTP API Source: https://opensre.com/docs/api Use the OpenSRE backend API: health, alert intake, investigations The OpenSRE backend serves one HTTP API (FastAPI, `gateway/web/webapp.py`). All routes live on a single port (default `8000`). To drive the agent in-process from your own code instead, see the [Python API](/docs/python-api). In production, always call the API over HTTPS — deploy behind a TLS-terminating load balancer (the backend is Terraform-managed, separately from this repo). ## Authentication The API has two auth schemes, matched to the caller: | Routes | Caller | Auth | | ---------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST /alerts`, `POST /investigate` | Machines (alert sources, schedulers) | Static bearer token: set `OPENSRE_ALERT_LISTENER_TOKEN` on the server and send `Authorization: Bearer `. Without the env var, these routes only accept loopback callers. | | `/api/*` (investigations) | Users via the web app | Clerk JWT: send `Authorization: Bearer `. Records are scoped to the token's organization. | | `GET /health`, `GET /healthz`, `GET /ok` | Probes | None | ## Health ```bash theme={null} curl https:///healthz # liveness: {"status": "ok"} curl https:///health # readiness: 503 until an LLM is configured ``` ## Alert intake Queue an alert for background investigation (returns `202`): ```bash theme={null} curl -X POST "https:///alerts" \ -H "Authorization: Bearer $OPENSRE_ALERT_LISTENER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"text": "CPU above 90% on checkout-db", "source": "grafana"}' ``` Bodies above 1 MiB are rejected with `413`. ## Synchronous investigation Run an investigation inline and get the report in the response (slow — the request stays open for the full pipeline; prefer the async API behind load balancers): ```bash theme={null} curl -X POST "https:///investigate" \ -H "Authorization: Bearer $OPENSRE_ALERT_LISTENER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"raw_alert": {"alert_name": "HighCPU", "severity": "critical"}}' ``` ## Async investigations (Clerk-authenticated) Enqueue and poll — the pattern web clients should use. Queued investigations are executed by a background worker in the web service; it is enabled by setting `OPENSRE_INVESTIGATION_WORKER=1` (the Terraform deployment sets this). Records live in Postgres when `DATABASE_URL` is set, else in process memory. Reports are written to local disk first and uploaded to S3 when `OPENSRE_ARTIFACTS_BUCKET` is configured. ```bash theme={null} # Enqueue (202) curl -X POST "https:///api/investigations" \ -H "Authorization: Bearer $CLERK_SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"raw_alert": {"alert_name": "HighCPU"}, "severity": "critical", "workspace_id": "T0123456"}' # → {"investigation_id": "3f6…", "status": "queued"} # workspace_id is optional — pass the Slack workspace (team) id when the # investigation originates from a Slack workspace. # Poll curl "https:///api/investigations/" \ -H "Authorization: Bearer $CLERK_SESSION_TOKEN" # → {"investigation_id": "3f6…", "status": "queued|running|completed|failed", # "report_s3_key": null, "report_url": null, "error": null} ``` Investigations belong to the Clerk organization in the token; reading another organization's investigation returns `404`. ## Status codes | Code | Meaning | | ----- | -------------------------------------------------------------------- | | `202` | Accepted and queued (`/alerts`, `POST /api/investigations`) | | `400` | Malformed JSON or missing required fields | | `401` | Missing or invalid bearer token (`/api/*`, or token-mode `/alerts`) | | `403` | Non-loopback caller and no `OPENSRE_ALERT_LISTENER_TOKEN` configured | | `404` | Unknown investigation id, or one outside your organization | | `413` | Alert body above 1 MiB | | `503` | LLM not configured (`/health`) or investigation pipeline failure | # Architecture audit tool Source: https://opensre.com/docs/architecture-audit-tool Clone a GitHub repo, run architecture shell passes, and persist full audit observations from the interactive shell OpenSRE provides GitHub-backed **architecture audit** tools for reviewing repository structure — import boundaries, file placement, size limits, and shim patterns. Use them from the interactive shell (`opensre`) when you want a structured architecture review before or alongside an incident investigation. ## Prerequisites * GitHub connected via [GitHub integration](/docs/github) **or** a token in `GITHUB_TOKEN` / `GH_TOKEN` * For private repos, ensure the token has `repo` read scope ## Typical workflow 1. **Clone** the target repository into a fixed temp workspace 2. Run architecture **shell heuristic passes** (import, placement, size, shim) via the action agent 3. **Save** the full untruncated observation list to disk 4. **Cleanup** the clone when finished In the REPL, describe what you want in plain language: ```text theme={null} > clone Tracer-Cloud/opensre and run an architecture audit on it ``` Or invoke tools explicitly through the action agent. ## Tools | Tool | What it does | Side effects | | -------------------------------- | ------------------------------------------------------------------------------------------------ | --------------------- | | `architecture_clone_repo` | Shallow-clone `owner/repo` into `opensre/workspace` under the system temp directory | Creates files on disk | | `architecture_cleanup_repo` | Delete the architecture workspace (refuses paths outside the allowed directory) | Deletes clone | | `architecture_save_observations` | Write the full observation list to `~/.opensre/{session_id}/{repo}-architecture-audit-{uuid}.md` | Writes a file | ### `architecture_clone_repo` ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre", "ref": "main" } ``` Returns `workspace_root` — the path to the clone. Always call `architecture_cleanup_repo` when finished. ### `architecture_save_observations` Call **after** cleanup and **before** the final summarized report. Pass the **complete** findings from all four shell passes (import, placement, size, shim) — not the short user-facing summary. ```json theme={null} { "repo_name": "opensre", "observations": "## Import pass\n- ...\n## Placement pass\n- ..." } ``` ## Verify GitHub access ```bash theme={null} opensre integrations verify github ``` Expected output includes MCP tool discovery; architecture tools additionally accept `GITHUB_TOKEN` when GitHub MCP is not configured. ## Gotchas * **Always cleanup.** `architecture_clone_repo` leaves a shallow clone on disk until `architecture_cleanup_repo` runs. The tool description enforces this workflow. * **Workspace is fixed.** Clones go under the architecture workspace in the system temp directory — you cannot point the clone at an arbitrary path. * **Save the full list.** `architecture_save_observations` expects untruncated pass output. Saving only the short report template loses audit evidence. * **Session id required.** Saving observations needs an active interactive-shell session (`~/.opensre/sessions/`). Start `opensre` before running the audit. ## Related docs * [GitHub](/docs/github) — token scopes and MCP setup * [GitHub workflow tools](/docs/github-workflow-tools) — engineering digests and PR status (separate from architecture audit) * [Interactive Shell Commands](/docs/interactive-shell-commands) — slash commands and session management # Argo CD Source: https://opensre.com/docs/argocd Connect Argo CD so OpenSRE can inspect GitOps application health, sync state, revisions, and drift during investigations OpenSRE queries the Argo CD REST API as a read-only evidence source during GitOps incident investigations. It can list visible applications, inspect one application's sync and health status, and fetch sanitized server-side diff output to show deployment drift. ## Prerequisites * Argo CD API server reachable from the machine running OpenSRE * A dedicated Argo CD account or API token with read access to the applications you want OpenSRE to inspect * The Argo CD base URL, for example `https://argocd.example.com` * Optional alert annotations that identify the affected Argo CD application, project, namespace, or revision ## Setup Argo CD is configured through environment variables or the persistent integration store. ### Option 1: Environment variables Add one authentication method to your `.env`: ```bash theme={null} ARGOCD_BASE_URL=https://argocd.example.com # Option A: API token auth. The token may be set with or without a "Bearer " prefix. ARGOCD_AUTH_TOKEN=*** # ARGOCD_TOKEN=*** # alias also supported # Option B: username/password auth. Use instead of ARGOCD_AUTH_TOKEN. # ARGOCD_USERNAME=opensre-readonly # ARGOCD_PASSWORD=*** # Optional scoping and TLS settings ARGOCD_PROJECT=default ARGOCD_APP_NAMESPACE=argocd ARGOCD_VERIFY_SSL=true ``` | Variable | Default | Description | | ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `ARGOCD_BASE_URL` | — | **Required.** Argo CD API base URL. Remote URLs must use `https://`; plain `http://` is accepted only for loopback or localhost development URLs. | | `ARGOCD_AUTH_TOKEN` | — | Argo CD bearer/API token. Use this or username/password, not both. | | `ARGOCD_TOKEN` | — | Alias for `ARGOCD_AUTH_TOKEN`. | | `ARGOCD_USERNAME` | — | Username for Argo CD session login. Use together with `ARGOCD_PASSWORD`. | | `ARGOCD_PASSWORD` | — | Password for Argo CD session login. Use together with `ARGOCD_USERNAME`. | | `ARGOCD_PROJECT` | — | Optional Argo CD project filter for listing and application-specific requests. | | `ARGOCD_APP_NAMESPACE` | — | Optional application namespace passed as `appNamespace` for application-specific requests. | | `ARGOCD_VERIFY_SSL` | `true` | Whether to verify TLS certificates. Set to `false` only for trusted local or lab environments. | OpenSRE rejects ambiguous auth configuration. Do not set a bearer token and username/password at the same time. ### Option 2: Persistent store You can also add Argo CD to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "argocd-prod", "service": "argocd", "status": "active", "credentials": { "base_url": "https://argocd.example.com", "bearer_token": "***", "project": "default", "app_namespace": "argocd", "verify_ssl": true } } ] } ``` The store also accepts `auth_token` or `token` as aliases for `bearer_token`. For username/password auth, omit `bearer_token` and set `username` and `password` instead. ### Option 3: Multiple Argo CD instances For multiple Argo CD instances, set `ARGOCD_INSTANCES` to a JSON array. The first valid instance is used as the default integration for investigations. ```bash theme={null} export ARGOCD_INSTANCES='[ { "name": "prod", "tags": {"env": "prod"}, "credentials": { "base_url": "https://argocd.prod.example.com", "bearer_token": "***", "project": "default" } }, { "name": "staging", "tags": {"env": "staging"}, "base_url": "https://argocd.staging.example.com", "username": "opensre-readonly", "password": "***" } ]' ``` When `ARGOCD_INSTANCES` is set, the single-instance `ARGOCD_BASE_URL` and auth variables are ignored for this service. `opensre integrations verify argocd` validates the resolved default instance. ## Verify Run: ```bash theme={null} opensre integrations verify argocd ``` Expected output: ```text theme={null} Service: argocd Status: passed Detail: Connected to Argo CD and listed 3 applications. ``` Verification performs a read-only application list call. It proves OpenSRE can reach Argo CD and list visible applications with the configured credentials; it does not write to Argo CD or sync applications. ## Usage in investigations When Argo CD is configured and an incoming alert contains GitOps context, OpenSRE can add Argo CD evidence to the investigation plan. OpenSRE recognizes these explicit alert fields: | Field | Location | Purpose | | ------------------------------------ | ---------------------------------------- | -------------------------------------------------- | | `argocd_application` or `argocd_app` | Top-level alert payload or `annotations` | Name of the affected Argo CD application. | | `application_name` | `annotations` | Generic application name fallback. | | `argocd_revision` | Top-level alert payload or `annotations` | Revision mentioned by the alert. | | `revision` | `annotations` | Generic revision fallback. | | `argocd_project` | Top-level alert payload or `annotations` | Argo CD project for scoped application requests. | | `argocd_app_namespace` | Top-level alert payload or `annotations` | Argo CD application namespace for scoped requests. | OpenSRE also looks for GitOps hints in alert text such as `argocd`, `argo cd`, `argo-cd`, `gitops`, `outofsync`, or `outofsynced`. Example alert: ```json theme={null} { "alert_name": "checkout-api OutOfSync", "annotations": { "summary": "Argo CD reports checkout-api is OutOfSync", "argocd_application": "checkout-api", "argocd_project": "default", "argocd_revision": "abc123" } } ``` Then run OpenSRE with the alert payload: ```bash theme={null} opensre investigate -i alert.json ``` ## Evidence collected ### `argocd_application_status` Fetches application status from Argo CD. * With `application_name`, it returns a compact summary for that application: sync status, health status, current revision, operation phase/message, destination, images, and recent deployment history. * Without `application_name`, it lists visible applications, optionally scoped by `ARGOCD_PROJECT`. The investigation agent uses this evidence to determine whether a deployment is `OutOfSync`, `Degraded`, on an unexpected revision, or correlated with a recent rollout. ### `argocd_application_diff` Fetches Argo CD server-side diff output for one application. * Requires `application_name`. * Returns `drift_detected`, `diff_count`, and sanitized diff records. * Helps identify Kubernetes objects whose live state differs from the desired GitOps state. ## Security best practices * Use a dedicated read-only Argo CD account or token for OpenSRE. * Store credentials in `.env` or `~/.opensre/integrations.json`, not in source code. * Use `https://` for remote Argo CD URLs. Plain `http://` is accepted only for loopback or localhost development URLs. * Do not disable `ARGOCD_VERIFY_SSL` for production instances. * OpenSRE redacts bearer tokens, passwords, token-like strings, and Kubernetes `Secret` diffs before surfacing Argo CD errors or diff evidence. * The integration is read-only: it lists applications, reads application summaries, and reads server-side diff data. It does not sync, modify, or delete Argo CD resources. ## Troubleshooting | Symptom | Fix | | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | **Status: missing** | Set `ARGOCD_BASE_URL` and exactly one auth method, or add an active `argocd` store entry. | | **Remote `http://` URL rejected** | Use `https://` for remote Argo CD. Use plain HTTP only for `localhost`, `127.0.0.1`, or `::1` development endpoints. | | **401 Unauthorized** | Check the token, or verify that username/password login can create an Argo CD session. | | **403 Forbidden** | Ensure the account can list applications and read the target application. | | **SSL error** | Fix the certificate chain or, for a trusted lab only, set `ARGOCD_VERIFY_SSL=false`. | | **No diff evidence** | Confirm the alert provides `argocd_application` or `argocd_app`; the diff tool requires an application name. | | **Application list succeeds but a named app fails** | Check `ARGOCD_PROJECT` and `ARGOCD_APP_NAMESPACE`, and confirm the account has access to that application. | # AWS Source: https://opensre.com/docs/aws Connect AWS so OpenSRE can map your infrastructure and investigate cloud-related alerts OpenSRE uses AWS to map your environment: Lambda functions, EKS clusters, S3 buckets, and more. It reads infrastructure state to build investigation context when cloud-related alerts fire. ## Prerequisites * AWS account with IAM permissions * Either a role ARN (recommended) or static access keys ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **AWS** when prompted and provide your credentials. ### Option 2: Environment variables (IAM role) ```bash theme={null} AWS_ROLE_ARN=arn:aws:iam::123456789012:role/OpenSREReadOnly AWS_EXTERNAL_ID=your-external-id # optional AWS_REGION=us-east-1 ``` ### Option 3: Environment variables (static keys) ```bash theme={null} AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY AWS_SESSION_TOKEN=... # optional, for temporary credentials AWS_REGION=us-east-1 ``` | Variable | Default | Description | | ----------------------- | ----------- | --------------------------------------- | | `AWS_ROLE_ARN` | — | IAM role to assume (recommended) | | `AWS_EXTERNAL_ID` | — | External ID for role assumption | | `AWS_REGION` | `us-east-1` | AWS region | | `AWS_ACCESS_KEY_ID` | — | Static access key (if not using role) | | `AWS_SECRET_ACCESS_KEY` | — | Static secret key | | `AWS_SESSION_TOKEN` | — | Session token for temporary credentials | Either `AWS_ROLE_ARN` or `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` is required. ## IAM permissions OpenSRE requires read-only access. Attach the following managed policies to the IAM role or user: * `ReadOnlyAccess` (AWS managed) — or a custom policy scoped to the services you want OpenSRE to inspect For least-privilege, the minimum services used are: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:GetCallerIdentity", "ec2:Describe*", "ecs:Describe*", "ecs:List*", "eks:Describe*", "eks:List*", "lambda:List*", "lambda:Get*", "s3:ListBucket", "s3:GetObject", "logs:FilterLogEvents", "logs:GetLogEvents", "cloudwatch:GetMetricData", "cloudwatch:ListMetrics" ], "Resource": "*" } ] } ``` ## Verify ```bash theme={null} opensre integrations verify aws ``` Expected output: ``` Service: aws Status: passed Detail: Authenticated via assume-role in us-east-1 as arn:aws:iam::123456789012:role/OpenSREReadOnly (account 123456789012) ``` ## Troubleshooting | Symptom | Fix | | --------------------------------- | -------------------------------------------------------------------- | | **AccessDenied on STS** | Ensure the caller has `sts:AssumeRole` permission on the target role | | **InvalidClientTokenId** | Check that `AWS_ACCESS_KEY_ID` is correct and the key is active | | **Could not connect to endpoint** | Check `AWS_REGION` and network connectivity | | **ExpiredTokenException** | Refresh your session token or rotate the access key | ## Security best practices * Use **IAM roles** instead of static keys wherever possible. * Scope IAM permissions to only the AWS services OpenSRE needs to inspect. * Rotate static access keys regularly. * Enable CloudTrail so all OpenSRE API calls are auditable. # AWS Lambda Source: https://opensre.com/docs/aws_lambda Connect AWS Lambda so OpenSRE can inspect function configuration, invocation logs, and runtime errors during investigations. OpenSRE integrates with AWS Lambda to inspect function configuration, retrieve recent invocation logs from CloudWatch, and investigate runtime failures during incident response. ## Prerequisites * An AWS account with AWS Lambda functions * AWS credentials configured for the runtime * Permission to access Lambda and CloudWatch Logs ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **AWS Lambda** when prompted. ### Option 2: Environment variables Add the following to your `.env`: ```bash theme={null} AWS_ACCESS_KEY_ID=your-access-key AWS_SECRET_ACCESS_KEY=your-secret-key AWS_REGION=us-east-1 AWS_SESSION_TOKEN=your-session-token # optional ``` | Variable | Required | Description | | ----------------------- | -------- | ---------------------------------------------- | | `AWS_ACCESS_KEY_ID` | Yes | AWS access key ID | | `AWS_SECRET_ACCESS_KEY` | Yes | AWS secret access key | | `AWS_REGION` | No | AWS region (defaults to `us-east-1`) | | `AWS_SESSION_TOKEN` | No | Session token when using temporary credentials | ## Verify ```bash theme={null} opensre integrations verify aws_lambda ``` Expected output: ```text theme={null} Service: aws_lambda Status: passed Detail: Successfully connected to AWS Lambda ``` ## Investigation tools When investigating AWS Lambda alerts, OpenSRE provides: * **Lambda Configuration** — inspect runtime, handler, timeout, memory, IAM role, environment variables, and deployed version. * **Lambda Inspect** — retrieve deployment package metadata and inspect function contents when available. * **Lambda Invocation Logs** — retrieve recent invocation logs from CloudWatch Logs. * **Lambda Errors** — investigate recent runtime failures and execution errors. All operations are read-only. ## Gotcha Lambda invocation history is retrieved from **CloudWatch Logs**. Ensure the configured AWS credentials have permission to access both **AWS Lambda** and **CloudWatch Logs**, otherwise log retrieval may fail even if Lambda access succeeds. # Azure Monitor Source: https://opensre.com/docs/azure-monitor Connect Azure Monitor Log Analytics so OpenSRE can pull KQL log evidence during investigations OpenSRE queries Azure Monitor Log Analytics through the public Query REST API to surface relevant logs during alert investigations. Each query is bounded by a `take` clause so result sets stay capped at a safe row limit. ## Prerequisites * Azure subscription with at least one **Log Analytics Workspace** collecting logs * A **Microsoft Entra ID (Azure AD) app registration** authorized to query the workspace * The **Log Analytics Reader** role granted on the workspace (or its resource group / subscription) to the app's service principal * Network access from the OpenSRE environment to `https://api.loganalytics.io` (or the sovereign cloud equivalent) over HTTPS ## Setup ### Option 1: Environment variables Add to your `.env`: ```bash theme={null} AZURE_LOG_ANALYTICS_WORKSPACE_ID=00000000-0000-0000-0000-000000000000 AZURE_LOG_ANALYTICS_TOKEN= AZURE_LOG_ANALYTICS_ENDPOINT=https://api.loganalytics.io AZURE_TENANT_ID= # optional, informational AZURE_SUBSCRIPTION_ID= # optional, informational AZURE_MAX_RESULTS=100 # optional, capped at 200 ``` | Variable | Default | Description | | ---------------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------- | | `AZURE_LOG_ANALYTICS_WORKSPACE_ID` | — | **Required.** Log Analytics Workspace ID (GUID) from the Azure portal | | `AZURE_LOG_ANALYTICS_TOKEN` | — | **Required.** Microsoft Entra ID OAuth2 bearer token with `Data.Read` on the workspace | | `AZURE_LOG_ANALYTICS_ENDPOINT` | `https://api.loganalytics.io` | Override for sovereign clouds (e.g. `https://api.loganalytics.azure.us` for Azure Government) | | `AZURE_TENANT_ID` | — | Microsoft Entra ID tenant ID (informational; useful for multi-tenant audits) | | `AZURE_SUBSCRIPTION_ID` | — | Azure subscription ID (informational) | | `AZURE_MAX_RESULTS` | `100` | Per-query row cap; OpenSRE clamps to a hard maximum of `200` | ### Option 2: Persistent store Credentials are persisted to `~/.opensre/integrations.json` with `0o600` permissions: ```json theme={null} { "version": 1, "integrations": [ { "id": "azure-prod", "service": "azure", "status": "active", "credentials": { "workspace_id": "00000000-0000-0000-0000-000000000000", "access_token": "", "endpoint": "https://api.loganalytics.io", "tenant_id": "", "subscription_id": "", "max_results": 100 } } ] } ``` ## Getting credentials ### 1. Find the Workspace ID 1. In the Azure portal, open **Log Analytics workspaces** and select your workspace. 2. On the workspace **Overview** page, copy **Workspace ID** (a GUID). ### 2. Register an Azure AD application 1. Open **Microsoft Entra ID** → **App registrations** → **New registration**. 2. Give the app a name (e.g. `opensre-log-analytics`) and register it as a single-tenant app. 3. From the app's **Overview** page, copy the **Application (client) ID** and the **Directory (tenant) ID**. 4. Open **Certificates & secrets** → **New client secret**, copy the **secret value** (it is shown only once). ### 3. Grant `Log Analytics Reader` on the workspace 1. Open the Log Analytics workspace in the portal. 2. Go to **Access control (IAM)** → **Add** → **Add role assignment**. 3. Pick the **Log Analytics Reader** role and assign it to the service principal created above. ### 4. Obtain a bearer token (client credentials flow) ```bash theme={null} curl -s -X POST \ "https://login.microsoftonline.com/$AZURE_TENANT_ID/oauth2/v2.0/token" \ -d "grant_type=client_credentials" \ -d "client_id=$AZURE_CLIENT_ID" \ -d "client_secret=$AZURE_CLIENT_SECRET" \ -d "scope=https://api.loganalytics.io/.default" \ | jq -r '.access_token' ``` Set the resulting token as `AZURE_LOG_ANALYTICS_TOKEN`. Tokens expire (usually after 60 minutes) — see the **Token rotation** note in *Security best practices*. ## Investigation tool OpenSRE exposes one tool against an Azure Monitor workspace: ### `query_azure_monitor_logs` `POST`s a KQL query to `/v1/workspaces//query` and returns the first table flattened into row dicts. Arguments the planner supplies: * **`query`** — KQL query text. If omitted, OpenSRE falls back to `AppTraces | order by TimeGenerated desc | take `. * **`time_range_minutes`** — sent as the `timespan` (`PTM`); defaults to `60`. * **`limit`** — per-query row cap; defaults to `50` and is clamped to `max_results` (hard limit `200`). OpenSRE always appends a `| take ` clause to the query if one is not present, so the workspace never returns more rows than the configured cap. ## Verify ```bash theme={null} opensre integrations verify azure ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL azure local env passed Configured for Azure Log Analytics workspace 00000000-0000-0000-0000-000000000000 via https://api.loganalytics.io. ``` The verify step is a credential-shape check — it does not call the workspace. To exercise the live path, point OpenSRE at a synthetic alert that names `azure` as the source and inspect the resulting evidence. ## Example KQL queries Recent application errors: ```kql theme={null} AppTraces | where SeverityLevel >= 3 | where TimeGenerated > ago(15m) | project TimeGenerated, OperationName, Message, AppRoleName | order by TimeGenerated desc | take 50 ``` Error count by severity over the last hour: ```kql theme={null} AppTraces | where TimeGenerated > ago(1h) | summarize count() by SeverityLevel | order by SeverityLevel desc ``` Failed dependency calls correlated with a request id: ```kql theme={null} AppDependencies | where Success == false | where TimeGenerated > ago(30m) | project TimeGenerated, Name, ResultCode, DurationMs, OperationId | order by TimeGenerated desc | take 50 ``` ## Troubleshooting | Symptom | Fix | | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **401 Unauthorized** | Token is missing, expired, or scoped to the wrong audience. Regenerate with `scope=https://api.loganalytics.io/.default` and confirm the service principal has **Log Analytics Reader** on the workspace. | | **403 Forbidden** | The token is valid but the principal lacks `Data.Read`. Re-check the role assignment on the workspace (or its parent resource group). | | **Empty result set** | Either the KQL `where` filter excludes everything or the workspace has no data in the requested timespan. Run the same query in **Logs** in the portal to confirm. | | **Wrong endpoint / DNS error** | Sovereign clouds use a different host (e.g. `https://api.loganalytics.azure.us` for US Government, `https://api.loganalytics.azure.cn` for China). Set `AZURE_LOG_ANALYTICS_ENDPOINT` accordingly. | | **`Missing workspace_id` / `Missing access_token`** | One or both required credentials are absent. Confirm both env vars (or both fields in the persistent store) are populated. | ## Security best practices * Use a **dedicated app registration** for OpenSRE — do not reuse a personal token or a broadly-scoped service principal. * Grant only **Log Analytics Reader** on the workspace; OpenSRE only needs read access to query data. * Keep the client secret out of source control — store it in `.env` or in a secret manager and only export it long enough to mint a token. * **Rotate the bearer token** before its 60-minute expiry. Long-lived deployments should re-mint the token from the client secret on a schedule rather than pasting a static token into `.env`. * The integration is **read-only**: OpenSRE only issues `POST /v1/workspaces//query` requests with a `take`-bounded KQL string. # Azure SQL Source: https://opensre.com/docs/azure-sql Connect Azure SQL so OpenSRE can diagnose database issues and query performance during investigations When a database alert fires, you need answers fast. OpenSRE connects to your Azure SQL instances to quickly diagnose what's going wrong — checking server health, finding those slow queries that are bogging things down, monitoring resource usage, and analyzing query execution plans to pinpoint bottlenecks. ## What you'll need Before getting started, make sure you have: * An Azure SQL Database instance up and running * Network connectivity from your OpenSRE environment to your Azure SQL server * Database credentials ready (username and password) * Your server hostname handy ## Getting connected ### The easy way: Interactive setup If you prefer guided steps, just run: ```bash theme={null} opensre integrations setup ``` Pick **Azure SQL** from the menu and follow the prompts. ### The flexible way: Environment variables Add these to your `.env` file: ```bash theme={null} AZURE_SQL_SERVER=myserver.database.windows.net AZURE_SQL_DATABASE=mydb AZURE_SQL_USERNAME=sqladmin AZURE_SQL_PASSWORD=your_password AZURE_SQL_ENCRYPT=true ``` | Variable | Default | Description | | -------------------- | ------------------------------- | ----------------------------------------- | | `AZURE_SQL_SERVER` | — | **Required.** Azure SQL server hostname | | `AZURE_SQL_DATABASE` | — | **Required.** Database name | | `AZURE_SQL_USERNAME` | — | **Required.** SQL authentication username | | `AZURE_SQL_PASSWORD` | — | **Required.** SQL authentication password | | `AZURE_SQL_ENCRYPT` | `true` | Encrypt your connection for security | | `AZURE_SQL_PORT` | `1433` | SQL Server port | | `AZURE_SQL_DRIVER` | `ODBC Driver 18 for SQL Server` | ODBC driver name | ### The permanent way: Integration store You can also save your connection details to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "azure-sql-prod", "service": "azure_sql", "status": "active", "credentials": { "server": "myserver.database.windows.net", "database": "mydb", "username": "sqladmin", "password": "your_password", "encrypt": true } } ] } ``` ## Finding your connection details Your Azure SQL server hostname looks like `myserver.database.windows.net`. You can find it: 1. Open the Azure Portal 2. Navigate to your SQL Database resource 3. Look for **Server name** in the overview panel 4. Copy it and add `.database.windows.net` if needed ## Network access: Firewall setup Azure SQL uses firewall rules to control who can connect. Make sure OpenSRE can reach your server: 1. In Azure Portal, go to your SQL server → **Security** → **Firewalls and virtual networks** 2. Add your OpenSRE environment's IP address 3. Or check **Allow Azure services and resources to access this server** if running in Azure If you're not sure of your IP, start with a permissive rule temporarily, get OpenSRE working, then lock it down. ## Investigation tools When OpenSRE investigates an Azure SQL-related alert, these diagnostic tools are available: ### Server status Retrieves service tier, resource utilization, connection counts, and database size. Useful for spotting DTU or vCore throttling. ### Current queries Lists active sessions and running queries to identify lock contention or long-running operations. ### Slow queries Surfaces top resource-consuming queries from Query Store or DMVs to pinpoint performance regressions. ### Wait stats Reports cumulative wait types to identify I/O, lock, or CPU bottlenecks. ### Resource stats Returns CPU, memory, and I/O utilization metrics for the target database. ## Test the connection Ready to verify everything works? ```bash theme={null} opensre integrations verify azure_sql ``` Expected output: ``` Service: azure_sql Status: passed Detail: Connected to Azure SQL Database ... ``` ## Troubleshooting | Symptom | Fix | | ----------------------------- | -------------------------------------------------------------------------------------------------------------- | | **Connection timeout** | Add your OpenSRE IP to the Azure SQL firewall rules. | | **Login failed** | Confirm `AZURE_SQL_USERNAME` and `AZURE_SQL_PASSWORD`. Check that SQL authentication is enabled on the server. | | **SSL/certificate error** | Keep `AZURE_SQL_ENCRYPT=true`. Install the correct ODBC driver (`ODBC Driver 18 for SQL Server`). | | **Permission denied on DMVs** | Grant `VIEW SERVER STATE` and `VIEW DATABASE STATE` to the OpenSRE user. | ## Security best practices * Use a **dedicated read-only SQL user** for OpenSRE — avoid admin credentials. * Keep **encryption enabled** (`AZURE_SQL_ENCRYPT=true`) in production. * Restrict firewall rules to the OpenSRE environment's egress IP. * Store credentials in `.env` or the integration store, never in source code. # Background investigations Source: https://opensre.com/docs/background-investigations Run investigations asynchronously in the interactive shell and receive RCA completion notifications. OpenSRE's interactive shell supports a **session-local background mode** for investigations. When background mode is enabled: * new investigations run asynchronously * the shell stays free for more questions and follow-ups * completed RCAs are tracked in-session * completion notifications can be sent to email (via the [`smtp`](/docs/smtp) integration), Telegram (via the [`telegram`](/docs/messaging/telegram) integration), Rocket.Chat (via the [`rocketchat`](/docs/messaging/rocketchat) integration), Buzz (via the [`buzz`](/docs/messaging/buzz) integration), or any combination This first version is **session-local only**. If the REPL process exits, in-flight background jobs stop with it. *** ## Commands | Command | What it does | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `/background on` | Enable async investigation launches | | `/background off` | Return to normal foreground execution | | `/background status` | Show background mode, tracked job count, and active notify channels | | `/background list` | List tracked jobs | | `/background show ` | Show the RCA summary and the per-channel notification result for one job | | `/background use ` | Promote a completed job into the active follow-up context | | `/background notify list` | Show the channels a completed RCA is delivered to (default: **none**) | | `/background notify set ` | Set completion channels — `email`, `telegram`, `rocketchat`, `buzz`, or a comma-separated combination | *** ## Typical flow 1. Start the interactive shell. 2. Enable background mode: ```text theme={null} /background on ``` 3. Choose where the completed RCA is delivered. **Channels are empty by default — skip this and no notification is sent:** ```text theme={null} /background notify set telegram # or: email, rocketchat, buzz, or email,telegram /background notify list # confirm ``` 4. Start an investigation: ```text theme={null} /investigate datadog ``` or paste a fresh alert in free text. 5. Keep using the shell while the RCA runs. 6. When it completes: * inspect it with `/background show ` * adopt it into active follow-up context with `/background use ` *** ## RCA summary contents Completed background jobs store: * root cause * top analysis items * recommended next steps * internal stats: * tool call count * investigation loop count * validity score *** ## Completion notifications Notifications are **off by default** — pick your channels with `/background notify set`. | Channel | Delivered through | Set up with | | ------------ | ------------------------------------------------- | --------------------------------------- | | `email` | [`smtp`](/docs/smtp) integration | `opensre integrations setup smtp` | | `telegram` | [`telegram`](/docs/messaging/telegram) integration | `opensre integrations setup telegram` | | `rocketchat` | [`rocketchat`](/docs/messaging/rocketchat) integration | `opensre integrations setup rocketchat` | | `buzz` | [`buzz`](/docs/messaging/buzz) integration | `opensre integrations setup buzz` | All channels send the same summary — **Root cause**, **Top analysis**, **What to do next**, plus a short internal stats section. Telegram, Rocket.Chat, and Buzz messages are plain text and are capped at the 4,096-character limit. Check what was actually delivered with `/background show `: the `notify` row shows one result per channel — `sent`, `failed: `, or a setup hint such as `missing telegram integration: TELEGRAM_BOT_TOKEN is not set.` A notification problem never fails the investigation. If a channel is unconfigured or delivery errors, the RCA still completes, stays in `/background list`, and can still be promoted with `/background use`. *** ## Current v1 limits * interactive shell only — there is no `opensre background …` CLI command * jobs are not persisted across REPL restarts * completion does not automatically replace your active follow-up context * you must run `/background use ` to promote a finished RCA # Better Stack Telemetry Source: https://opensre.com/docs/betterstack Connect Better Stack so OpenSRE can pull log evidence from your Telemetry sources during investigations OpenSRE uses Better Stack's ClickHouse SQL Query API to read log evidence during investigations. It queries the configured source via `remote(_logs)` for recent rows and `s3Cluster(primary, _s3)` for historical rows, bounded by the alert window. ## Prerequisites * A Better Stack account with at least one **Telemetry source** collecting logs * A **ClickHouse HTTP client** credential pair (username + password) generated from the dashboard * The **region-specific query endpoint** for your workspace (e.g. `https://eu-nbg-2-connect.betterstackdata.com`) * Network access from the OpenSRE environment to that endpoint over HTTPS ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup betterstack ``` You will be prompted for the query endpoint, username, password, and an optional comma-separated list of source IDs (planner hint). ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} BETTERSTACK_QUERY_ENDPOINT=https://eu-nbg-2-connect.betterstackdata.com BETTERSTACK_USERNAME= BETTERSTACK_PASSWORD= BETTERSTACK_SOURCES=t123456_myapp,t123456_gateway ``` | Variable | Default | Description | | ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `BETTERSTACK_QUERY_ENDPOINT` | — | **Required.** Region-specific SQL API host (e.g. `https://eu-nbg-2-connect.betterstackdata.com`) | | `BETTERSTACK_USERNAME` | — | **Required.** Username from **Connect ClickHouse HTTP client** | | `BETTERSTACK_PASSWORD` | — | **Required.** Password from the same dashboard flow | | `BETTERSTACK_SOURCES` | *(empty)* | Optional comma-separated list of **base source IDs** (e.g. `t123456_myapp`). The integration appends `_logs` / `_s3` internally. If omitted, the planner must derive the source from alert metadata (via a `betterstack_source` annotation) | ### Option 3: Persistent store Credentials are persisted to `~/.opensre/integrations.json` with `0o600` permissions: ```json theme={null} { "version": 1, "integrations": [ { "id": "betterstack-prod", "service": "betterstack", "status": "active", "credentials": { "query_endpoint": "https://eu-nbg-2-connect.betterstackdata.com", "username": "", "password": "", "sources": ["t123456_myapp"] } } ] } ``` ## Generating credentials From the Better Stack dashboard: 1. Open **Telemetry** and pick the source you want OpenSRE to query. 2. In the source sidebar, open **Integrations** → **Connect ClickHouse HTTP client**. 3. Copy the generated **username**, **password**, and **query endpoint**. The endpoint's subdomain encodes the region (e.g. `eu-nbg-2-connect`, `us-connect`). 4. The **base source ID** (e.g. `t123456_myapp`) is shown above the integration panel and is the value to supply for `BETTERSTACK_SOURCES`. Use the base name only — OpenSRE appends `_logs` and `_s3` internally. ## Investigation tool OpenSRE exposes one tool against a Better Stack source: ### `query_betterstack_logs` Returns `(dt, raw)` pairs by UNIONing: * Recent rows from `remote(_logs)` * Historical rows from `s3Cluster(primary, _s3) WHERE _row_type = 1` Arguments the planner supplies: * **`source`** — the base identifier (e.g. `t123456_myapp`). Falls back to the first configured `sources` entry when omitted. * **`since` / `until`** — ISO-8601 timestamps that bound the `dt` column. Optional, typically derived from the alert window. * **`limit`** — row cap; defaults to `500`. All queries run with `FORMAT JSONEachRow` and `output_format_pretty_row_numbers=0`. Source names are validated against `^[A-Za-z0-9_]+$` to prevent identifier injection. ## Triggering investigation from an alert When an alert carries a `betterstack_source` annotation, the planner wires it through to `query_betterstack_logs` automatically: ```json theme={null} { "title": "[betterstack] Agent silent (no info logs in 5min)", "state": "alerting", "alert_source": "betterstack", "commonAnnotations": { "summary": "Agent silent (no info logs in 5min): info < 1", "betterstack_source": "t123456_myapp" } } ``` ## Verify ```bash theme={null} opensre integrations verify betterstack ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL betterstack store passed Connected to Better Stack SQL API at https://eu-nbg-2-connect.betterstackdata.com ``` The verify step issues a cheap probe (`SELECT 1 FORMAT JSONEachRow`) against the configured endpoint using the stored credentials. ## Troubleshooting | Symptom | Fix | | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Authentication failed (401)** | Regenerate credentials via the dashboard's **Connect ClickHouse HTTP client** flow; confirm `BETTERSTACK_USERNAME` / `BETTERSTACK_PASSWORD` match exactly. | | **Endpoint not found / DNS error** | The region subdomain is wrong. Copy the endpoint directly from the dashboard (e.g. `eu-nbg-2-connect`, `us-connect`, `eu-fsn-3-connect`). | | **`invalid source name` error on query** | The `source` argument contains characters outside `[A-Za-z0-9_]`. Use the base ID shown in the dashboard — no dashes, no quotes, no whitespace. | | **Empty result set for a known-busy source** | Check that the alert window (`since` / `until`) actually overlaps with the source's `dt` range; historical rows older than recent retention live in `s3Cluster(primary, _s3)`. | | **`planner did not configure a source` error** | Either set `BETTERSTACK_SOURCES`, or ensure the alert payload includes a `betterstack_source` annotation. | ## Security best practices * Use a **dedicated ClickHouse HTTP client credential** for OpenSRE — not your personal dashboard login. * Keep the credential pair out of source control — use `.env` or the persistent store (`~/.opensre/integrations.json`). * The integration is **read-only**: OpenSRE only issues `SELECT` statements against `remote(...)` and `s3Cluster(...)` table functions. * Source identifiers are allowlisted against `^[A-Za-z0-9_]+$` before being interpolated into SQL, preventing identifier-injection attacks from the alert payload. * Rotate credentials periodically via the Better Stack dashboard. # Community Giveaway Source: https://opensre.com/docs/bi-weekly-giveaway Contribute to OpenSRE, get recognised, and win prizes every two weeks. Contribute. Get recognised. Every two weeks, the maintainer team picks **three winners** from active contributors in the current window. OpenSRE bi-weekly giveaway: what counts, rules, and tiered prizes * Merged PRs and resolved issues * Code reviews and documentation * Consistent activity over the window * Quality contributions over rushed ones * Must contribute within the active window * Winners chosen by the maintainer team * Both volume and quality are tracked * One win per person per month ## Prizes Tiered by place. Each winner receives an **Amazon gift card** and an **OpenSRE merch bundle**. | Place | Reward | Value | | ----- | ------------------------------- | ----- | | 1st | Amazon gift card + merch bundle | \$50 | | 2nd | Amazon gift card + merch bundle | \$35 | | 3rd | Amazon gift card + merch bundle | \$25 | ## How to participate 1. Contribute during the current two-week window — **merged PRs**, **resolved issues**, **code reviews**, and **documentation** all count toward eligibility. 2. Stay active consistently through the window; quality matters more than rushing many small changes. 3. Join [Discord](https://discord.gg/opensre) for winner announcements each cycle. New to the repo? Issues tagged **`good first issue`** or **`help wanted`** are a good entry point, but they are not required — any qualifying contribution in the window counts. | Resource | Link | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Good first issues | [Browse on GitHub](https://github.com/Tracer-Cloud/opensre/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) | | Help wanted | [Browse on GitHub](https://github.com/Tracer-Cloud/opensre/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) | | Good First Issues guide | [Contributor guide](https://github.com/Tracer-Cloud/opensre/blob/main/docs/good-first-issues/README.md) | | PR review flow | [Greptile · human · automerge](/docs/pr-review-flow) | | Winner announcements | [OpenSRE Discord](https://discord.gg/opensre) | ## Past winners | Date | 1st | 2nd | 3rd | | ------------ | --------- | -------- | ------------- | | May 15, 2026 | Kevs | AniketXD | cerencamkiran | | May 2, 2026 | muddlebee | Davidson | Yash Saini | Winners are selected at maintainer discretion based on contribution quality and activity in the active window. If you win, the team will reach out on Discord to arrange your prize. # Bitbucket Source: https://opensre.com/docs/bitbucket Connect Bitbucket so OpenSRE can correlate recent commits and code changes with incidents OpenSRE queries Bitbucket to retrieve recent commits, file contents, and code search results — helping trace which change in a Bitbucket-hosted repository triggered an incident. ## Prerequisites * Bitbucket Cloud account * App password with repository read access ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Bitbucket** when prompted and provide your workspace, username, and app password. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} BITBUCKET_WORKSPACE=your-workspace-slug BITBUCKET_USERNAME=your-username BITBUCKET_APP_PASSWORD=your-app-password ``` | Variable | Default | Description | | ------------------------ | ------- | -------------------------------------- | | `BITBUCKET_WORKSPACE` | — | **Required.** Bitbucket workspace slug | | `BITBUCKET_USERNAME` | — | **Required.** Bitbucket username | | `BITBUCKET_APP_PASSWORD` | — | **Required.** Bitbucket app password | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "bitbucket-prod", "service": "bitbucket", "status": "active", "credentials": { "workspace": "your-workspace-slug", "username": "your-username", "app_password": "your-app-password" } } ] } ``` ## Creating an app password 1. In Bitbucket, go to **Personal settings** → **App passwords** 2. Click **Create app password** 3. Give it a label (e.g., `opensre`) 4. Enable the following permissions: **Repositories: Read** 5. Copy the generated password The workspace slug is the identifier in your Bitbucket URL: `https://bitbucket.org//` ## Investigation tools When OpenSRE investigates a Bitbucket-related alert, three tools are available: * **Commits** — retrieves recent commits for a repository, optionally filtered by file path * **File contents** — fetches the contents of a file at a specific revision * **Code search** — searches code across the workspace or a specific repository All operations are read-only. ## Verify ```bash theme={null} opensre integrations verify bitbucket ``` Expected output: ``` Service: bitbucket Status: passed Detail: Authenticated as Your Name; workspace: your-workspace ``` ## Troubleshooting | Symptom | Fix | | --------------------------- | --------------------------------------------------------------- | | **401 Unauthorized** | Check username and app password combination | | **Workspace not found** | Verify the workspace slug — it's case-sensitive | | **403 Forbidden** | Ensure the app password has **Repositories: Read** permission | | **Code search unavailable** | Code search requires a Bitbucket Cloud Standard or Premium plan | ## Security best practices * Use an **app password** rather than your account password — app passwords can be revoked individually. * Scope permissions to **Repositories: Read** only. * Store credentials in `.env`, not in source code. # ClickHouse Source: https://opensre.com/docs/clickhouse Connect ClickHouse so OpenSRE can inspect query activity and system health during investigations OpenSRE queries ClickHouse system tables to surface slow queries, active connections, replica status, and table statistics — helping diagnose database performance issues during alert investigations. ## Prerequisites * ClickHouse 21.x or later * Network access from the OpenSRE environment to your ClickHouse instance * A user with read access to system tables ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **ClickHouse** when prompted and provide your host and credentials. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} CLICKHOUSE_HOST=your-clickhouse-host CLICKHOUSE_PORT=8123 CLICKHOUSE_DATABASE=default CLICKHOUSE_USER=default CLICKHOUSE_PASSWORD=your-password CLICKHOUSE_SECURE=false # set to true for HTTPS ``` | Variable | Default | Description | | --------------------- | --------- | -------------------------------------------- | | `CLICKHOUSE_HOST` | — | **Required.** ClickHouse hostname or IP | | `CLICKHOUSE_PORT` | `8123` | HTTP(S) port (8123 for HTTP, 8443 for HTTPS) | | `CLICKHOUSE_DATABASE` | `default` | Default database | | `CLICKHOUSE_USER` | `default` | Username | | `CLICKHOUSE_PASSWORD` | — | Password | | `CLICKHOUSE_SECURE` | `false` | Use HTTPS (`true`/`false`) | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "clickhouse-prod", "service": "clickhouse", "status": "active", "credentials": { "host": "your-clickhouse-host", "port": 8443, "database": "default", "username": "opensre_readonly", "password": "your-password", "secure": true } } ] } ``` ## Creating a read-only user ```sql theme={null} CREATE USER opensre_readonly IDENTIFIED BY 'secure-password'; GRANT SELECT ON system.* TO opensre_readonly; GRANT SELECT ON default.* TO opensre_readonly; ``` ## Investigation tools When OpenSRE investigates a ClickHouse-related alert, three diagnostic tools are available: * **Query activity** — reads `system.query_log` to surface recent slow and failed queries * **System health** — reads `system.metrics` for active connections, query count, replica status, and uptime * **Table stats** — reads `system.parts` for table-level row counts and storage sizes All operations are read-only. ## Verify ```bash theme={null} opensre integrations verify clickhouse ``` Expected output: ``` Service: clickhouse Status: passed Detail: Connected to ClickHouse 23.8.1; database: default ``` ## Troubleshooting | Symptom | Fix | | ---------------------------------- | ----------------------------------------------------------------------------- | | **Connection refused** | Check host, port, and firewall. Use port `8443` with `CLICKHOUSE_SECURE=true` | | **Authentication failed** | Verify username and password | | **SSL error** | Set `CLICKHOUSE_SECURE=true` and use port `8443` | | **Access denied on system tables** | Grant `SELECT ON system.*` to the OpenSRE user | ## Security best practices * Create a **dedicated read-only user** with access only to `system.*` and the databases OpenSRE needs. * Use **HTTPS** (`CLICKHOUSE_SECURE=true`) in production. * Store credentials in `.env`, not in source code. # Closed-Loop Learning Source: https://opensre.com/docs/closed-loop-learning Turn production investigation misses into regression evals # Closed-Loop Learning OpenSRE captures accuracy feedback after every investigation. When you mark a result as **partial** or **inaccurate**, it is classified into a triage taxonomy and recorded as a *miss*. The `opensre misses` command surface lets you review trends, track recurrence, and convert top misses into reproducible benchmark scenarios — closing the loop from production usage back into the eval suite. ## Quick reference | Command | What it does | | ---------------------------------- | ----------------------------------------------------------------------- | | `opensre misses list` | Show recent misses with alert, taxonomy, rating, and root cause. | | `opensre misses stats` | Taxonomy breakdown plus recurring `(alert, taxonomy)` pairs. | | `opensre misses export --out PATH` | Write per-case `alert.json` files the benchmark runner can consume. | | `opensre misses convert MISS_ID` | Convert a single miss into a scenario payload (stdout or `--out FILE`). | ## How a miss is captured After every investigation the CLI shows the accuracy prompt. If you pick **partial** or **inaccurate** you'll be asked for a short note and a taxonomy bucket: * **Retrieval gap** — the agent did not fetch the evidence it needed. * **Reasoning gap** — it had the evidence but drew the wrong conclusion. * **Tool failure** — a tool errored, timed out, or returned bad data. * **Routing/prompt failure** — the wrong tools or plan were selected. * **Unknown** — choose this only when none of the above clearly fit. The miss is written to `~/.opensre/misses.jsonl` and an `investigation_miss_classified` event is emitted to PostHog with the run provenance, taxonomy, and (when available) `user_id` / `org_id`. The original feedback record in `~/.opensre/feedback.jsonl` is untouched. ## Reviewing trends ```bash theme={null} # Everything captured in the last week opensre misses stats --since 7d # Drill into just the retrieval gaps opensre misses list --since 14d --taxonomy retrieval_gap # Machine-readable output for dashboards or pipelines opensre misses stats --since 30d --json ``` `stats` reports the **count per taxonomy** and the **recurring `(alert_name, taxonomy)` pairs** (seen more than once). Recurring pairs are the strongest signal that a regression scenario is overdue. ## Converting misses to regressions `opensre misses export` writes one scenario per recurring `(alert, taxonomy)` pair, ordered by how often it has recurred. The output uses the same benchmark scenario `alert.json` shape, so the benchmark runner consumes it without any adapter changes: ```bash theme={null} opensre misses export \ --since 7d --top 10 \ --out tests/benchmarks/production_misses/ ``` Each case directory contains an `alert.json` whose `commonAnnotations.scoring_points` dict (`expected_root_cause`, `expected_category`, `miss_notes`) carries the rubric for grading — the same location `opensre investigate --evaluate` already reads from, and the same one `strip_scoring_points_from_alert` removes before the agent sees the alert. The `_meta` block carries non-rubric provenance (`miss_id`, `original_run_id`, `taxonomy`). Commit the directory under `tests/benchmarks/` and the next benchmark run will include the new regressions. ## Weekly triage workflow | Step | Owner | SLA | | ----------------------------------------------------------------------------------------- | ---------------- | -------------- | | Run `opensre misses stats --since 7d` and review top recurring pairs | On-call engineer | Monday morning | | Run `opensre misses export --since 7d --top 10 --out tests/benchmarks/production_misses/` | On-call engineer | Monday | | Open a PR adding the new scenarios with a `benchmark` label | On-call engineer | Tuesday | | Run the benchmark workflow against the PR branch | Reviewer | Wednesday | | Track fix-rate week-over-week using PostHog `investigation_miss_classified` trends | Eng lead | Ongoing | PostHog dashboards built on `investigation_miss_classified` (grouped by `taxonomy` and `alert_name`) provide the week-over-week trend view referenced by the SLAs. ## Privacy Miss records live entirely on the engineer's machine in `~/.opensre/misses.jsonl`. To delete everything captured locally, remove the file. The `investigation_miss_classified` PostHog event carries identifiers and structured metadata only: * `miss_id`, `feedback_id`, `run_id` * `taxonomy`, `rating`, `has_detail` (boolean — whether a note was provided, never the note itself) * `alert_name`, `pipeline_name`, `root_cause_category` * Optional `user_id`, `org_id` when running on a hosted/JWT path The free-text note (`taxonomy_detail`) and the captured `root_cause` string are **never** sent to PostHog — they only exist in the local JSONL store, so removing `~/.opensre/misses.jsonl` removes them entirely. # CloudOpsBench benchmark Source: https://opensre.com/docs/cloudopsbench Run opensre+LLM against the 452-scenario Cloud-OpsBench corpus (Wang et al, arXiv:2603.00468v1) and compare to published LLM-alone baselines. ## Overview CloudOpsBench is a 452-scenario Kubernetes root-cause-analysis benchmark published by Wang et al ([arXiv:2603.00468v1](https://arxiv.org/abs/2603.00468), Feb 2026). The paper uses a *State Snapshot* paradigm — each fault case is a frozen JSON repository served via mocked `kubectl`-style tool calls — so every evaluation is bit-for-bit reproducible and needs no live cluster. OpenSRE wraps this corpus through a small reusable benchmark framework that adds cost tracking, integrity guards (pre-registration, per-stratum reporting, negative results, COI disclosure), per-LLM dispatch with version pinning, and self-contained markdown + HTML reports. The goal is to publish the opensre+LLM column against the paper's LLM-alone baselines on the same scenarios. ```text theme={null} Paper baseline opensre+LLM (this benchmark) ───────────── ────────────────────────── DeepSeek-V3.2 0.73 A@1 → target 0.78+ GPT-5 0.67 A@1 → target 0.78+ GPT-4o 0.49 A@1 → target 0.65+ Claude-4-Sonnet 0.50 A@1 → target 0.65+ ``` ## What you need to run it CloudOpsBench needs **no live infrastructure**. The frozen snapshots are the environment. ```bash theme={null} # 1. Python 3.12+ (project standard) # 2. Benchmark-dedicated LLM API keys — keep separate from production opensre keys export ANTHROPIC_API_KEY=... # Claude-4-Sonnet via Anthropic direct export OPENAI_API_KEY=... # GPT-5, GPT-4o export DEEPSEEK_API_KEY=... # DeepSeek-V3.2 # 3. Pull the corpus (one-time, a few hundred MB) make download-cloudopsbench-hf ``` You do **not** need: AWS credentials, an EKS cluster, kind/minikube, Bedrock, GPU, Grafana, Datadog, or Prometheus. ## Quick start ### List adapters ```bash theme={null} uv run python -m tests.benchmarks._framework.cli list ``` ### Validate a config ```bash theme={null} uv run python -m tests.benchmarks._framework.cli validate \ tests/benchmarks/configs/cloudopsbench_smoke.yml ``` The config lint catches anti-patterns (`runs_per_case < 3`, missing `pre_registration_path`, oversized grids, system-path `output_dir`). Validation returns non-zero on any failure. ### Dev-mode run `--dev` skips the integrity gates so you can smoke-test the wiring without writing a pre-registration file. The run ID gets a `dev-` prefix so dev results can't be silently promoted. ```bash theme={null} uv run python -m tests.benchmarks._framework.cli run \ tests/benchmarks/configs/cloudopsbench_smoke.yml --dev ``` ### Production run A production run requires: * A **pre-registration YAML** at `pre_registration_path` listing per-model expected deltas, committed to git before the run starts (integrity Mechanism 1) * `seed:` set in config (Mechanism 6) * Adapter declaration of `data_contamination_checked = True` (Mechanism 7) * At least one validity metric declared by the adapter (Mechanism 3) ```bash theme={null} uv run python -m tests.benchmarks._framework.cli run \ tests/benchmarks/configs/cloudopsbench_v1.yml ``` On completion, the run directory contains `report.json` (machine-readable), `report.md` (human-readable summary), `report.html` (self-contained, no external CSS/JS), and `cases/*.json` (per-cell artifacts). ### Re-render an existing report ```bash theme={null} uv run python -m tests.benchmarks._framework.cli report \ .bench-results/example// ``` ## Config reference ```yaml theme={null} benchmark: cloudopsbench modes: - opensre+llm # opensre wrapping the LLM # - llm_alone # paper provides LLM-alone numbers; rerun only if not trusting them llms: - claude-4-sonnet - deepseek-v3.2 - gpt-5 - gpt-4o model_versions: # pinned to exact provider snapshots claude-4-sonnet: claude-sonnet-4-5-20250929 deepseek-v3.2: deepseek-chat-v3.2 gpt-5: gpt-5-2025-08-07 gpt-4o: gpt-4o-2024-11-20 runs_per_case: 3 # replication for variance estimate (Box-Hunter-Hunter Ch 3.4) workers: 4 # serial across LLMs, parallel within cost_budget_usd: 1000 # hard cap; run aborts cleanly when exceeded seed: 42 # required for reproducible case selection (M6) filters: # optional case subsetting systems: [boutique] difficulty: [hard, medium] output_dir: .bench-results/cloudopsbench-v1/ report_formats: [json, markdown, html] pre_registration_path: tests/benchmarks/configs/preregistrations/v1.yml ``` ### Env-var overrides for CI These let CI override knobs without editing the YAML: | Variable | Purpose | | ------------------------------- | --------------------------- | | `OPENSRE_BENCH_WORKERS` | Override `workers:` | | `OPENSRE_BENCH_COST_BUDGET_USD` | Override `cost_budget_usd:` | ## Integrity guarantees The framework enforces 11 honest-results mechanisms at the code level. There is no bypass short of editing the framework itself. ### Pre-flight (before any case runs) `IntegrityGuard.pre_flight` raises `IntegrityViolation` if any of these hold: * **M1 — Pre-registration**: `pre_registration_path` unset, missing, or empty. Forces the engineer to commit expected deltas before seeing results. * **M3 — Validity metrics**: adapter declares no validity metric (no Streetlight Effect). * **M6 — Seeded selection**: `seed:` is `None` (no cherry-picking). * **M7 — Contamination check**: adapter has not declared `data_contamination_checked = True`. All violations surface in a single exception so the engineer fixes everything in one pass, not one-fix-rerun-discover-next. ### Report-validation (before the report is emitted) `IntegrityGuard.report_validation` refuses to publish a report if: * **M3** — Not every adapter-declared metric is in the report * **M4** — Per-stratum breakdown missing or contains only `all` (no aggregate-only reporting) * **M5** — Raw per-case artifacts directory missing * **M9** — `negative_results` is empty * **M10** — `coi_disclosure` is empty * **M1** — Pre-registration path not carried into the report ### Two more mechanisms are operational, not code-enforced * **M8 — External replication** of ≥1 cell by a third party before public claim * **M11 — Blinded LLM-as-judge calibration** (BDIL Phase B; tracked separately) ## Cost tracking The framework registers a usage hook on `core/llm/transports/sdk/llm_clients.py`'s `LLMClient`, `OpenAILLMClient`, and `BedrockLLMClient`. Every successful LLM call feeds `(model, tokens_in, tokens_out)` into a `CostTracker`. The tracker enforces the configured `cost_budget_usd` as a hard cap — the next call that would exceed budget raises `CostBudgetExceeded` and the runner halts cleanly with a partial-completion report. Per-cell `tokens_in / tokens_out / cost_usd` is currently 0 (aggregate cost is correct; per-cell delta capture is a follow-up). Total run cost in `report.json` is honest. ## Metrics Paper's 13 deterministic metrics plus 3 framework-added validity metrics: | Family | Metric | Source | | -------------------- | ---------------------------------------------------------------------------- | ---------------------------------- | | Outcome | `a1, a3, tcr, exact, in_order, any_order` | Paper § 4.2.1 | | Process — alignment | `rel, cov` | Paper § 4.2.2 | | Process — efficiency | `steps, mtti` | Paper § 4.2.2 | | Process — robustness | `iac, rar, ztdr` | Paper § 4.2.2 | | Validity | `citation_grounding_rate, entity_existence_rate, kubectl_actionability_rate` | Framework (regex + universe check) | All 16 metrics are deterministic (string / set comparison) — no LLM-as-judge at evaluation time. ## Existing production entry points `make test-cloudopsbench` and `opensre tests cloudopsbench` route through `tests/benchmarks/cloudopsbench/run_suite.py`, which is the legacy imperative-CLI surface. The framework runner is the new YAML-config surface and coexists with it during the transition. Both call into the same adapter, scoring code, and replay backend. ## Reference * Paper: Wang et al, *Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems*, [arXiv:2603.00468v1](https://arxiv.org/abs/2603.00468), 28 Feb 2026 — [GitHub](https://github.com/LLM4Ops/Cloud-OpsBench) * HF dataset: [`tracer-cloud/cloud-ops-bench-dataset`](https://huggingface.co/datasets/tracer-cloud/cloud-ops-bench-dataset) * Framework source: [`tests/benchmarks/_framework/`](https://github.com/Tracer-Cloud/opensre/tree/main/tests/benchmarks/_framework) * Adapter source: [`tests/benchmarks/cloudopsbench/`](https://github.com/Tracer-Cloud/opensre/tree/main/tests/benchmarks/cloudopsbench) # AWS CloudTrail Source: https://opensre.com/docs/cloudtrail_events Trace AWS configuration-change causality — who changed what, and when — during incidents OpenSRE uses AWS CloudTrail to answer the first question of every cloud post-mortem: **"who changed what, and when?"** When an AWS alert fires, the planner can look up recent management events — IAM changes, security-group mutations, EKS/Lambda config updates, and resource deletions — scoped to a resource, a principal, or a time window. CloudTrail lookups are read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot mutate any resources. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/aws) (role ARN recommended) — CloudTrail reuses the same account credentials and region, so no extra setup is needed * IAM permission for the single read-only CloudTrail action listed below ## How it works CloudTrail is account-wide, so the tool becomes available to the planner whenever the [AWS integration](/docs/aws) is configured — there is nothing resource-specific to set up. The region comes from the AWS integration (or `AWS_REGION`, defaulting to `us-east-1`). ## Tools | Tool | AWS API call | What it returns | | -------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `lookup_cloudtrail_events` | `cloudtrail:LookupEvents` | Recent management events — event name, time, source, acting username, affected resources, AWS region, source IP, and any error code. | ### Parameters | Parameter | Default | Description | | ------------------ | ----------- | ------------------------------------------------------------------------------------ | | `resource_name` | — | Filter to events touching a specific resource name/ARN (the most specific filter). | | `event_source` | — | Filter by AWS service event source, e.g. `iam.amazonaws.com`, `ec2.amazonaws.com`. | | `username` | — | Filter by the acting principal / IAM username. | | `region` | `us-east-1` | AWS region to query. | | `duration_minutes` | `60` | Look-back window. CloudTrail retains 90 days of history (the upper bound). | | `max_results` | `50` | Maximum events to return (CloudTrail caps this at 50 per call). | | `next_token` | — | Pagination token from a previous truncated response; pass it to fetch the next page. | CloudTrail's `LookupEvents` API accepts **only one filter attribute per call**. When more than one filter is supplied, the tool sends the most specific one, in priority order: `resource_name` → `username` → `event_source`. With no filter, it returns recent account-wide events for the window. CloudTrail returns at most 50 events per page. When more matching events exist, the response sets **`truncated: true`** and returns a **`next_token`** — pass it back via the `next_token` parameter to fetch the next page, so a busy account or wide window never silently drops events. A single event can touch more resources than the transport layer returns inline (for example `CreateTags` across a fleet). When the affected-resources list for an event is trimmed, that event carries **`resources_truncated: true`**, signalling that the real blast radius is wider than the resources shown — so a large fan-out change is never silently understated. ### Use cases * Finding who modified an IAM policy, role, or security group just before an incident * Tracing configuration changes to a specific resource (by resource name) * Auditing every action taken by a principal (by username) * Reviewing recent activity from a single AWS service (by event source) * Establishing change causality at the start of a post-mortem ## IAM permissions The tool only needs one read-only CloudTrail action: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudtrail:LookupEvents" ], "Resource": "*" } ] } ``` Attach this policy to the same IAM role or user already configured for the [AWS integration](/docs/aws). If you are already using the AWS managed `ReadOnlyAccess` policy, this action is already covered. **Execution identity:** the AWS integration's `role_arn` / credentials gate *availability* and supply the region, but the lookup itself runs through boto3's standard credential chain (environment variables, shared config, or the host's instance role) — the configured role is **not** assumed for the call. Ensure the identity the OpenSRE process runs as can perform `cloudtrail:LookupEvents`. This matches the other AWS tools (RDS/EKS). ## Troubleshooting | Symptom | Fix | | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **AccessDenied on `cloudtrail:LookupEvents`** | Add the IAM policy above to the role or user used by the AWS integration. | | **No events returned** | Widen `duration_minutes`, loosen the filter, or confirm you are querying the region where the activity occurred. Only management events are returned; data events are not. | | **ThrottlingException** | CloudTrail `LookupEvents` is rate-limited to two requests per second per account per region. Retry with a narrower window. | | **Tool reports the wrong region** | Set `AWS_REGION`, or check the `region` field on the configured AWS integration. | # How to use Tracer with AWS CloudWatch Source: https://opensre.com/docs/comparisons/Tracer_vs_AWS_CloudWatch Execution-level insight beneath cloud-native infrastructure monitoring AWS CloudWatch is a cloud-native monitoring service that reports the health, state, and utilization of AWS infrastructure and managed services. It provides metrics, logs, and events for resources such as EC2, ECS, EKS, AWS Batch, and storage services. Tracer complements CloudWatch by observing how workloads actually execute on that infrastructure and by attributing resource usage and cost to pipelines, tasks, and runs rather than to infrastructure alone. If you're new to Tracer or want a conceptual overview, see [How Tracer fits in your stack](/docs/comparisons/overview). ## What cloud-native monitoring does well Cloud-native monitoring tools such as CloudWatch are designed to report the state of cloud resources. They provide: * Metrics emitted by AWS services and infrastructure * Instance, container, and service health signals * Logs and events from managed resources * Alerts based on thresholds and service state These capabilities make CloudWatch effective for operating AWS environments and ensuring infrastructure and services are available and healthy. ## Where cloud-native metrics stop Cloud-native monitoring relies on service-reported metrics and resource state. It does not observe execution behavior inside workloads and does not have awareness of pipeline or task semantics. It does not show: * What individual processes or containers are doing at runtime * Whether allocated resources are actively used or idle * Execution stalls caused by I/O or network contention * Short-lived jobs that complete between reporting intervals * Why infrastructure remains allocated after workloads finish * How cost maps to specific pipelines, tasks, or tools As a result, performance issues and cost increases are often attributed to resources or time windows, rather than to the execution units that caused them. ## Infrastructure state versus execution behavior Cloud-native monitoring answers questions such as: * Which instances are running? * How much capacity is allocated? * Are services healthy? It does not answer: * Which task or process consumed those resources * Whether work was compute-bound, I/O-bound, or idle * Why infrastructure remained allocated without active execution Understanding these differences requires observing execution from the host, not just reporting infrastructure state. ## What Tracer adds Tracer observes execution directly from the operating system and container runtime. When used alongside cloud-native monitoring, it adds: * Execution-level visibility for pipelines, runs, tasks, and tools * Observed CPU, memory, disk, and network behavior * Insight into stalls, idle execution, and contention * Attribution of resource usage and cost to observed execution Tracer focuses on how resources are used, not just whether they exist. ## Infrastructure state analysis with Tracer/sweep Tracer/sweep extends Tracer's visibility to cloud infrastructure state using read-only access. It identifies: * Idle compute resources with no active execution * Orphaned nodes no longer associated with schedulers or clusters * Unattached or residual storage left behind by completed workloads These resources may not appear clearly in service-level dashboards once detached from active services. Tracer/sweep operates at the same cloud-account scope as cloud-native monitoring, but evaluates infrastructure state through observed execution rather than service metadata. ## Automation and control boundaries Tracer's products do not perform automated actions. * They do not start, stop, or resize infrastructure * They do not make predictions about future workload requirements * They do not act on forecasts or heuristics Recommendations are derived from observed execution behavior and infrastructure state. All decisions remain under user control. ## Example: service metrics versus observed execution CloudWatch shows high instance utilization during a pipeline run. Tracer reveals that: * CPU usage remains low * Tasks spend most of their time blocked on disk I/O * Instances remain allocated after execution completes This indicates execution inefficiency and infrastructure waste rather than insufficient capacity. ## Observability comparison ``` # Markdown syntax Source: https://opensre.com/docs/essentials/markdown Text, title, and styling in standard markdown ## Titles Best used for section headers. ```md theme={null} ## Titles ``` ### Subtitles Best used for subsection headers. ```md theme={null} ### Subtitles ``` Each **title** and **subtitle** creates an anchor and also shows up on the table of contents on the right. ## Text formatting We support most markdown formatting. Simply add `**`, `_`, or `~` around text to format it. | Style | How to write it | Result | | ------------- | ----------------- | ----------------- | | Bold | `**bold**` | **bold** | | Italic | `_italic_` | *italic* | | Strikethrough | `~strikethrough~` | ~~strikethrough~~ | You can combine these. For example, write `**_bold and italic_**` to get ***bold and italic*** text. You need to use HTML to write superscript and subscript text. That is, add `` or `` around your text. | Text Size | How to write it | Result | | ----------- | ------------------------ | ---------------------- | | Superscript | `superscript` | superscript | | Subscript | `subscript` | subscript | ## Linking to pages You can add a link by wrapping text in `[]()`. You would write `[link to google](https://google.com)` to [link to google](https://google.com). Links to pages in your docs need to be root-relative. Basically, you should include the entire folder path. For example, `[link to text](/writing-content/text)` links to the page "Text" in our components section. Relative links like `[link to text](../text)` will open slower because we cannot optimize them as easily. ## Blockquotes ### Singleline To create a blockquote, add a `>` in front of a paragraph. > Dorothy followed her through many of the beautiful rooms in her castle. ```md theme={null} > Dorothy followed her through many of the beautiful rooms in her castle. ``` ### Multiline > Dorothy followed her through many of the beautiful rooms in her castle. > > The Witch bade her clean the pots and kettles and sweep the floor and keep the fire fed with wood. ```md theme={null} > Dorothy followed her through many of the beautiful rooms in her castle. > > The Witch bade her clean the pots and kettles and sweep the floor and keep the fire fed with wood. ``` ### LaTeX Mintlify supports [LaTeX](https://www.latex-project.org) through the Latex component. 8 x (vk x H1 - H2) = (0,1) ```md theme={null} 8 x (vk x H1 - H2) = (0,1) ``` # Navigation Source: https://opensre.com/docs/essentials/navigation The navigation field in docs.json defines the pages that go in the navigation menu The navigation menu is the list of links on every website. You will likely update `docs.json` every time you add a new page. Pages do not show up automatically. ## Navigation syntax Our navigation syntax is recursive which means you can make nested navigation groups. You don't need to include `.mdx` in page names. ```json Regular Navigation theme={null} "navigation": { "tabs": [ { "tab": "Docs", "groups": [ { "group": "Getting Started", "pages": ["quickstart"] } ] } ] } ``` ```json Nested Navigation theme={null} "navigation": { "tabs": [ { "tab": "Docs", "groups": [ { "group": "Getting Started", "pages": [ "quickstart", { "group": "Nested Reference Pages", "pages": ["nested-reference-page"] } ] } ] } ] } ``` ## Folders Simply put your MDX files in folders and update the paths in `docs.json`. For example, to have a page at `https://yoursite.com/your-folder/your-page` you would make a folder called `your-folder` containing an MDX file called `your-page.mdx`. You cannot use `api` for the name of a folder unless you nest it inside another folder. Mintlify uses Next.js which reserves the top-level `api` folder for internal server calls. A folder name such as `api-reference` would be accepted. ```json Navigation With Folder theme={null} "navigation": { "tabs": [ { "tab": "Docs", "groups": [ { "group": "Group Name", "pages": ["your-folder/your-page"] } ] } ] } ``` ## Hidden pages MDX files not included in `docs.json` will not show up in the sidebar but are accessible through the search bar and by linking directly to them. # Reusable snippets Source: https://opensre.com/docs/essentials/reusable-snippets Reusable, custom snippets to keep content in sync One of the core principles of software development is DRY (Don't Repeat Yourself). This is a principle that applies to documentation as well. If you find yourself repeating the same content in multiple places, you should consider creating a custom snippet to keep your content in sync. ## Creating a custom snippet **Pre-condition**: You must create your snippet file in the `snippets` directory. Any page in the `snippets` directory will be treated as a snippet and will not be rendered into a standalone page. If you want to create a standalone page from the snippet, import the snippet into another file and call it as a component. ### Default export 1. Add content to your snippet file that you want to re-use across multiple locations. Optionally, you can add variables that can be filled in via props when you import the snippet. ```mdx snippets/my-snippet.mdx theme={null} Hello world! This is my content I want to reuse across pages. My keyword of the day is {word}. ``` The content that you want to reuse must be inside the `snippets` directory in order for the import to work. 2. Import the snippet into your destination file. ```mdx destination-file.mdx theme={null} --- title: My title description: My Description --- import MySnippet from '/snippets/path/to/my-snippet.mdx'; ## Header Lorem ipsum dolor sit amet. ``` ### Reusable variables 1. Export a variable from your snippet file: ```mdx snippets/path/to/custom-variables.mdx theme={null} export const myName = 'my name'; export const myObject = { fruit: 'strawberries' }; ``` 2. Import the snippet from your destination file and use the variable: ```mdx destination-file.mdx theme={null} --- title: My title description: My Description --- import { myName, myObject } from '/snippets/path/to/custom-variables.mdx'; Hello, my name is {myName} and I like {myObject.fruit}. ``` ### Reusable components 1. Inside your snippet file, create a component that takes in props by exporting your component in the form of an arrow function. ```mdx snippets/custom-component.mdx theme={null} export const MyComponent = ({ title }) => (

{title}

... snippet content ...

); ``` MDX does not compile inside the body of an arrow function. Stick to HTML syntax when you can or use a default export if you need to use MDX. 2. Import the snippet into your destination file and pass in the props ```mdx destination-file.mdx theme={null} --- title: My title description: My Description --- import { MyComponent } from '/snippets/custom-component.mdx'; Lorem ipsum dolor sit amet. ``` # Global Settings Source: https://opensre.com/docs/essentials/settings Mintlify gives you complete control over the look and feel of your documentation using the docs.json file Every Mintlify site needs a `docs.json` file with the core configuration settings. Learn more about the [properties](#properties) below. ## Properties Name of your project. Used for the global title. Example: `mintlify` An array of groups with all the pages within that group The name of the group. Example: `Settings` The relative paths to the markdown files that will serve as pages. Example: `["customization", "page"]` Path to logo image or object with path to "light" and "dark" mode logo images Path to the logo in light mode Path to the logo in dark mode Where clicking on the logo links you to Path to the favicon image Hex color codes for your global theme The primary color. Used for most often for highlighted content, section headers, accents, in light mode The primary color for dark mode. Used for most often for highlighted content, section headers, accents, in dark mode The primary color for important buttons The color of the background in both light and dark mode The hex color code of the background in light mode The hex color code of the background in dark mode Array of `name`s and `url`s of links you want to include in the topbar The name of the button. Example: `Contact us` The url once you click on the button. Example: `https://mintlify.com/docs` Link shows a button. GitHub shows the repo information at the url provided including the number of GitHub stars. If `link`: What the button links to. If `github`: Link to the repository to load GitHub information from. Text inside the button. Only required if `type` is a `link`. Array of version names. Only use this if you want to show different versions of docs with a dropdown in the navigation bar. An array of the anchors, includes the `icon`, `color`, and `url`. The [Font Awesome](https://fontawesome.com/search?q=heart) icon used to feature the anchor. Example: `comments` The name of the anchor label. Example: `Community` The start of the URL that marks what pages go in the anchor. Generally, this is the name of the folder you put your pages in. The hex color of the anchor icon background. Can also be a gradient if you pass an object with the properties `from` and `to` that are each a hex color. Used if you want to hide an anchor until the correct docs version is selected. Pass `true` if you want to hide the anchor until you directly link someone to docs inside it. One of: "brands", "duotone", "light", "sharp-solid", "solid", or "thin" Override the default configurations for the top-most anchor. The name of the top-most anchor Font Awesome icon. One of: "brands", "duotone", "light", "sharp-solid", "solid", or "thin" An array of navigational tabs. The name of the tab label. The start of the URL that marks what pages go in the tab. Generally, this is the name of the folder you put your pages in. Configuration for API settings. The base url for all API endpoints. If `baseUrl` is an array, it will enable for multiple base url options that the user can toggle. The authentication strategy used for all API endpoints. The name of the authentication parameter used in the API playground. If method is `basic`, the format should be `[usernameName]:[passwordName]` The default value that's designed to be a prefix for the authentication input field. E.g. If an `inputPrefix` of `AuthKey` would inherit the default input result of the authentication field as `AuthKey`. Configurations for the API playground Whether the playground is showing, hidden, or only displaying the endpoint with no added user interactivity `simple` Enabling this flag ensures that key ordering in OpenAPI pages matches the key ordering defined in the OpenAPI file. This behavior will soon be enabled by default, at which point this field will be deprecated. A string or an array of strings of URL(s) or relative path(s) pointing to your OpenAPI file. Examples: ```json Absolute theme={null} "openapi": "https://example.com/openapi.json" ``` ```json Relative theme={null} "openapi": "/openapi.json" ``` ```json Multiple theme={null} "openapi": ["https://example.com/openapi1.json", "/openapi2.json", "/openapi3.json"] ``` An object of social media accounts where the key:property pair represents the social media platform and the account url. Example: ```json theme={null} { "x": "https://x.com/mintlify", "website": "https://mintlify.com" } ``` One of the following values `website`, `facebook`, `x`, `discord`, `slack`, `github`, `linkedin`, `instagram`, `hacker-news` Example: `x` The URL to the social platform. Example: `https://x.com/mintlify` Configurations to enable feedback buttons Enables a button to allow users to suggest edits via pull requests Enables a button to allow users to raise an issue about the documentation Customize the dark mode toggle. Set if you always want to show light or dark mode for new users. When not set, we default to the same mode as the user's operating system. Set to true to hide the dark/light mode toggle. You can combine `isHidden` with `default` to force your docs to only use light or dark mode. For example: ```json Only Dark Mode theme={null} "modeToggle": { "default": "dark", "isHidden": true } ``` ```json Only Light Mode theme={null} "modeToggle": { "default": "light", "isHidden": true } ``` A background image to be displayed behind every page. See example with [Infisical](https://infisical.com/docs) and [FRPC](https://frpc.io). # FAQ Source: https://opensre.com/docs/faq Frequently asked questions about OpenSRE OpenSRE is an open-source framework for building AI SRE agents that investigate production incidents using your existing observability stack, cloud context, and runbooks. It connects to 60+ integrations, runs a structured RCA pipeline, and delivers findings to Slack, local files, or your messaging channel of choice. Install OpenSRE, run onboarding, then investigate a sample alert: ```bash theme={null} curl -fsSL https://install.opensre.com | bash opensre onboard opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` See the full walkthrough in [Quickstart](/docs/quickstart). Most users start with the **local CLI** (`curl` or Homebrew install, then `opensre onboard`). For org-wide hosted connectors, use [Enterprise setup](/docs/install-enterprise) at [app.tracer.cloud](https://app.tracer.cloud). Self-hosted gateway deployment is documented in [Deployment](/docs/deployment) — Docker, env vars, and health-check curl examples included. Yes. Deploy the gateway as a FastAPI app using the repo Dockerfile or your host's Python workflow. Set `LLM_PROVIDER` and the matching provider key (for example `ANTHROPIC_API_KEY` when `LLM_PROVIDER=anthropic`). See [Deployment](/docs/deployment) and [Environment variables](/docs/configuration/environment-variables). OpenSRE supports Anthropic, OpenAI, OpenRouter, Gemini, Bedrock, Azure OpenAI, and CLI-backed providers via `LLM_PROVIDER` plus the matching API key. Full provider matrix, defaults, and troubleshooting: [LLM providers](/docs/llm-providers). Usually yes. OpenSRE integrates with observability (Datadog, Grafana, Sentry), cloud (AWS, Kubernetes), incident tools (PagerDuty, OpsGenie), databases, and messaging (Slack, Telegram). Browse the [Integrations overview](/docs/integrations-overview) catalog and run `opensre integrations verify` after setup. Running `opensre` starts the interactive incident-response shell. Describe issues in plain language, stream investigations live, and ask follow-up questions in the same session. Slash commands (`/help`, `/investigate`, `/verify datadog`) are documented in [Interactive Shell Commands](/docs/interactive-shell-commands). OpenSRE supports reversible [masking](/docs/masking) before external LLM calls and command-history redaction via [Interactive Shell Privacy](/docs/interactive-shell-privacy). Anonymous product telemetry can be disabled with `OPENSRE_NO_TELEMETRY=1`. For vulnerability reports, email `support@opensre.com`. # Features Source: https://opensre.com/docs/features What OpenSRE can do today ## Investigation workflow OpenSRE runs a structured RCA pipeline for each alert: 1. **Extract context** from the alert payload and connected integrations 2. **Plan evidence collection** across logs, metrics, deploys, and dependencies 3. **Test hypotheses** in a tool-calling loop until confidence is high enough to stop 4. **Publish findings** as `problem.md`, `theory/hypothesis_*.md`, and `report.md` (or JSON with `--output`) **Try it:** ```bash theme={null} opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` **Gotcha:** The investigation uses whatever integrations are connected at run time. Run `opensre integrations verify` before your first production alert. See [Investigations overview](/docs/investigation-overview). ## Integrations (60+ tools) Connect observability, cloud, databases, incident management, messaging, and workflow systems so investigations query the same tools your engineers use. **Try it:** ```bash theme={null} opensre integrations setup opensre integrations verify datadog ``` **Config:** Credentials live in `.env` and `~/.opensre/integrations.json`. See [Environment variables](/docs/configuration/environment-variables). **Gotcha:** [Multi-instance integrations](/docs/multi-instance-integrations) use env vars like `DD_INSTANCES` or `GRAFANA_INSTANCES` for prod/staging pairs. See [Integrations overview](/docs/integrations-overview). ## Interactive shell The TTY REPL (`opensre` with no subcommand) supports plain-language incident descriptions, slash commands, session history, fleet monitoring, and scheduled deliveries. **Try it:** ```bash theme={null} opensre # at the prompt: /help /verify datadog ``` **Config:** Set `LLM_PROVIDER` and the matching API key before starting. Use `/effort` (OpenAI/Codex) or `OPENSRE_REASONING_EFFORT` to tune reasoning depth. **Gotcha:** The action planner never denies a turn — describe compound requests in one message and it chains slash commands automatically. See [Interactive Shell Commands](/docs/interactive-shell-commands). ## Masking and safe sharing Reversible masking replaces sensitive infrastructure identifiers (pods, clusters, hostnames, account IDs) with stable placeholders before external LLM calls, then restores originals in user-facing output. **Try it:** Enable masking in onboarding or set the masking env vars documented on [Masking](/docs/masking), then run an investigation and confirm placeholders appear in LLM-facing evidence but not in your local `report.md`. **Gotcha:** Command-history redaction is separate — see [Interactive Shell Privacy](/docs/interactive-shell-privacy). ## Remote runtime investigations Investigate deployed OpenSRE services by name. OpenSRE gathers live deployment status, recent logs, and health probes, then runs the standard RCA pipeline. **Try it:** From the REPL: `/remote list`, then describe the service you want to investigate. **Config:** Requires a configured remote runtime target. See [Remote runtime investigation](/docs/remote-runtime-investigation). **Gotcha:** Remote investigations use the same integration catalog as local runs — connect observability tools first. # Fix a Sentry issue Source: https://opensre.com/docs/fix-sentry-issue Paste a Sentry issue URL and have the Pi coding agent propose a fix — as a reviewable diff or a pull request. The **Sentry issue-fix tool** lets you point OpenSRE at a [Sentry](https://sentry.io) issue and have the [Pi](https://pi.dev) coding agent propose a fix. OpenSRE fetches the issue context, runs Pi in your current repository, and returns a summary plus the git diff. By default it **only edits the working tree** so you can review the diff. When you ask it to `open_pr` (and enable the ship switch), it also commits the fix to a fresh branch, pushes it, and opens a **pull request** — it **never pushes to your base/`main` branch**. This is a **mutating** tool — it changes files on disk and can open a PR. It is **disabled by default**: the fix step needs `PI_ISSUE_FIX_ENABLED=1`, Sentry configured, and the Pi CLI installed. Opening a PR needs a **second** switch, `PI_ISSUE_FIX_SHIP_ENABLED=1`, plus a GitHub token. Enable each deliberately. ## Quick reference | Env var | What it does | | --------------------------------------- | ------------------------------------------------------------------------------- | | `PI_ISSUE_FIX_ENABLED` | Opt-in switch for the fix step. Set to `1` to enable. Off by default. | | `PI_ISSUE_FIX_SHIP_ENABLED` | Second opt-in required to open a PR (`open_pr`). Off by default. | | `GITHUB_TOKEN` / `GH_TOKEN` | GitHub token used to open the pull request (needs `repo`/`pull_request` write). | | `SENTRY_ORG_SLUG` / `SENTRY_AUTH_TOKEN` | Sentry org + token used to fetch the issue (`SENTRY_URL` for self-hosted). | | `PI_CODING_MODEL` | Pi model used for the fix (shared with the Pi coding tool). | | `PI_CODING_WORKSPACE` | Repository Pi edits. Defaults to the current directory. | | `PI_CODING_TIMEOUT_SECONDS` | Per-run timeout (default 600, clamped 60–1800). | | Parameter | What it does | | ------------ | --------------------------------------------------------------------------------------------- | | `sentry_url` | The Sentry issue URL to fix (required). | | `open_pr` | When `true`, commit the fix to a fresh branch and open a PR. Defaults to `false` (diff only). | ## Enable it 1. Configure Sentry (org + token) and install/authenticate the Pi CLI: ```bash theme={null} export SENTRY_ORG_SLUG=your-org export SENTRY_AUTH_TOKEN=... # token with issue read access npm i -g @earendil-works/pi-coding-agent ``` 2. Turn the tool on: ```bash theme={null} export PI_ISSUE_FIX_ENABLED=1 export PI_CODING_MODEL=anthropic/claude-haiku-4-5 # optional ``` 3. (Optional) Let OpenSRE open the fix as a pull request: ```bash theme={null} export PI_ISSUE_FIX_SHIP_ENABLED=1 export GITHUB_TOKEN=... # token with PR-create access to the repo ``` ## Using it from the interactive shell Once enabled, just ask in plain language in `opensre` — the action agent recognizes the intent and runs the tool: ``` fix this sentry issue https://your-org.sentry.io/issues/12345/ and open a pull request ``` Say "open a pull request" (or "ship it") to open a PR; leave it out to only get the diff. Use a *fix* verb — "investigate"/"diagnose" routes to the read-only investigation pipeline instead. ## How it works 1. You paste a Sentry issue URL (e.g. `https://your-org.sentry.io/issues/12345/`) and ask OpenSRE to fix it. 2. OpenSRE resolves the issue from Sentry and builds a short, **masked** task (title, error, culprit, location) — your Sentry token is never sent to Pi. 3. Pi edits the current repository to implement the fix. 4. You get back `success`, a `summary`, `changed_files`, and the `diff` to review. 5. If you asked for `open_pr` (and shipping is enabled), OpenSRE commits the change to a fresh `opensre/sentry-fix--` branch, pushes it, opens a PR into your base branch, and returns the `branch_name`, `pr_url`, and `pr_number`. ## Opening a pull request Ask OpenSRE to *fix the issue and open a PR*. The fix always lands on a new namespaced branch and is proposed via PR: * OpenSRE **never** commits to or pushes your base/`main` branch, and never force-pushes. * The PR is opened **into** the repo's default branch from the new branch, with a body linking the Sentry issue and listing the changed files. * If the fix succeeds but the PR can't be opened (e.g. missing token, push rejected), you still get the `diff` and `changed_files` back, plus the `error_kind`, so you can ship it manually. ## Supported URLs * `https://.sentry.io/issues//` * `https://sentry.io/organizations//issues//` * self-hosted `https://sentry..com/organizations//issues//` ## Notes * Without `open_pr`, nothing is committed or pushed; you review the diff and decide what to do. * If the tool is disabled, the URL is unsupported, Sentry is unconfigured, Pi is missing, or a PR can't be opened, it returns a clear `error_kind` instead of silently doing the wrong thing. # Local Agent Fleet Source: https://opensre.com/docs/fleet Slash-command surface for monitoring, coordinating, and exchanging context between local AI agents (Claude Code, Cursor, Aider, ...) ## Overview OpenSRE treats every other AI agent running on your machine — Claude Code, Cursor, Aider, Codex CLI, Gemini, and friends — as a microservice and applies normal SRE practice: golden signals, SLOs, and incident response. The whole fleet view lives behind one slash command in the interactive shell: ```text theme={null} > /fleet ``` Subcommands drill into specific surfaces. Below: how the agent fleet is discovered (`opensre fleet scan`), the dashboard itself (`/fleet`), then live tail of agent stdout (`/fleet trace`), then the cross-agent context bus (`/fleet bus`). ## `opensre fleet scan` — discover running agent sessions `opensre fleet scan` enumerates running AI-coding-agent sessions visible to the current user. The classifier inspects the process table (via `ps -axo pid,ppid,args`) and labels each candidate by executable plus known argv shapes — no PID is registered until you ask for it with `--register`. ### Strict mode (default) Recognizes the typical CLI invocations for Claude Code, Cursor, Aider, Codex, Gemini, and Antigravity (`agy`). For Claude Code specifically, all of the following process shapes are treated as a session: ```text theme={null} claude claude --resume claude -r claude --prefill "" claude --print "" claude -p "" claude --continue claude -c claude code … claude --input-format stream-json … claude --output-format stream-json … ``` Equals-form flags (e.g. `claude --resume=`, `claude --print=`) are accepted for any flag in the list above. The Claude Desktop GUI and its Electron helpers are filtered out cross-platform — `Claude.app/Contents/`, `Claude Helper (Renderer)`, `/snap/claude/`, `/usr/lib/claude-desktop`, AppImage mount points (`/.mount_Claude…`), and the Windows `Program Files\Claude\` / `AppData\Local\Programs\Claude\` install locations are recognized as desktop artifacts and never labeled as `claude-code`. The negative filter matches against `argv[0]` only, so a CLI installed under a Claude-flavored prefix (for example `/opt/Claude/claude`) is still surfaced. The same cross-platform desktop filter also rejects **Codex Desktop** (`Codex.app/Contents/MacOS/Codex`, `/snap/codex/`, `/.mount_Codex…`, Flatpak `com.openai.codex`, Windows `Program Files\Codex\` and `AppData\Local\Programs\codex\`) and **Cursor Desktop** (`Cursor.app/Contents/MacOS/Cursor`, `/snap/cursor/`, `/.mount_Cursor…`, Flatpak `com.cursor.cursor`, Windows `Program Files\Cursor\` and `AppData\Local\Programs\cursor\`), so neither GUI is mislabeled as the `codex` or `cursor` CLI in strict or `--all` mode. The macOS hints deliberately target only the main bundle binary so Electron helper subprocesses (e.g. `Cursor Helper (Plugin)` running Cursor's AI agent) remain eligible for the loose `--all` matcher. ### Loose mode (`--all`) `opensre fleet scan --all` relaxes the argv requirements so that helper and broker processes whose argv contains agent-shaped tokens are also surfaced. The Claude / Codex / Cursor Desktop negative filter is still applied — `--all` never mislabels desktop processes as `claude-code`, `codex`, or `cursor`. ### Registering discovered sessions ```text theme={null} opensre fleet scan --register ``` writes every discovered session into the local agent registry so that the rest of the fleet surface (`/fleet`, `/fleet trace`, `/fleet bus`) can target it by PID. Without `--register`, the command is read-only. ## `/fleet` — fleet dashboard The dashboard renders a seven-column table of every registered or discovered local AI agent. Run from the interactive REPL: ```text theme={null} > /fleet agents agent pid uptime cpu% tokens/min $/hr status claude-code-8421 8421 2h12m 18.4 320 $0.08 running codex-13442 13442 11m 4.2 175 $0.04 running cursor-agent-9999 9999 47m 0.6 - - running ``` ### Column data sources | Column | Source | Notes | | ------------ | --------------------------------------------- | ---------------------------------------------------------- | | `agent` | `AgentRecord.name` | Registered name or discovery-generated `-`. | | `pid` | `AgentRecord.pid` | OS process id. | | `uptime` | sampler probe (psutil `create_time`) | Compact form: `45s` / `12m` / `2h12m` / `3d4h`. | | `cpu%` | sampler probe (psutil) | Trailing 100 ms `cpu_percent`. | | `tokens/min` | per-PID 60 s rolling window | Real for `claude-code` and `codex`; `-` for the rest. | | `$/hr` | observed cost from token rate × model pricing | Renders `-` when the model is unknown. | | `status` | sampler probe (psutil) | `running` / `sleeping` / `zombie` / etc. | ### `tokens/min` semantics The cell shows the **sum of tokens emitted in the trailing 60 seconds**, scaled to a per-minute figure. Three states: * **Real value** (`320`, `1.2k`): the agent's provider has a working meter and the on-disk session log was readable. Today this is `claude-code` (reads `~/.claude/projects//.jsonl`) and `codex` (reads `$CODEX_HOME/sessions/YYYY/MM/DD/rollout-*.jsonl`, default `$CODEX_HOME=~/.codex`). * **`0`**: the agent was observed at least once but emitted nothing in the last 60 s. Honest UX — distinguishes "idle session" from "not observable". * **`-`**: never observed. Either the provider has no meter yet (cursor, aider, gemini-cli, antigravity-cli, opencode, kimi, copilot in this PR), the session log is unreadable (rare; macOS hardened-runtime processes can deny `psutil.cwd()`), or the interactive REPL is not running (non-interactive `opensre fleet list` never starts the sampler). Claude Code cache-read/cache-creation tokens are included in the visible activity count because they are separate input work. Codex `cached_input_tokens` are treated as a discounted subset of `input_tokens`, so they affect `$/hr` but are not added again to the visible `tokens/min` total. ### `$/hr` semantics `$/hr` is a **projected hourly burn rate** derived from the same trailing 60 s window the `tokens/min` column reports — not the actual spend over the last hour: ```text theme={null} $/hr = cost_of_usage_buckets_in_the_trailing_60s_window(model) × 60 ``` Reads as "if the agent sustains the current ritmo for one hour, it will cost this much". Useful as an operational signal: it tracks `cpu%` in spirit, reacts immediately when an agent goes idle or switches model, and keeps memory bounded (a \~12-entry deque per PID at the default 5 s tick instead of an unbounded hour-long history). If you need actual spend over the last hour, that's a different metric — open a follow-up issue. Pricing uses per-bucket rates for input, output, cached input, cache read, and cache creation where the provider emits those counters. The local pricing table is a vendored `models.dev` snapshot for the Claude Code and Codex models supported here, with optional `agents.yaml` input/output overrides. Pricing returns `None` for unknown models and the cell falls back to `-`; the dashboard never invents a rate. The yaml `hourly_budget_usd` field is **not** the cell content — it's reserved for a future budget-alarm feature. Model resolution order (highest to lowest): 1. **NDJSON model hint** from the meter (most accurate — reflects the model the running session is currently using). 2. **`agents.yaml` override** (`AgentBudget.model`): ```yaml theme={null} agents: claude-code-8421: model: claude-sonnet-4-5 codex-13442: model: gpt-5-codex ``` 3. **Provider env var** (`CLAUDE_CODE_MODEL`, `CODEX_MODEL`) read via psutil. May fail on macOS hardened-runtime processes, in which case the cell falls back to `-` unless one of the two higher priority sources resolved. ### Provider coverage today | Provider | `tokens/min` | `$/hr` | | --------------------------------------------------------------------------------- | ------------ | ------ | | `claude-code` | real | real | | `codex` | real | real | | `cursor`, `aider`, `gemini-cli`, `antigravity-cli`, `opencode`, `kimi`, `copilot` | `-` | `-` | Adding a real meter for another provider is a self-contained follow-up PR: implement the `TokenMeter` and `TokenSource` protocols, register both instances, add tests. No changes to the sampler, tracker, view, or pricing layer are needed. ## `/fleet trace` — live stdout tail `/fleet trace ` opens a live tail of an agent's stdout inside the OpenSRE interactive shell — the equivalent of `kubectl logs -f` for the local AI agent fleet. Use it when the `/fleet` dashboard shows an agent that looks **stuck**, **looping**, or **noisy** and you want to see what it's actually printing without leaving the REPL. ```text theme={null} > /fleet trace 8421 trace claude-code (pid 8421) Ctrl+C to stop … live agent output … ^C · trace ended ``` ### Trace usage ```text theme={null} /fleet trace ``` `` is the operating-system process id of the agent to attach to. The pid does **not** have to be in the OpenSRE registry; if it is, the agent's registered name is shown in the header. Otherwise the header falls back to `pid `. ### Platform support Only **regular files** backing fd 1 of the target process are supported. TTY/PTY/pipe/socket/anon-inode targets are rejected at attach time with a precise reason — tailing those would compete with the legitimate consumer for bytes and produce corrupted output. | Platform | Resolver | Supported targets | | ----------------------- | ------------------------------------------- | ----------------- | | **Linux** | `os.readlink("/proc//fd/1")` | regular files | | **macOS** (best-effort) | `lsof -F ftn -p `, only `t REG` blocks | regular files | | **Windows** | not supported | — | The most common useful case is an agent whose stdout was redirected to a log file (for example `claude > ~/.claude/log` or `nohup`-launched agents). TTY-bound foreground processes cannot be tailed. If a target cannot be tailed, `/fleet trace` exits with one of: * `cannot trace …: stdout is on a terminal; live tail not supported` * `cannot trace …: stdout is a pipe; live tail not supported` * `cannot trace …: stdout is a socket; live tail not supported` * `cannot trace …: no such pid ` * `cannot trace …: stdout target /path no longer exists` * `cannot trace …: cannot inspect pid (permission denied)` ### Memory The live view is bounded by a **4 MiB ring buffer per session**. When the buffer fills, the oldest whole chunks are dropped first, so the visible tail always reflects the latest output. Internally the reader thread also publishes through a bounded queue and drops the oldest chunk on overflow — burst writers cannot blow up memory. There is **no backlog replay**: only output emitted *after* attach is shown. The reader seeks the file to EOF on attach. ### Stopping a trace A single **Ctrl+C** returns to the REPL prompt. The session is closed, the reader thread joins, and the file descriptor is released. This is deliberately different from the LLM-streaming surface (`/fleet`, `/investigate` and friends), where a Ctrl+C double-press is required so a stray keypress doesn't abort an in-flight response. ### Trace limitations * **stdout only.** Stderr (fd 2) is not tailed in this version. * **No backlog replay.** Pre-attach bytes are not visible. * **Not for TTY/PTY targets.** Foreground processes whose stdout is the controlling terminal cannot be tailed; a future change may add PTY interception for OpenSRE-spawned agents. * **Log rotation is not detected.** If the underlying file is rotated or replaced (logrotate-style), the tail keeps following the original inode until the process exits. * **No secret redaction.** Output is rendered as raw bytes (with UTF-8 decoded under `errors="replace"`). Redaction of secrets in the live tail is tracked separately under the `monitor-local-agents` Phase 3 hygiene work. * **Quiet stdout while the PID is still alive.** The reader follows file EOF like `tail -f`: if the process stops writing while it remains alive, the last chunk stays on screen and nothing new appears until more bytes land or you detach. That is normal idling — not necessarily a exited or stuck agent reader. * **ANSI and terminal sequences.** Trace output passes through Rich with ANSI interpretation, same trust model as dumping `kubectl logs` into a TTY: buggy or hostile agents can emit control sequences affecting the viewer. Only trace processes you trust; there is no sandboxing step. ## `/fleet bus` — shared context channel The bus is an opt-in, local-only pub/sub channel that carries findings between agents. One agent publishes a finding (e.g. *"the auth bug is in `services/auth.py:42`"*) and every attached subscriber sees it live. The inspector is the REPL itself: ```text theme={null} > /fleet bus tailing /fleet bus — Ctrl-C to exit [claude-code:8421] services/auth.py:42 — null deref on missing token [cursor:9133] services/auth.py:42 — confirmed, repro on commit abc123 ^C (detached) > ``` Ctrl-C returns to the prompt. Messages already published are not replayed to late subscribers. The bus provides **at-most-once delivery with no ordering guarantees** — a frame may be dropped if a subscriber's socket is slow or disconnected, and two publishers writing concurrently may be interleaved in different orders at different subscribers. Do not assume per-publisher or global FIFO ordering. ### Transport * **Socket**: Unix-domain stream socket at `~/.opensre/agents-bus.sock`. * **PID sidecar**: `~/.opensre/agents-bus.sock.pid` (mode `0600`). The broker writes its PID here on `start()` and removes it on `stop()`. The liveness probe used by every `publish()` / `subscribe()` reads this file rather than connecting to the socket — connection probing would otherwise register a short-lived phantom subscriber on every call. **The directory must be writable.** If the PID file write fails (disk full, permission denied, ...), the broker refuses to start and the `OSError` propagates to the caller. This is intentional: silently running without a sidecar would let peers see the broker as dead, unlink its socket, and silently split the bus. * **Permissions**: `0600` — only the user who started the broker can read or write it. * **Wire format**: JSON Lines (one JSON object per `\n`-terminated frame). * **Topology**: self-electing broker. The first `publish()` or `subscribe()` call that finds no live socket binds it and runs an in-process daemon thread that fans frames out. Other processes attach as plain clients. If the broker dies, the next operation re-elects — agents can publish and subscribe even when OpenSRE itself is not running. ### Message schema The wire payload mirrors the shape of `evidence` records in `core/state/models.py` so a finding can later be lifted into an investigation without renaming fields. | Field | Type | Required | Notes | | ---------------- | ------ | -------- | ----------------------------------------------------------------------------------- | | `agent` | string | yes | `":"`, e.g. `"claude-code:8421"`. Same convention as `WriteEvent.agent`. | | `topic` | string | yes | `"finding"` is the canonical value; other topics are reserved for future phases. | | `summary` | string | yes | One-line human-readable description. | | `source` | string | no | One of the `EvidenceSource` literals (`github`, `datadog`, ...) or free-form. | | `path` | string | no | `"file.py:42"` style location. Optional. | | `data` | object | no | Free-form payload. Default `{}`. | | `id` | string | no | UUID. Generated if omitted. | | `timestamp` | string | no | ISO-8601 UTC. Generated if omitted. | | `schema_version` | int | no | Currently `1`. | Example frame on the wire (single line, broken here for readability): ```json theme={null} { "agent": "claude-code:8421", "topic": "finding", "summary": "null deref on missing token", "source": "github", "path": "services/auth.py:42", "data": {"commit": "abc123"}, "id": "f4c4...", "timestamp": "2026-05-09T15:04:42+00:00", "schema_version": 1 } ``` ### Publishing from another agent Any process that can speak Unix-domain sockets can publish. The simplest path is to import the helper: ```python theme={null} from tools.system.fleet_monitoring import BusMessage, publish publish(BusMessage( agent="claude-code:8421", topic="finding", summary="null deref on missing token", source="github", path="services/auth.py:42", data={"commit": "abc123"}, )) ``` Publishers without a Python dependency on OpenSRE can connect directly: ```bash theme={null} python - <<'EOF' import json, os, socket, uuid, datetime sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sock.connect(os.path.expanduser("~/.opensre/agents-bus.sock")) sock.sendall((json.dumps({ "agent": "claude-code:8421", "topic": "finding", "summary": "null deref on missing token", "path": "services/auth.py:42", "id": str(uuid.uuid4()), "timestamp": datetime.datetime.now(datetime.UTC).isoformat(), "schema_version": 1, }) + "\n").encode()) sock.close() EOF ``` ### Limits and trust boundary * **Local-only.** The bus never leaves the machine. The socket has no network binding. * **Trusted-peer channel — treat findings as unverified input.** The bus has no authentication beyond filesystem permissions: any process running as your user can publish arbitrary findings. This is intentional — the bus is designed for cooperative agents, not adversarial ones. Downstream consumers (agents, the REPL, investigation state) **must not act on a bus finding without independent confirmation**; treat it as a hint or lead, not a verified fact. A compromised or misbehaving agent on the same user account can inject any payload it likes. * **Frame cap.** Frames over 64 KiB are dropped with a warning — a finding payload that big is almost certainly a bug. * **At-most-once, unordered delivery.** A frame is dropped silently if a subscriber is slow or disconnected at broadcast time. Two publishers writing concurrently may arrive in different orders at different subscribers. Do not build logic that depends on delivery guarantees or ordering. * **No replay buffer.** Subscribers see only what is published *after* they attach. A persistent ring buffer is a candidate for a follow-up phase. ## Related * `/fleet` — the registered fleet dashboard. * `/fleet budget` — per-agent hourly budgets. * `/fleet conflicts` — file-write conflicts between local AI agents. # GitHub Source: https://opensre.com/docs/github Connect GitHub so OpenSRE can read and write issues/PRs, search code, inspect commits, and correlate changes with incidents OpenSRE connects to GitHub so the agent can work with issues, pull requests, repositories, Actions, and code — and correlate recent commits with incidents. ## What you can do Once GitHub is connected, the interactive-shell **action agent** uses **`github_cli`** (authenticated `gh`) for flexible reads and writes — create/list/view issues and PRs, assign, label, comment, merge, search, releases, workflow runs, and `gh api` — without a separate approval gate. Prefer this over shell `gh` / `!gh`. These requests stay on the action path (not the conversational gather/answer loop). | Area | How OpenSRE does it | | -------------------------------------------------- | ------------------------------------------------------ | | Ad-hoc issues, PRs, repos, search, API | `github_cli` (action agent) | | Engineering digests, PR readiness, security alerts | [GitHub workflow tools](github-workflow-tools) | | Security and quality fixes and PRs | [GitHub security and quality fix](github-security-fix) | | Failing PR CI fixes and branch pushes | [GitHub PR CI fix](github-ci-fix) | | Slack → GitHub issue create/update/close | Propose + approve via workflow mutation tools | | Investigation: code, commits, files, issue search | Dedicated GitHub investigation tools | | Failed deploys / workflow runs | [GitHub Actions tools](integrations/github-actions) | ## First launch (macOS & Windows) The first time you launch the interactive shell on macOS or Windows, OpenSRE asks you to sign in to GitHub in your browser before the prompt appears. This runs the same browser device-flow sign-in as Option 1 below, then connects GitHub automatically. Sign in once and OpenSRE remembers it for future launches. You can bypass this step — for example if GitHub sign-in is unavailable: ```bash theme={null} OPENSRE_SKIP_GITHUB_LOGIN=1 opensre ``` The first-launch prompt never runs on Linux or in CI/automation, and is skipped when GitHub is already configured. ## Prerequisites * GitHub account with repository access * One of: browser sign-in (recommended), a personal access token, or GitHub Copilot MCP access * For chat `github_cli`: the `gh` binary on `PATH` (OpenSRE supplies the token). Homebrew installs of OpenSRE pull `gh` via formula dependency; the curl installer soft-installs it when missing. ## Setup ### Option 1: Interactive CLI (browser sign-in) ```bash theme={null} opensre integrations setup ``` Select **GitHub** when prompted, then choose **Authorize in browser**. OpenSRE opens GitHub's device authorization page and prints a one-time code — approve it in your browser and the token is captured automatically. No personal access token is required. This uses GitHub's OAuth device flow, which has no client secret. The public OAuth App client id ships with OpenSRE; override it with `OPENSRE_GITHUB_OAUTH_CLIENT_ID` if you register your own app. If you prefer, the same prompt lets you **paste a token (PAT)** instead. ### Option 2: Environment variables ```bash theme={null} GITHUB_MCP_AUTH_TOKEN=ghp_your_personal_access_token GITHUB_MCP_URL=https://api.githubcopilot.com/mcp/ # default GITHUB_MCP_MODE=streamable-http # default GITHUB_MCP_TOOLSETS=repos,issues,pull_requests,actions # default ``` | Variable | Default | Description | | -------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------- | | `GITHUB_MCP_AUTH_TOKEN` | — | GitHub personal access token. Required unless you authorize in the browser (Option 1) | | `GITHUB_TOKEN` / `GH_TOKEN` | — | Alternate token env names also accepted by `github_cli` and some workflow tools | | `GITHUB_MCP_URL` | `https://api.githubcopilot.com/mcp/` | GitHub MCP server URL | | `GITHUB_MCP_MODE` | `streamable-http` | Transport mode: `streamable-http`, `sse`, or `stdio` | | `GITHUB_MCP_TOOLSETS` | `repos,issues,pull_requests,actions` | Comma-separated toolsets to enable | | `GITHUB_MCP_COMMAND` | — | Command to run (required for `stdio` mode only) | | `GITHUB_MCP_ARGS` | — | Space-separated args for `stdio` mode | | `OPENSRE_GITHUB_OAUTH_CLIENT_ID` | *(built-in)* | OAuth App client id for browser sign-in (device flow). Override to use your own app | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "github-prod", "service": "github", "status": "active", "credentials": { "url": "https://api.githubcopilot.com/mcp/", "mode": "streamable-http", "auth_token": "ghp_your_token", "toolsets": ["repos", "issues", "pull_requests", "actions"] } } ] } ``` ## Creating a personal access token 1. In GitHub, go to **Settings** → **Developer settings** → **Personal access tokens** → **Tokens (classic)** 2. Click **Generate new token** 3. Select the following scopes: `repo`, `read:org` (add write scopes if you want chat mutations via `github_cli`) 4. Copy the token For GitHub Enterprise Server, set `GITHUB_MCP_URL` to your enterprise MCP endpoint. ## Transport modes | Mode | When to use | | ----------------- | ---------------------------------------------------------------------------------- | | `streamable-http` | Default. Works with GitHub Copilot MCP and most hosted instances | | `sse` | For older MCP servers using Server-Sent Events | | `stdio` | For running a local MCP server process (`npx @modelcontextprotocol/server-github`) | ## Verify ```bash theme={null} opensre integrations verify github ``` Expected output: ``` Service: github Status: passed Detail: GitHub MCP validated for your-username; discovered 18 tools including repository source investigation helpers ``` ## Troubleshooting | Symptom | Fix | | --------------------------------------- | ------------------------------------------------------------------------------------------------------ | | **Authentication failed** | Check that the token has `repo` scope and is not expired | | **Required tools missing** | Ensure toolsets include `repos` — it provides `get_file_contents`, `list_commits`, etc. | | **`github_cli` fails / gh not found** | Install [GitHub CLI](https://cli.github.com/) so `gh` is on `PATH`; confirm token via verify | | **Connection refused** | Verify `GITHUB_MCP_URL` is reachable and the MCP server is running | | **Browser sign-in unavailable** | Set `OPENSRE_GITHUB_OAUTH_CLIENT_ID` to a device-flow-enabled OAuth App, or fall back to pasting a PAT | | **First-launch sign-in is blocking me** | Set `OPENSRE_SKIP_GITHUB_LOGIN=1` to bypass the first-launch GitHub prompt | ## Security best practices * Prefer the **least privilege** that matches how you use OpenSRE (read-only for investigation-only; write scopes if you want chat to create/edit issues and PRs). * Limit token scope to the repositories OpenSRE needs. * Store the token in `.env`, not in source code. # Fix GitHub PR CI Source: https://opensre.com/docs/github-ci-fix Use OpenSRE to inspect failing GitHub Actions checks on a pull request, apply a fix, and push it to the PR branch. OpenSRE can inspect failing GitHub Actions checks on a pull request, apply a code fix in your local checkout, commit it, and push it back to the PR's existing head branch. This is mutating. OpenSRE asks before checking out the PR branch, editing files, committing, and pushing. ## Set it up ```bash theme={null} export CODING_WORKSPACE=/path/to/repo # optional; defaults to cwd export GITHUB_TOKEN=... # repo write access plus workflow/check read access ``` OpenSRE uses the configured coding-agent backend through `CODING_AGENT=auto` by default. ## Use it In the interactive shell: ```text theme={null} fix CI on https://github.com/Tracer-Cloud/opensre/pull/4597 and push fix failing checks on Tracer-Cloud/opensre#4597 the current PR CI is failing, fix and push ``` ## What happens 1. OpenSRE reads PR metadata and failing GitHub Actions checks. 2. It fetches compact log excerpts for failing Actions jobs. 3. It verifies the local checkout's `origin` matches the PR repository. 4. It refuses fork PRs because it only pushes to branches in the same repository. 5. After approval, it checks out the PR head branch and applies the fix. 6. If files changed, it commits only the fix-run changes and pushes the PR branch. If no checks are failing, OpenSRE says so and does not push. # Fix GitHub security and quality findings Source: https://opensre.com/docs/github-security-fix Use OpenSRE to remediate Dependabot, code-scanning, and Code Quality findings, then open a pull request. OpenSRE can turn one GitHub security or quality finding into a local fix and, when approved, a pull request. It supports Dependabot alerts, code-scanning/ CodeQL alerts, and GitHub Code Quality standard findings. This is mutating. OpenSRE asks before editing files, then asks again before committing, pushing, and opening a PR. ## Set it up ```bash theme={null} export CODING_WORKSPACE=/path/to/repo # optional; defaults to cwd export GITHUB_TOKEN=... # repo write access plus security-alert read access ``` OpenSRE has built-in local fixers for some Code Quality findings, such as unused imports and unused local variables. For findings that need broader code reasoning, it automatically uses the first coding agent CLI it finds installed and logged in: Pi, Claude Code, or Codex. No configuration is needed when one of those CLIs already works on your machine. To pin a specific agent instead of auto-detection: ```bash theme={null} export CODING_AGENT=claude-code # pi | claude-code | codex (default: auto) export CODING_MODEL=... # optional model override for that agent ``` ## Required access | Need | GitHub access | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Read Dependabot alerts | `security_events` on classic PATs for private repos, `public_repo` for public-only use, or Dependabot alerts read on fine-grained tokens | | Read code-scanning alerts | Code scanning alerts read | | Read Code Quality findings | `repo` on classic PATs, `public_repo` for public-only use, or Code quality read on fine-grained tokens | | Push a branch and open a PR | Contents write and pull requests write, or equivalent `repo` access | ## Use it In the interactive shell: ```text theme={null} hey fix the security issues fix the security and quality issues in Tracer-Cloud/opensre and raise a PR fix the code quality findings on https://github.com/Tracer-Cloud/opensre/security/quality fix the security issues in Tracer-Cloud/opensre and raise a PR ``` For a broad repo request, OpenSRE selects one open supported finding by severity. If you do not name a repo, it uses the current checkout's GitHub `origin`. For a specific alert, include the alert URL or say the alert type and number: ```text theme={null} fix Dependabot alert 12 in this repo fix the code scanning errors on https://github.com/acme/app/security/code-scanning and raise a PR fix https://github.com/acme/app/security/code-scanning/7 and open a PR fix Code Quality finding 42 in Tracer-Cloud/opensre and raise a PR ``` OpenSRE asks before editing files, then asks again before committing, pushing, and opening the PR. ## What happens 1. OpenSRE reads the GitHub alert or Code Quality finding details. 2. It verifies the local checkout's `origin` matches the finding repository. 3. OpenSRE first tries a built-in local fixer when one safely applies. 4. If no built-in fixer applies, OpenSRE runs the auto-detected (or pinned) coding agent CLI to implement the fix in the local checkout. 5. If PR shipping is requested and approved, OpenSRE commits only the files the fix run changed, pushes an `opensre/github-security-fix-*` branch, and opens a PR into the default branch. Secret-scanning alerts are not auto-fixed. Revoke or rotate the secret first, then plan repository cleanup separately. # GitHub workflow tools Source: https://opensre.com/docs/github-workflow-tools Use GitHub-backed tools for read snapshots, status reports, follow-up summaries, and approved issue mutations OpenSRE provides GitHub workflow tools for engineering coordination. The public workflow is: 1. Read a GitHub snapshot. 2. Generate a report or community follow-up summary. 3. Render a mutation proposal for explicit Slack-sourced task requests. 4. Execute the proposal only through the runtime approval path. The workflow tools deliberately separate read-only snapshots from mutating execution. Headless or non-approved runs can produce proposals, but they cannot execute workflow mutations. For ad-hoc chat mutations outside the Slack proposal flow (create issue, assign, merge, `gh api`, etc.), use **`github_cli`** instead — see [GitHub](github). ## Setup Workflow tools use the same GitHub MCP credentials as the [GitHub integration](/docs/github). ```bash theme={null} opensre integrations setup github opensre integrations verify github ``` | Requirement | Details | | ----------------- | ---------------------------------------------------------------------------- | | **Token** | `GITHUB_MCP_AUTH_TOKEN`, or browser sign-in via `opensre integrations setup` | | **Scopes** | `repo` minimum; add security alert scopes for `list_github_security_alerts` | | **Alternate env** | `GITHUB_TOKEN` / `GH_TOKEN` accepted by some workflow tools | ### Example REPL turn ```text theme={null} > generate a Slack-ready morning check-in for Tracer-Cloud/opensre · gathering via GitHub · list work items… ``` See [GitHub workflow tools](#tools) below for the full tool table and mutation approval flow. ## Tools | Tool | Role | Side effects | | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | `list_github_work_items` | Reads issues and classifies work as `taken`, `up_for_grabs`, or `unassigned`. | None | | `summarize_github_pr_status` | Reads PR detail endpoints, authoritative mergeability, check runs, and blocking reasons. | None | | `list_github_security_alerts` | Reads Dependabot, secret-scanning, and code-scanning alerts when token scope allows. | None | | `generate_work_status_report` | Produces a Slack-ready status report from work items and PR status. | None | | `summarize_community_followups` | Reads repository issue comments, then summarizes unanswered questions, agenda items, and suggested replies. | None | | `propose_github_issue_mutation_from_slack` | Builds a deterministic proposal for creating, updating, or closing a GitHub issue from an explicit Slack request. | None | | `execute_github_issue_mutation` | Executes an approved proposal. | Mutating; requires runtime approval | ## Workflow ### 1. Read snapshot Use read tools first: * `list_github_work_items` * `summarize_github_pr_status` * `list_github_security_alerts` when security status is relevant `summarize_github_pr_status` does not trust list-endpoint mergeability. It fetches each PR detail endpoint so `mergeable` and `mergeable_state` are authoritative. Unknown mergeability is reported as `unknown`, not as ready to merge. ### 2. Report or summarize Use `generate_work_status_report` for morning check-ins, Slack updates, blockers, owners, and next actions. If the report tool performs its own GitHub reads and a required read fails, it returns: * `available: false` * `incomplete: true` * an `errors` list It must not produce a false "no blockers" report from partial data. Use `summarize_community_followups` for contributor questions, community meeting agenda items, and suggested replies. It reads repository issue comments directly with pagination instead of performing one request per issue. ### 3. Propose mutation Only when the user explicitly asks to turn a Slack request into a GitHub task, call `propose_github_issue_mutation_from_slack`. Good proposal triggers: * "Add this Slack request to the project task list." * "Create a GitHub task from this thread." * "Update issue 42 with this Slack context." * "Close task 51; PR #2973 shipped." The proposal includes: * operation: `create`, `update`, or `close` * target issue, when applicable * rendered GitHub payload * source Slack link * deterministic `proposal_id` * idempotency marker for retry detection ### 4. Execute mutations `execute_github_issue_mutation` is the only **workflow** mutating tool. It has no `confirm` argument. Ad-hoc GitHub writes in chat go through `github_cli`. Mutation behavior: * **create**: searches for the idempotency marker first; creates only if no existing issue is found. * **update**: fetches the issue, adds the Slack follow-up as a comment unless the proposal marker is already present, and patches title/labels/assignees only when those fields are explicitly present. It never replaces the issue body. * **close**: fetches the issue, adds a closing comment unless the proposal marker is already present, then patches `state=closed` and `state_reason=completed`. It never replaces the issue body. Do not expose `execute_github_issue_mutation` on investigation surfaces. Investigation and headless runs may render proposals, but should not execute mutations directly. ## Required GitHub access Set a GitHub token through the configured GitHub integration, `GITHUB_TOKEN`, or `GH_TOKEN`. Security alert endpoints require token scopes that GitHub enforces separately. If the token cannot read one alert class, the tool returns an error for that alert type while still returning any alert classes it can read. ## Example prompts * "Which PRs are mergeable, blocked, or unknown in `Tracer-Cloud/opensre`?" * "What issues are taken vs up for grabs?" * "Generate a Slack-ready morning check-in from current GitHub work." * "List unanswered community questions from recent issue comments." * "Propose a GitHub issue from this Slack request and keep the source link." * "Execute this approved GitHub issue proposal." # GitLab Source: https://opensre.com/docs/gitlab OpenSRE's GitLab integration gives the investigation agent read access to your merge requests, commits, pipelines, and files for root cause analysis. Optionally, it can write investigation findings back as a note on the relevant MR. *** ## Step 1: Create a GitLab Access Token OpenSRE supports both **Personal Access Tokens** and **Project Access Tokens**. A Project Access Token is recommended for production — it is scoped to a single project and does not depend on a user account. ### Personal Access Token (quickest for local use) 1. In GitLab, go to your avatar → **Edit profile** → **Access Tokens**. 2. Click **Add new token**. 3. Give it a name (e.g. `opensre`) and set an expiry date. 4. Select the following scopes: * `read_api` — required for reading MRs, commits, pipelines, and files * `api` — required only if you enable MR write-back (posting findings as MR notes) 5. Click **Create personal access token** and copy the value immediately — it is shown only once. ### Project Access Token (recommended for server deployments) 1. Open your GitLab project → **Settings** → **Access Tokens**. 2. Click **Add new token**. 3. Give it a name, set a role of **Reporter** (or **Developer** if write-back is needed), and select the same scopes as above. 4. Click **Create project access token** and copy the value. If your GitLab instance is self-hosted, make sure your OpenSRE server can reach it over the network before proceeding. *** ## Step 2: Configure the Integration in OpenSRE Run the setup wizard and select **GitLab**: ```bash theme={null} opensre onboard ``` When prompted, enter: * **GitLab base URL** — leave as `https://gitlab.com/api/v4` for GitLab.com, or change to your self-hosted instance URL (e.g. `https://gitlab.example.com/api/v4`) * **GitLab access token** — from Step 1 The wizard validates your token by calling the GitLab API and writes the following to your `.env` file: | Variable | Description | | --------------------- | ------------------------------------------------------ | | `GITLAB_ACCESS_TOKEN` | Token used for all GitLab API calls | | `GITLAB_BASE_URL` | API base URL (defaults to `https://gitlab.com/api/v4`) | GitLab tools use this configured integration for the base URL and token. Do not pass `gitlab_url` or `gitlab_token` as tool arguments; those values are resolved from the integration configuration. A self-hosted instance host such as `https://gitlab.example.com` is normalized to `https://gitlab.example.com/api/v4`. *** ## What the Agent Can Do Once connected, OpenSRE automatically uses GitLab as an investigation source when an alert contains a GitLab MR reference. The agent can: | Capability | What it reads | | -------------- | ------------------------------------------------------------- | | Merge requests | MR title, description, author, reviewers, labels, diff stats | | Commits | Commit messages, authors, and changed files in a branch or MR | | Pipelines | Pipeline status, failed jobs, and job logs | | Files | File contents at a specific ref for diff analysis | ### Use GitLab tools in the interactive shell After connecting GitLab, include a GitLab project or file URL in your request. OpenSRE infers the project scope from the current message and keeps it for follow-up questions in the same session. ```text theme={null} Read https://gitlab.com/group/project/-/blob/main/runbooks/api.md and summarize the recovery steps. ``` Project URLs enable commits, merge request, and pipeline tools: ```text theme={null} Check recent changes and failed pipelines for https://gitlab.com/group/project. ``` OpenSRE also checks recent conversation, `GITLAB_PROJECT_ID`, `GITLAB_REF`, `GITLAB_FILE_PATH`, and the current repository's GitLab origin remote when the current message does not contain a URL. *** ## MR Write-back (optional) OpenSRE can post a formatted investigation summary directly as a note on the GitLab MR that triggered the alert. This is opt-in and disabled by default. To enable it, set the following environment variable: ```bash theme={null} GITLAB_MR_WRITEBACK=true ``` When enabled, after each investigation OpenSRE posts a collapsible `### RCA Finding` note on the MR. The note is truncated to 4000 characters if needed. **Requirements for write-back:** * Your access token must have the `api` scope (not just `read_api`) * The alert payload must include a GitLab MR IID and project ID so OpenSRE can identify the target MR *** ## Troubleshooting **Validation fails with 401** Your token is invalid or has expired. Regenerate it in GitLab and re-run `opensre onboard`. **Validation fails with 403** Your token does not have sufficient scopes. Ensure `read_api` is selected (and `api` if using write-back). **Self-hosted instance not reachable** Verify that `GITLAB_BASE_URL` ends in `/api/v4` and that your OpenSRE server can reach the GitLab host. Test with: ```bash theme={null} curl -H "Authorization: Bearer " https://your-gitlab-host/api/v4/user ``` **MR write-back not posting** Check that `GITLAB_MR_WRITEBACK=true` is set in your environment and that the alert payload contains `gitlab.merge_request_iid` and `gitlab.project_id` fields. Review OpenSRE server logs for `[publish] GitLab MR` entries. # Google Docs Source: https://opensre.com/docs/google-docs Connect Google Docs so OpenSRE can write investigation reports to your Google Drive OpenSRE can write investigation findings directly to Google Drive as formatted Google Docs — creating a persistent, shareable record of each incident investigation linked to your team's existing Drive folder. ## Prerequisites * Google Cloud project with the Google Drive API enabled * Service account with access to a shared Drive folder ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Google Docs** when prompted and provide your credentials file path and folder ID. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} GOOGLE_CREDENTIALS_FILE=/path/to/service-account.json GOOGLE_DRIVE_FOLDER_ID=your-google-drive-folder-id ``` | Variable | Default | Description | | ------------------------- | ------- | -------------------------------------------------------------- | | `GOOGLE_CREDENTIALS_FILE` | — | **Required.** Path to the service account JSON file | | `GOOGLE_DRIVE_FOLDER_ID` | — | **Required.** Google Drive folder ID for investigation reports | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "google-docs-prod", "service": "google_docs", "status": "active", "credentials": { "credentials_file": "/path/to/service-account.json", "folder_id": "1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgVE2upms" } } ] } ``` ## Creating a service account 1. In Google Cloud Console, go to **IAM & Admin** → **Service Accounts** 2. Click **Create Service Account** 3. Give it a name (e.g., `opensre-docs`) and click **Create** 4. Skip role assignment and click **Done** 5. Click on the service account → **Keys** → **Add Key** → **Create new key** (JSON) 6. Download the JSON file ## Granting Drive folder access 1. In Google Drive, open the folder where reports should be saved 2. Click **Share** 3. Add the service account email (e.g., `opensre-docs@your-project.iam.gserviceaccount.com`) 4. Set the permission to **Editor** ## Finding the folder ID The folder ID appears in the Drive URL: `https://drive.google.com/drive/folders/` ## Verify ```bash theme={null} opensre integrations verify google_docs ``` Expected output: ``` Service: google_docs Status: passed Detail: Connected to Drive folder 1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgVE2upms (3 items in folder) ``` ## Troubleshooting | Symptom | Fix | | ------------------------------ | ---------------------------------------------------------------------------- | | **Credentials file not found** | Check the path in `GOOGLE_CREDENTIALS_FILE` — use an absolute path | | **403 Forbidden** | The service account hasn't been added to the Drive folder with Editor access | | **Drive API not enabled** | Enable the **Google Drive API** in your Google Cloud project | | **Folder not found** | Confirm the folder ID and that the service account has access | ## Security best practices * Keep the service account JSON file outside of your repository — add it to `.gitignore`. * Grant the service account access **only** to the specific Drive folder it needs. * Rotate the service account key periodically via Google Cloud Console. # Grafana Source: https://opensre.com/docs/grafana Connect Grafana so OpenSRE can query metrics, dashboards, and alerts during investigations OpenSRE queries Grafana (Cloud or self-hosted) for metrics, dashboard context, and alert annotations during investigations. ## Overview 1. [Grafana Cloud / self-hosted setup](#grafana-cloud--self-hosted-setup) 2. [Local Grafana Setup (Minikube example)](#local-grafana-setup-minikube-example) ## Grafana Cloud / self-hosted setup ### Prerequisites * Grafana instance URL (Cloud stack URL or self-hosted origin) * Service account token with read access — see [Grafana service account tokens](https://grafana.com/docs/grafana/latest/administration/service-accounts/#add-a-token-to-a-service-account-in-grafana) ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup grafana ``` Provide the instance URL and service account token when prompted. ### Option 2: Environment variables ```bash theme={null} GRAFANA_INSTANCE_URL=https://your-stack.grafana.net GRAFANA_READ_TOKEN=glsa_your_service_account_token GRAFANA_VERIFY_SSL=true # optional — set false only for local/lab GRAFANA_CA_BUNDLE=/path/to/internal-ca.pem # optional — internal CA ``` | Variable | Default | Description | | ---------------------- | ------- | ---------------------------------------------------------- | | `GRAFANA_INSTANCE_URL` | — | **Required.** Grafana base URL | | `GRAFANA_READ_TOKEN` | — | **Required.** Service account token | | `GRAFANA_VERIFY_SSL` | `true` | Set `false` to skip TLS verification (lab only) | | `GRAFANA_CA_BUNDLE` | — | PEM file for private CA (recommended for internal Grafana) | For prod/staging pairs, use `GRAFANA_INSTANCES` — see [Multi-instance integrations](/docs/multi-instance-integrations). ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "grafana-prod", "service": "grafana", "status": "active", "credentials": { "endpoint": "https://your-stack.grafana.net", "api_key": "glsa_your_token", "verify_ssl": true } } ] } ``` ### Option 4: Hosted web app (OpenSRE Cloud) 1. In [app.tracer.cloud](https://app.tracer.cloud), go to **Integrations** → **Grafana** 2. Enter a name, instance URL, and service account token 3. Click **Save** Connect Grafana ### Verify ```bash theme={null} opensre integrations verify grafana ``` ### Self-signed or internal CA certificates If your Grafana instance is self-hosted behind a certificate signed by an internal/private CA, `opensre onboard` will prompt: * **Verify SSL certificate?** — answer `No` to skip TLS verification entirely (only recommended for local/lab instances). * **Path to CA bundle for SSL verification** — point this at a PEM file containing your internal CA certificate to keep full TLS verification while trusting your organization's CA. This is the recommended option for a real internal Grafana deployment, since it still validates the certificate chain and hostname. These can also be set directly via `GRAFANA_VERIFY_SSL` (`true`/`false`) and `GRAFANA_CA_BUNDLE` (path to a PEM file) in `.env`. ## Local Grafana Setup (Minikube example) ### Steps 1. Start minikube: ```bash theme={null} minikube start ``` 2. Add prometheus community and podinfo helm repositories: ```bash theme={null} helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add podinfo https://stefanprodan.github.io/podinfo helm repo update ``` 3. Create monitoring namespace and install kube-prometheus stack: ```bash theme={null} kubectl create namespace monitoring helm install kube-stack prometheus-community/kube-prometheus-stack -n monitoring ``` 4. Create podinfo namespace and install podinfo application: ```bash theme={null} kubectl create namespace podinfo helm install podinfo podinfo/podinfo -n podinfo --set serviceMonitor.enabled=true ``` 5. (Optional) Check the status of the pods in the podinfo and monitoring namespaces: ```bash theme={null} kubectl -n podinfo get pods kubectl -n monitoring get pods ``` 6. (Run in separate terminals) Port forward the podinfo service: ```bash theme={null} kubectl -n podinfo port-forward deploy/podinfo 8080:9898 ``` 7. (Run in separate terminals) Port forward the prometheus service: ```bash theme={null} kubectl -n monitoring port-forward svc/kube-stack-kube-prometheus-prometheus 9090:9090 ``` 8. (Run in separate terminals) Port forward the grafana service and expose it on all network interfaces: ```bash theme={null} kubectl -n monitoring port-forward --address 0.0.0.0 svc/kube-stack-grafana 3000:80 ``` 9. Patch the kube-stack-prometheus service to allow it to scrape the podinfo service: ```bash theme={null} kubectl patch prometheus kube-stack-kube-prometheus-prometheus \ -n monitoring \ --type=merge \ -p '{"spec":{"serviceMonitorSelector":{},"serviceMonitorNamespaceSelector":{}}}' ``` ### Credentials for Grafana Run this command to get the Grafana admin password: ```bash theme={null} kubectl get secret kube-stack-grafana --namespace monitoring -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` The username is `admin` and the password is the output of the command above. ### Access Access the: * podinfo microservices application at [http://localhost:8080](http://localhost:8080) * the prometheus dashboard at [http://localhost:9090](http://localhost:9090) * the grafana dashboard at [http://localhost:3000](http://localhost:3000). ### Simulate load on the podinfo application Open a new terminal and run the following command to simulate load on the podinfo application: ```bash theme={null} while true; do curl -s http://localhost:8080/status/500 >/dev/null; sleep 1; done & while true; do curl -s http://localhost:8080/delay/5 >/dev/null; sleep 1; done & ``` ### Grafana Dashboard Run the following queries in the Grafana dashboard to visualize the metrics: * For the number of requests to the podinfo application: ```promql theme={null} sum(rate(http_requests_total{job="podinfo"}[1m])) by (status) ``` * For the average response time of the podinfo application: ```promql theme={null} histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="podinfo"}[1m])) by (le)) ``` Spike in Error Rate in Grafana ### Setting up Prometheus Alerts Run this command to create a Prometheus alert for the podinfo application which triggers when the error rate is high: ```bash theme={null} kubectl apply -f - <<'YAML' apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: podinfo-alerts namespace: monitoring labels: release: kube-stack spec: groups: - name: podinfo.rules rules: - alert: PodinfoHighErrorRate expr: increase(http_requests_total{status="500"}[5m]) > 5 for: 30s labels: severity: critical annotations: summary: "Podinfo error rate is high" YAML ``` The prometheus alert will be fired after 30 seconds of the error rate being high. You can check the alert in the prometheus dashboard at [http://localhost:9090/alerts](http://localhost:9090/alerts). Prometheus Alert Firing ### Integrating with OpenSRE 1. Get the LAN IP address of your machine: ```bash theme={null} hostname -I | awk '{print $1}' #linux ipconfig getifaddr en0 #macOS (Get-NetIPAddress -InterfaceAlias "Wi-Fi" -AddressFamily IPv4).IPAddress #Windows PowerShell # Remember to replace "Wi-Fi" with the name of your network interface if it's different. ``` 2. Create a service account token in Grafana following the official documentation: [ How to add a token to a service account in Grafana](https://grafana.com/docs/grafana/latest/administration/service-accounts/#add-a-token-to-a-service-account-in-grafana) 3. Run opensre integration add grafana and follow the prompts to enter the required information: * Instance URL: Enter the endpoint URL for your Grafana instance (e.g., `http://:3000`) * Service account token: Paste the Service Account token you created in Grafana Successful Grafana Integration with OpenSRE # Grafana Log Sink Source: https://opensre.com/docs/grafana-log-sink # Grafana Log Sink This integration forwards OpenSRE investigation events into Grafana as: * Loki log streams (push API) * Grafana annotations (via /api/annotations) ## What it does Investigation events are converted into a single low-cardinality Loki stream per report and an optional Grafana annotation summarising the investigation. ## Configuration Environment variables (used for Loki push, including Loki-only mode without a Grafana integration): * `GRAFANA_LOKI_PUSH_URL` — full Loki push endpoint (e.g. `https:///loki/api/v1/push`) * `GRAFANA_WRITE_TOKEN` — optional bearer token for pushing to Loki When used via the integrations catalog the Grafana integration credentials (`endpoint`, `api_key`, or `username`/`password`) are used to build a Grafana client for annotation creation. Loki-only setups can set only the env vars above; no Grafana integration record is required. ## Querying in Grafana Explore Example LogQL queries for investigation logs: * `{job="opensre", source="investigation"} | json | severity="critical"` * `{job="opensre"} | json | alert_name=~".*payments.*"` ## Annotations Annotations created by OpenSRE include the tags `opensre` and `investigation` plus a severity tag (e.g. `critical`). Filter annotations in Grafana by tags to discover OpenSRE investigations. ## Permissions * Loki push endpoint must accept pushes from the service (push URL or write token) * Creating annotations requires Grafana `Editor` role or higher # Grafana Annotations Source: https://opensre.com/docs/grafana_annotations Correlate incidents with **deployments and config changes** from any source. OpenSRE reads [Grafana annotations](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/annotate-visualizations/) — the standard, source-agnostic "what changed and when" marker — so the agent can answer *"did a deploy or config change precede this alert?"* even when the change did **not** come from a GitHub push (ArgoCD/Flux syncs, `helm upgrade`, Jenkins/CircleCI jobs, Terraform applies, manual hotfixes). This complements the GitHub deploy timeline, which only sees GitHub-originated deploys. ## Requirements No new setup or credentials — it reuses your existing **Grafana** integration. If Grafana is connected (see [Grafana](/docs/grafana)), the annotations tool is available automatically during investigations. ## Parameters | Parameter | Description | | -------------------- | --------------------------------------------------------------------------- | | `from` | ISO 8601 window start (e.g. `2026-05-30T14:00:00Z`). Overrides the default. | | `to` | ISO 8601 window end. Overrides the default. | | `tags` | Optional list of annotation tags to filter by (e.g. `["deployment"]`). | | `time_range_minutes` | Window size when `from`/`to` are omitted (default `60`, ending now). | | `limit` | Maximum annotations to return (default `100`). | ## Example ```text theme={null} query_grafana_annotations(from="2026-05-30T14:00:00Z", to="2026-05-30T15:00:00Z", tags=["deployment"]) ``` ```json theme={null} { "source": "grafana_annotations", "total": 1, "annotations": [ { "time": "2026-05-30T14:41:09Z", "time_end": null, "text": "deploy checkout-api v2.8.1", "tags": ["deployment", "checkout-api"], "dashboard_uid": null } ] } ``` Each annotation also carries `time_end` (set only for region annotations) and `dashboard_uid` (set only when the annotation is attached to a dashboard); both are `null` otherwise. The agent uses results like this to flag a likely change-induced regression and tie the suspected root cause to a specific deploy. ## Emitting annotations Make your deploys visible by writing a Grafana annotation when you ship. Most CD tools can post to Grafana's annotations API, for example: ```bash theme={null} curl -s -X POST "$GRAFANA_URL/api/annotations" \ -H "Authorization: Bearer $GRAFANA_TOKEN" \ -H "Content-Type: application/json" \ -d '{"time": 1717079669000, "text": "deploy checkout-api v2.8.1", "tags": ["deployment", "checkout-api"]}' ``` Tagging deploy annotations consistently (e.g. `deployment`) lets the agent filter precisely during an investigation. # groundcover Source: https://opensre.com/docs/groundcover Investigate incidents with [groundcover](https://groundcover.com) logs and traces, queried with **gcQL** (groundcover Query Language) over groundcover's public, read-only **MCP** endpoint. This is the initial integration: configuration, verification, and three read-only tools (logs, traces, and the gcQL reference). More signals (metrics, APM, Kubernetes events/entities, monitors, monitor issues) and alert-source routing land in follow-up releases. v1 is read-only — no monitor creation, silencing, or other mutating actions. ## Commands | Command | What it does | | ----------------------------------------- | -------------------------------------------------------------------- | | `opensre integrations setup groundcover` | Store a groundcover service-account token and routing settings | | `opensre integrations verify groundcover` | Connect to the MCP endpoint, list tools, and check workspace routing | ## 1. Create a read-only service-account token In groundcover: **Settings → Access → Service Accounts** → create a service account with a **read-only** policy → create an **API key** and copy it (shown once). OpenSRE never performs write actions. ## 2. Configure Run the interactive setup: ```bash theme={null} opensre integrations setup groundcover ``` …or set environment variables: ```bash theme={null} export GROUNDCOVER_API_KEY="" # alias: GROUNDCOVER_MCP_TOKEN # Optional — defaults shown: export GROUNDCOVER_MCP_URL="https://mcp.groundcover.com/api/mcp" export GROUNDCOVER_TIMEZONE="UTC" # Only for multi-workspace / multi-backend accounts: export GROUNDCOVER_TENANT_UUID="" export GROUNDCOVER_BACKEND_ID="" ``` Single-workspace accounts need only the token. If your account has multiple workspaces or backends, verification tells you exactly which value to set. Multiple instances: ```bash theme={null} export GROUNDCOVER_INSTANCES='[ {"name":"prod","api_key":"...","tenant_uuid":"...","backend_id":"prod"}, {"name":"staging","api_key":"...","tenant_uuid":"...","backend_id":"staging"} ]' ``` `opensre integrations list` shows integrations saved to the local store (via `setup`); environment-variable configuration is still picked up by `verify` and at runtime. ## 3. Verify ```bash theme={null} opensre integrations verify groundcover ``` Verification connects to the MCP endpoint, confirms the expected read-only tool surface is present, and lists your workspaces. It returns `passed`, `missing` (token not configured), or `failed` with an actionable message — for example, naming the missing or mistyped `GROUNDCOVER_TENANT_UUID` / `GROUNDCOVER_BACKEND_ID` when the account is ambiguous. Tokens are never printed. ## Tools | Tool | Use it for | | --------------------------------- | ------------------------------------------------------------ | | `get_groundcover_query_reference` | The gcQL syntax reference — call once before writing queries | | `query_groundcover_logs` | Application errors, exceptions, and log events | | `query_groundcover_traces` | Slow/failing spans and request correlations | ## Writing efficient gcQL gcQL is a pipe-based language: a query starts with a filter (or `*`) and pipes through operators. These rules keep queries fast and valid: * **Lead with the filter directly** — `level:error | …`, not `* | filter level:error`. The `| filter` pipe is for post-aggregation conditions on computed aliases. * **Project or aggregate — don't pull raw rows blindly.** Use `| fields a, b, …` to select columns, or `| stats …` to aggregate. A bare select-all (` | limit N`) can be rejected by the backend. * **Keep the time window narrow.** The default is the last 1 hour; widen only after an empty or inconclusive result. * **Always include `| limit N`** — it caps rows returned (not data scanned), so for wide ranges prefer `stats`/aggregations. * **Discover fields** with `* | field_names` (or `get_groundcover_query_reference`). Examples: ```text theme={null} # Recent error logs for a workload (projected) level:error workload:checkout | fields _time, instance, content | limit 50 # Error count per workload in one query level:error | stats by (env, cluster, namespace, workload) count() as errors | sort by (errors desc) | limit 20 # Slowest spans for a service (projected) duration_seconds>0.5 workload:checkout | fields _time, span_name, duration_seconds, status_code | limit 50 # 5xx rate per workload (HTTP spans use status_code; status:error is universal) status_code>=500 | stats by (workload) count() as errors | sort by (errors desc) | limit 20 ``` ## Troubleshooting | Symptom | Fix | | | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------- | | `missing` on verify | Set `GROUNDCOVER_API_KEY` (or `GROUNDCOVER_MCP_TOKEN`). | | | | `401 Unauthorized` | Regenerate the service-account token; ensure it has read access. | | | | `Account has N workspaces…` | Set `GROUNDCOVER_TENANT_UUID` to the listed tenant. | | | | `…has N backends…` | Set `GROUNDCOVER_BACKEND_ID` to the listed backend. | | | | `failed to query …` | Project with \` | fields …`or aggregate with` | stats …\`; narrow the time window; avoid a bare raw select-all. | | Empty results | Run \`\* | field\_names\` to find real field names; avoid leading-wildcard globs. | | | Not shown in `integrations list` | `list` shows the local store — run `opensre integrations setup groundcover` (env vars still work for `verify`/runtime). | | | # Helm (CLI) Source: https://opensre.com/docs/helm Connect Helm 3 so OpenSRE can list releases, inspect status and history, and read rendered values and manifests during Kubernetes investigations OpenSRE runs the **Helm 3** command-line client on the machine where the agent executes. It uses **read-only** subcommands (`helm list`, `helm status`, `helm history`, `helm get values`, `helm get manifest`) with explicit `--kube-context` and `--kubeconfig` flags so investigations target the same cluster your engineers use. **Helm 2 is not supported.** Verification checks `helm version` and requires a Helm **3.x** client. ## Prerequisites * **Helm 3** installed and on `PATH` (or configured via `helm_path`) * **`kubectl` access** to the cluster (kubeconfig on disk or in the default search path) * Optional: alert annotations or Kubernetes labels that identify a Helm release and namespace (see [Usage in investigations](#usage-in-investigations)) ## Setup Configure Helm like other local integrations: run `opensre integrations setup helm`, use environment variables, and/or `~/.opensre/integrations.json`. ### Option 1: Environment variables Enable the integration and point it at your cluster: ```bash theme={null} # Required gate — set to 1, true, or yes OSRE_HELM_INTEGRATION=1 # Optional overrides (defaults shown where applicable) HELM_PATH=helm HELM_KUBE_CONTEXT= HELM_KUBECONFIG= # Default namespace hint when the alert does not specify one HELM_NAMESPACE= ``` | Variable | Default | Description | | ----------------------- | ------- | ----------------------------------------------------------------------------------------------------------------- | | `OSRE_HELM_INTEGRATION` | — | **Required** to activate Helm from env. Must be `1`, `true`, or `yes` (case-insensitive). | | `HELM_PATH` | `helm` | Helm binary name or absolute path. | | `HELM_KUBE_CONTEXT` | — | Passed to every Helm invocation as `--kube-context`. | | `HELM_KUBECONFIG` | — | Passed as `--kubeconfig` (path to the kubeconfig file). | | `HELM_NAMESPACE` | — | `default_namespace` in the resolved integration; used as a fallback namespace when the alert does not supply one. | To raise the maximum size of stored manifest text (see [Advanced](#advanced)): ```bash theme={null} # Integer, minimum 1024; default in code is 600_000 characters HELM_MANIFEST_MAX_CHARS=600000 ``` ### Option 2: Persistent store Add an active `helm` record to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "helm-prod", "service": "helm", "status": "active", "credentials": { "helm_path": "helm", "kube_context": "prod-admin", "kubeconfig": "", "default_namespace": "production" } } ] } ``` **Credential field aliases** (store / API compatibility): `context` for `kube_context`, `kubeconfig_path` or `kube_config` for `kubeconfig`, and `namespace` for `default_namespace`. ## Verify ```bash theme={null} opensre integrations verify helm ``` A passing check runs `helm version` (must report Helm 3) and a minimal `helm list -A --max 1 -o json` against your cluster to validate JSON output and reachability. ## Usage in investigations When Helm is configured, OpenSRE adds a `helm` entry to `detect_sources` only if the alert shows **Helm-specific** context — avoiding generic “deployment” noise. **Annotations** (including top-level enriched fields merged into annotations): | Key | Purpose | | ------------------------------------ | ------------------------ | | `helm_release` / `helm_release_name` | Release name | | `helm_namespace` | Namespace hint | | `helm_chart`, `helm_revision` | Extra Helm signals | | Keys prefixed with `meta.helm.sh/` | Treated as Helm metadata | **Labels** (from Prometheus/Alertmanager-style `labels` / `commonLabels` on the alert payload): | Label | Purpose | | -------------------------------- | --------------------------------- | | `meta.helm.sh/release-name` | Release name | | `meta.helm.sh/release-namespace` | Namespace paired with the release | **Alert text:** Strong phrases such as `helm upgrade`, `helm install`, `helm rollback`, `helm chart`, or `helm release` in `summary` / `description` / `message` (or top-level `alert_name` / `error_message`) can also enable the Helm source when other hints are absent. **Top-level payload fields** (dict alerts only): `helm_release`, `helm_release_name`, `helm_namespace`, etc., are consulted as fallbacks. Example minimal alert: ```json theme={null} { "labels": { "meta.helm.sh/release-name": "my-api", "meta.helm.sh/release-namespace": "prod" }, "annotations": { "summary": "Errors after helm upgrade" } } ``` Then run: ```bash theme={null} opensre investigate -i alert.json ``` ## Investigation tools | Tool | What it does | | --------------------------- | ------------------------------------------------------------------------------------------------------ | | `helm_list_releases` | Lists releases (JSON from `helm list`); uses all namespaces when no namespace filter is set. | | `helm_release_status` | `helm status -o json` for one release. | | `helm_release_history` | `helm history -o json` for one release. | | `helm_get_release_values` | `helm get values -o json` (user-supplied values; JSON `null` from Helm is treated as an empty object). | | `helm_get_release_manifest` | Rendered manifest YAML (may be truncated; see below). | ## Evidence keys Post-processing writes **distinct** evidence keys so parallel tools do not overwrite each other: | Evidence key | Content | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `helm_releases` | Parsed release list | | `helm_release_status` | Status object from `helm status` | | `helm_release_history` | Revision history array | | `helm_release_values` | Values object | | `helm_release_manifest` | Rendered manifest YAML **string**; unchanged key name. When the manifest exceeds the size cap, this value is **truncated** text, not a separate key. | | `helm_manifest_truncated` | Boolean: `true` when `helm_release_manifest` was truncated because of `HELM_MANIFEST_MAX_CHARS` / the client default cap. | ## Advanced * **Manifest size:** Very large charts can produce multi-megabyte manifests. The client truncates manifest text by default; override with `HELM_MANIFEST_MAX_CHARS` (see Option 1). * **Local kind demo:** From a repo checkout with Docker, kind, kubectl, and Helm installed, run `./tests/e2e/kubernetes/helm/scripts/demo-helm-kind.sh` to create a sample cluster and release (see script comments for teardown). ## Security and operations * The integration is **read-only**: it does not `install`, `upgrade`, or `uninstall` releases. * `helm get values` output can include **secrets**; treat evidence like any other sensitive kubectl/Helm output. * Prefer a dedicated kubeconfig or context with **least privilege** if your policy requires it. ## Troubleshooting | Symptom | What to check | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Verify: missing** | Helm not in store / env gate `OSRE_HELM_INTEGRATION` not set / invalid JSON store entry. | | **Verify: Helm 3 required** | Upgrade to Helm 3 or point `helm_path` at a v3 binary. | | **Verify: list JSON / cluster** | kubeconfig and context; cluster reachable with `helm list -A` manually. | | **No Helm tools in plan** | Alert must include Helm annotations, `meta.helm.sh/*` labels, or strong Helm phrases (see above). | | **Wrong namespace / release not found** | Ensure `meta.helm.sh/release-namespace` or `helm_namespace` / `k8s_namespace` annotations match the release; set `default_namespace` in config when alerts lack namespace. | # Hermes log monitoring Source: https://opensre.com/docs/hermes Tail Hermes errors.log, classify incidents, alert on Telegram or Rocket.Chat; run offline classifier regression tests, then try a local watch demo. OpenSRE can **watch the log file written by a [Hermes](https://github.com/NousResearch/hermes-agent) deployment** (typically `errors.log`), turn new lines into structured incidents, optionally **deduplicate and escalate** them, and deliver alerts to **Telegram or Rocket.Chat**. This page is about that **log tail + classifier + delivery** path — separate from OpenSRE’s own runtime logs. Developer workflow: the **Hermes synthetic suite** under `tests/synthetic/hermes/` runs **offline** classifier checks (no LLM, no live infra); you can run that first, then **watch** a bundled scenario `errors.log` with `opensre hermes watch` as a small local demo. **Optional:** enable **`--investigate`** to attach the full OpenSRE RCA pipeline for **HIGH** / **CRITICAL** incidents ([quickstart](/docs/quickstart)). Delivery uses the same credential model as the rest of OpenSRE, for either provider. Telegram: configure once with `opensre integrations setup telegram` (or `opensre onboard`) — `hermes watch` resolves the token from the integration store, environment, or keyring; set `TELEGRAM_BOT_TOKEN` / `TELEGRAM_DEFAULT_CHAT_ID` directly, or pass `--chat-id`. Rocket.Chat: pass `--provider rocketchat` and configure via `opensre integrations setup rocketchat` or `ROCKETCHAT_*` env vars. See [Telegram](/docs/messaging/telegram) and [Rocket.Chat](/docs/messaging/rocketchat). *** ## Local demo: synthetic tests, then watch **1 — Offline “investigation” of the classifier (pytest, no Telegram)**\ The Hermes log-classifier synthetic suite feeds real-shaped `errors.log` slices through `IncidentClassifier` and asserts against `answer.yml`. This is the same regression loop documented in the repo’s [`tests/synthetic/hermes/README.md`](https://github.com/Tracer-Cloud/opensre/blob/main/tests/synthetic/hermes/README.md): ```bash theme={null} uv run pytest tests/synthetic/hermes -q ``` For incident-identification RCA fixtures (session topology, runtime hangs, cron delivery, KV cache drift), run the parallel suite: ```bash theme={null} uv run pytest tests/synthetic/hermes_rca -q ``` To run every synthetic-marked test under `tests/synthetic/` (includes Hermes and other packages): ```bash theme={null} make test-synthetic ``` **2 — Live watch on a fixture file (Telegram required)**\ Use a scenario’s `errors.log` that emits incidents under **default** classifier settings (see **Quick check with a bundled fixture** under [CLI: live watch](#cli-live-watch) — the `002-gateway-systemd-crash-loop` example matches the screenshot there). From the repo root: ```bash theme={null} export TELEGRAM_BOT_TOKEN=… export TELEGRAM_DEFAULT_CHAT_ID=… uv run opensre hermes watch \ --log-path tests/synthetic/hermes/002-gateway-systemd-crash-loop/errors.log \ --from-start ``` `--from-start` **replays** the file then keeps tailing — good for a one-shot demo of classification → Telegram. By default, watch **only reads new lines**, so without `--from-start` a static fixture produces no events until more lines are appended (use a **writable** copy of the scenario log if you want to experiment without editing files under `tests/`). Omit **`--investigate`** for a lightweight demo (**`--investigate`** runs the investigation pipeline with an LLM for **HIGH** / **CRITICAL** only; see below). *** ## What gets monitored | Item | Default | | -------- | -------------------------------------------------------------------- | | Log file | `~/.hermes/logs/errors.log` | | Override | Set `HERMES_LOG_PATH` to an absolute path if Hermes writes elsewhere | The watcher uses a rotation-safe tailer: if the file is missing at startup, it waits until it appears. Use `--from-start` if you intentionally want to replay existing file contents before live-tailing (by default **only new lines** are considered so Telegram is not flooded on restart). *** ## CLI: live watch Run from a machine that can read the Hermes log file and reach Telegram: ```bash theme={null} uv run opensre hermes watch ``` Stop with **Ctrl+C** or **SIGTERM** (e.g. under systemd). ### Quick check with a bundled fixture From the repo root you can point `--log-path` at a synthetic `errors.log`. Prefer a scenario with **ERROR** / **CRITICAL** lines so incidents fire under the **default** classifier (the live watcher does **not** read per-scenario `scenario.yml` thresholds): ```bash theme={null} uv run opensre hermes watch \ --log-path tests/synthetic/hermes/002-gateway-systemd-crash-loop/errors.log \ --from-start ``` Example **Telegram** output when replaying that fixture (critical exit, traceback, crash loop, and systemd error): Hermes incidents delivered to Telegram after watching the 002-gateway-systemd-crash-loop synthetic log **Why you might see no Telegram:** `warning_burst` needs **five** warnings from the same logger within **60 seconds** (`IncidentClassifier` defaults). Several synthetic scenarios set a lower threshold in `scenario.yml` for **pytest only**; the CLI watcher ignores that file. For example `000-telegram-polling-conflict` expects bursts of **three** warnings — the suite passes, but **`hermes watch` on that file emits no `warning_burst` incidents**, so nothing is sent. Use a log with hard errors (like `002-*`) or your real `errors.log` once Hermes is writing enough warnings. Correlator **dedup** can also reduce multiple deliveries (either provider) for the same fingerprint; check shutdown line `hermes-watch: correlator metrics delivered=…` — if `delivered=0`, the classifier did not emit any routable incidents for that run. ### Common options | Flag | Purpose | | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `--log-path PATH` | Hermes log file (defaults to `HERMES_LOG_PATH` expansion or `~/.hermes/logs/errors.log`) | | `--provider telegram\|rocketchat` | Delivery provider (default `telegram`) | | `--chat-id ID` | Overrides the provider's default destination (`TELEGRAM_DEFAULT_CHAT_ID` or `ROCKETCHAT_DEFAULT_CHANNEL`) for this run | | `--cooldown-seconds N` | Per-fingerprint cooldown before the same incident is sent again (default `300`) | | `--from-start` | Replay the file from the beginning, then tail | | `--investigate` / `--no-investigate` | Run an OpenSRE RCA for `HIGH` / `CRITICAL` incidents and append the summary to the delivered message | | `--correlate` / `--no-correlate` | Route through the correlator (dedup, escalation, routing). Default: **on** | | `--dedup-window-seconds` | Correlator dedup window when `--correlate` is on | | `--escalation-threshold` / `--escalation-window-seconds` | Repeat-hit escalation knobs when `--correlate` is on | ### Environment variables | Variable | Role | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `TELEGRAM_BOT_TOKEN` | Bot token, unless already configured via `opensre integrations setup telegram` / `onboard` (see [Telegram](/docs/messaging/telegram)) | | `TELEGRAM_DEFAULT_CHAT_ID` | Default Telegram destination when `--chat-id` is omitted | | `ROCKETCHAT_*` | Rocket.Chat token or webhook credentials, used when `--provider rocketchat` (see [Rocket.Chat](/docs/messaging/rocketchat)) | | `ROCKETCHAT_DEFAULT_CHANNEL` | Default Rocket.Chat destination when `--chat-id` is omitted | | `HERMES_LOG_PATH` | Default log path when `--log-path` is omitted | | `OPENSRE_HERMES_INVESTIGATE` | If set to `1` / `true` / `yes` / `on`, enables investigation when the CLI does not pass `--investigate` / `--no-investigate` | *** ## Watch with full OpenSRE RCA (`--investigate`) For production on-call, you normally run **one long-lived watcher** with the RCA bridge when you want LLM-backed summaries on serious incidents: ```bash theme={null} uv run opensre hermes watch --investigate ``` Configure **LLM + integrations** as for any OpenSRE investigation ([quickstart](/docs/quickstart)). When the classifier emits **HIGH** or **CRITICAL** incidents, each qualifying incident triggers **`run_investigation`** via the sink, regardless of `--provider`. Work runs on a **bounded thread pool** with a **timeout** so log tailing stays responsive; **MEDIUM** and lower stay on a lighter notification path by default. For **ad hoc** Hermes log context during **another** investigation (for example Grafana-driven RCA), use **`get_hermes_logs`** below — it samples the log and does not replace a long-running **`opensre hermes watch`** for continuous classification. *** ## Agent tool: `get_hermes_logs` During an OpenSRE investigation, the planner can call **`get_hermes_logs`** to read the same Hermes log in two modes: * **`op="scan"`** — one-shot window of the last *N* lines; returns parsed records and incidents the classifier would emit on that window. * **`op="tail"`** — incremental, cursor-based reads for “what’s new since last poll?”. The tool only allows paths under permitted directories (by default `~/.hermes` and the parent of `HERMES_LOG_PATH` when set) so arbitrary file reads are blocked. **Classifier state is not persisted between `tail` calls.** Each tool invocation uses a fresh classifier, so burst windows and traceback continuation state reset between calls. For accurate ongoing detection, use **`opensre hermes watch`** (long-lived classifier) rather than many separate `tail` calls. *** ## Synthetic regression suite Contributors add scenarios under `tests/synthetic/hermes/` (fixtures, `scenario.yml`, `answer.yml`). Layout, schema, and “adding a new scenario” steps are in [`tests/synthetic/hermes/README.md`](https://github.com/Tracer-Cloud/opensre/blob/main/tests/synthetic/hermes/README.md). *** ## Further reading * Implementation lives under `integrations/hermes/` in the OpenSRE repository (tailer, parser, classifier, correlator, sinks, CLI wiring). * Surface-attribution evaluation workflow for contributors: [`docs/hermes_runbook.mdx`](https://github.com/Tracer-Cloud/opensre/blob/main/docs/hermes_runbook.mdx) in the repo (not on the public docs nav). * If you also run [OpenClaw](/docs/openclaw) alongside Hermes, that page covers the OpenClaw bridge (context lookup and RCA write-back). * For false-positive risks when rules match both `message` and raw line text, see [issue #1874](https://github.com/Tracer-Cloud/opensre/issues/1874). # Hermes runbook Source: https://opensre.com/docs/hermes_runbook # Hermes Surface Attribution Runbook ## Purpose Hermes deployments often contain multiple subsystem families operating together: * LLM providers * messaging adapters * orchestration engines * runtime backends * memory systems * control and governance layers When an incident occurs, investigators must first determine **which subsystem family owns the failure** before deeper root-cause analysis can begin. The Surface Attribution evaluation track exists to validate that behavior. *** ## Attribution Workflow Hermes investigations should follow a consistent attribution process: ### Step 1: Identify the failing surface Determine which subsystem family is most likely responsible for the observed failure. Examples: | Symptom | Likely Surface | | ------------------------------ | -------------- | | Provider API failures | Provider | | Session routing failures | Runtime | | Workflow execution failures | Orchestration | | Context retrieval failures | Memory | | Approval / audit failures | Control | | Adapter communication failures | Messaging | *** ### Step 2: Compare against historical analogs Once a surface family is identified, compare the incident against previously validated Hermes RCA scenarios. The analog registry contains curated mappings across all Hermes RCA evaluation tracks. Goals: * reduce attribution drift * improve consistency * encourage evidence-based classification * detect recurring failure patterns *** ### Step 3: Generate a diagnostic follow-up Investigations should not stop at attribution. A valid attribution result should produce a targeted diagnostic question requesting additional evidence. Examples: * Can you provide the adapter response body? * Can you capture the request headers? * Can you inspect the runtime state snapshot? * Can you compare the adapter catalog against the configured routing table? Diagnostic questions should be: * actionable * evidence-seeking * surface-specific *** ## Scenario 050: Surface Sprawl / Unknown Adapter ### Goal Validate attribution behavior when an adapter is not directly recognized. ### Evaluation Criteria An investigation is expected to: 1. Identify the correct subsystem family 2. Select the closest historical analog 3. Produce a useful diagnostic follow-up ### Failure Modes Common attribution failures include: * assigning ownership to the wrong subsystem * selecting an unrelated analog scenario * generating generic follow-up questions * requesting evidence unrelated to the suspected surface *** ## Adapter Tuple Corpus The attribution corpus contains deterministic adapter combinations spanning: * messaging * provider * runtime * orchestration * memory * control The corpus is used to validate attribution consistency across a broad set of Hermes deployment configurations. Current coverage: * 23 attribution tuples *** ## Analog Registry The analog registry provides curated mappings across Hermes RCA Parts 1–4. Each analog contains: * scenario identifier * subsystem family * expected attribution target * diagnostic guidance The registry is intentionally deterministic and offline-runnable. *** ## Benchmarking ### Run offline validation ```bash theme={null} uv run python -m tests.synthetic.hermes_rca.run_suite --offline-only ``` ### Generate benchmark snapshots ```bash theme={null} uv run python -m tests.synthetic.hermes_rca.run_suite --offline-only --write-history ``` ### Generate benchmark reports ```bash theme={null} uv run python -m tests.synthetic.hermes_rca.benchmark_report ``` *** ## Meta Evaluation The surface attribution meta-suite validates attribution behavior across the adapter corpus. Run: ```bash theme={null} uv run pytest tests/e2e/hermes/meta/test_surface_sprawl.py -q ``` Current corpus coverage: * 23 adapter tuples The expected pass threshold is at least 80% of registered tuples. *** ## Design Principles Surface attribution evaluation is designed to be: * deterministic * provider-independent * offline-runnable * CI-friendly * extensible as new Hermes surfaces are added The evaluation framework intentionally separates attribution quality from root-cause quality so that ownership classification can be measured independently from deeper RCA reasoning. # Honeycomb Source: https://opensre.com/docs/honeycomb Connect Honeycomb so OpenSRE can query traces during investigations OpenSRE queries Honeycomb to surface distributed traces during alert investigations — identifying slow spans, high-error services, and query patterns correlated with incidents. ## Prerequisites * Honeycomb account (classic or environments-based) * API key with query access ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Honeycomb** when prompted and provide your API key and dataset. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} HONEYCOMB_API_KEY=your-api-key HONEYCOMB_DATASET=your-dataset # optional, defaults to __all__ HONEYCOMB_API_URL=https://api.honeycomb.io # optional ``` | Variable | Default | Description | | ------------------- | -------------------------- | -------------------------------------------------------- | | `HONEYCOMB_API_KEY` | — | **Required.** Honeycomb API key | | `HONEYCOMB_DATASET` | `__all__` | Dataset to query (classic) or `__all__` for environments | | `HONEYCOMB_API_URL` | `https://api.honeycomb.io` | Override for EU region or proxies | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "honeycomb-prod", "service": "honeycomb", "status": "active", "credentials": { "api_key": "your-api-key", "dataset": "__all__", "base_url": "https://api.honeycomb.io" } } ] } ``` ## Creating an API key 1. In Honeycomb, go to **Account** → **API Keys** 2. Click **Create API Key** 3. Set the environment and enable **Query Data** permission 4. Copy the key Use `__all__` as the dataset to query across all datasets in an environment. For EU accounts, set `HONEYCOMB_API_URL=https://api.eu1.honeycomb.io`. ## Verify ```bash theme={null} opensre integrations verify honeycomb ``` Expected output: ``` Service: honeycomb Status: passed Detail: Connected to https://api.honeycomb.io (environment production) and queried dataset __all__ ``` ## Troubleshooting | Symptom | Fix | | -------------------------- | ------------------------------------------------------------ | | **401 Unauthorized** | Confirm the API key is correct and has Query Data permission | | **Dataset not found** | Use `__all__` or check the exact dataset name in Honeycomb | | **EU account unreachable** | Set `HONEYCOMB_API_URL=https://api.eu1.honeycomb.io` | ## Security best practices * Use a dedicated **read-only** API key scoped to query access. * Store the API key in `.env`, not in source code. # How an investigation works Source: https://opensre.com/docs/how-investigations-work A plain-language walkthrough of what OpenSRE does between receiving an alert and posting a root-cause report ## At a glance | Step | What happens | You run | | ----------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------- | | Start | Alert or plain-language incident description enters the pipeline | `opensre investigate -i ` or `opensre` then describe the incident | | Investigate | Tool-calling loop queries connected integrations | `/investigate` or `/verify ` in the REPL | | Output | RCA artifacts written locally (and optionally to Slack) | Open `problem.md`, `theory/hypothesis_*.md`, `report.md` | | Export | Single JSON for automation | `opensre investigate -i --output ./rca.json` | **Config:** `LLM_PROVIDER` + matching API key; integrations via `opensre onboard` or `opensre integrations setup`. See [Environment variables](/docs/configuration/environment-variables). **Failure modes:** No integrations connected → limited evidence; LLM misconfigured → run fails at start; tool cap / stagnation breaker → investigation stops early with partial report (see stages below). Think of OpenSRE as an on-call engineer who never sleeps. It follows the same instincts a good on-call engineer would: get paged, decide whether it's actually worth waking up for, check the obvious dashboards first, take notes as it goes, stop once it has an answer (or has run out of useful things to check), and write it up clearly for whoever reads it next. ```mermaid theme={null} flowchart LR A[Alert arrives] --> B{Real incident\nor noise?} B -->|noise| Z[No action taken] B -->|real| C[Plan what to check] C --> D[Investigate] D --> E[Diagnose] E --> F[Report] ``` ## The six stages Before looking at anything alert-specific, OpenSRE checks which of your monitoring and infrastructure tools are actually connected — Grafana, Datadog, EKS, and whatever else you've set up. This defines the universe of things it's allowed to check for this investigation. OpenSRE reads the incoming alert and classifies it: a real incident, or noise (a greeting, a "thanks," a reply in an already-closed thread)? Genuine alerts proceed — including informational or "all clear" notifications, since those still carry signal. Chit-chat is ignored and nothing further happens. Rather than poking around at random, OpenSRE picks the handful of tools most likely to explain *this specific* alert — matched by source (a Grafana alert starts with Grafana queries, an EKS alert starts with pod and cluster tools) and ranked by relevance. This keeps the investigation focused instead of firing off every tool it has access to. This is where the real work happens. OpenSRE works through its plan one step at a time: run a check, look at what came back, decide what to check next. It keeps a running memory of everything it's found, so each new step builds on the last instead of starting from scratch. A few habits keep this efficient: * It won't run the exact same check twice — if it already has that answer, it reuses it instead of asking again. * If it finds itself going in circles without learning anything new, it stops and writes up what it already has rather than spinning forever. * There's a hard ceiling on how long a single investigation can run, so one stubborn incident can never run away with unbounded time or cost. Once OpenSRE has enough evidence — or has run out of useful things to check — it turns its findings into a structured diagnosis: what happened, the chain of cause and effect, which claims are backed by evidence versus still unconfirmed, and concrete remediation steps. It also attaches a confidence score, so you know how much to trust the conclusion at a glance. Finally, OpenSRE delivers the result however you've configured it — a Slack message in the incident channel, a GitLab comment, or local files you can read directly. See [Investigations overview](/docs/investigation-overview) for what these look like and how to run one yourself. ## Why it doesn't lose the thread on long investigations A thorny incident can mean touching a dozen tools and collecting a lot of evidence. OpenSRE actively manages what it's holding onto in its working memory so that early findings don't get pushed out by later ones — the same reason a good engineer keeps a running incident doc instead of trying to hold every detail in their head. If a long investigation starts accumulating more than it can usefully reason about at once, OpenSRE trims the least useful parts rather than letting the most important early evidence quietly fall out of view. You don't need to configure any of this — tool selection, note-keeping, and the stop conditions above all happen automatically. This page just explains what's going on behind the spinner. ## Related pages * [Investigations overview](/docs/investigation-overview) — how to start an investigation and where to find the output. * [Interactive Shell Commands](/docs/interactive-shell-commands) — the slash commands available while an investigation runs. * [Closed-loop learning](/docs/closed-loop-learning) — how OpenSRE improves future investigations from past outcomes. # incident.io Source: https://opensre.com/docs/incident_io Connect incident.io so OpenSRE can read incident context, updates, and metadata during investigations OpenSRE can use incident.io as incident context during RCA. The integration can list live incidents, fetch full incident metadata, read incident updates, and append final findings to the incident summary through the supported incident edit API. ## Prerequisites * incident.io account * incident.io API key with read access to incidents and incident updates * Write permission if you want OpenSRE to append findings to incident summaries ## Setup ### Interactive CLI ```bash theme={null} opensre integrations setup incident_io ``` ### Environment variables ```bash theme={null} INCIDENT_IO_API_KEY=your-api-key ``` | Variable | Default | Description | | ---------------------- | ------------------------- | ------------------------------------------------------ | | `INCIDENT_IO_API_KEY` | - | Required incident.io API key | | `INCIDENT_IO_BASE_URL` | `https://api.incident.io` | Optional API URL override for tests or private routing | ## Verify ```bash theme={null} opensre integrations verify incident_io ``` Verification performs a minimal `GET /v2/incidents` request. ## Investigation Behavior When an alert includes an incident.io incident ID or URL, OpenSRE makes the `incident_io_incidents` action available with `action="context"`. That reads: * incident metadata from `GET /v2/incidents/{id}` * incident updates from `GET /v2/incident_updates?incident_id={id}` If no incident ID is present, OpenSRE can use `action="list"` to list live incidents. ## Write-Back OpenSRE does not create native timeline events. incident.io's public v2 API supports editing incidents, so OpenSRE write-back uses: ```text theme={null} POST /v2/incidents/{id}/actions/edit ``` The action appends findings to the incident summary. Use `action="append_summary"` only after the RCA has useful findings or next steps ready to publish. # OpenSRE Source: https://opensre.com/docs/index OpenSRE is agentic alert investigation for production systems. It connects to your observability stack, infrastructure, and knowledge bases to investigate incidents before your team gets paged — then delivers root-cause analysis and recommended fixes to Slack, PagerDuty, or local files.

Tracer to OpenSRE

Coming from Tracer? Welcome. OpenSRE is the open-source home for these docs, while some live URLs, installers, assets, and Slack app labels still use the Tracer name during the transition.

## Get started Install OpenSRE and run your first alert investigation. Wire OpenSRE into observability, incident, code, and data systems. See how OpenSRE gathers evidence and produces root cause analysis. ## First 5 minutes ```bash theme={null} # Install (macOS / Linux) curl -fsSL https://install.opensre.com | bash # Configure LLM + integrations opensre onboard # Run a sample investigation opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` After the run completes, open the generated artifacts: * `problem.md` — incident framing * `theory/hypothesis_*.md` — hypotheses tested during the run * `report.md` — root cause, confidence, and next steps Or export JSON for automation: ```bash theme={null} opensre investigate -i --output ./rca.json ``` **Interactive shell** — run `opensre` with no subcommand to describe incidents in plain language and use slash commands (`/help`, `/verify datadog`, `/investigate`). Stuck? See [Quickstart troubleshooting](/docs/quickstart#troubleshooting) (Docker running, `make` installed, LLM configured). ## Start here Alert ingestion, evidence collection, masking, and remote runtime workflows. Sample outputs, quickstart demos, and CloudOpsBench benchmark walkthrough. ## How it works When an alert fires, OpenSRE autonomously: 1. **Ingests the alert** from metrics, logs, traces, or incident systems 2. **Assembles context** — ownership, deploys, dependencies, baselines 3. **Plans evidence collection** across connected integrations 4. **Investigates in a loop** — queries tools, updates hypotheses, stops when confidence is high enough 5. **Delivers a report** to Slack, local files, or your configured messaging channel
Investigation workflow
How OpenSRE Works
See [How an investigation works](/docs/how-investigations-work) for a plain-language walkthrough of each stage. # Install OpenSRE Source: https://opensre.com/docs/install Choose the right OpenSRE installation path for local or enterprise environments Pick the path that matches where you run OpenSRE. ## Which path? | Path | Best for | Get started | | ----------------------------- | --------------------------------------------------- | --------------------------------------- | | **Local CLI** | Laptop or self-hosted server; full `opensre` binary | [Install locally](/docs/install-local) | | **Enterprise (Tracer Cloud)** | Org-wide connectors in the hosted web app | [Enterprise setup](/docs/install-enterprise) | ## Install locally (most users) ```bash theme={null} # macOS / Linux curl -fsSL https://install.opensre.com | bash # Windows (PowerShell) irm https://install.opensre.com | iex ``` Then configure your LLM and integrations: ```bash theme={null} opensre onboard opensre integrations verify ``` See [Quickstart](/docs/quickstart) for a full walkthrough including a sample investigation. ## Environment-specific guides | Platform | Guide | | -------- | ---------------------------------------------- | | macOS | [macOS](/docs/environments/macos) | | Linux | [Linux (local)](/docs/environments/linux-local) | | Windows | [Windows (local)](/docs/environments/windows-local) | | Docker | [Docker](/docs/environments/docker) | ## Enterprise hosted setup For teams using [app.tracer.cloud](https://app.tracer.cloud), see [Enterprise setup](/docs/install-enterprise). ## Next steps * [Investigations overview](/docs/investigation-overview) * [Integrations overview](/docs/integrations-overview) * [Environment variables](/docs/configuration/environment-variables) # Enterprise setup (Tracer Cloud) Source: https://opensre.com/docs/install-enterprise Set up OpenSRE for your organization in app.tracer.cloud This guide is for teams that manage OpenSRE from the hosted web app at [app.tracer.cloud](https://app.tracer.cloud). ## 1) Create your organization account Sign up at [tracer.cloud](https://tracer.cloud) and continue in [app.tracer.cloud/sign-up](https://app.tracer.cloud/sign-up) to create or join your organization workspace. ## 2) Complete workspace onboarding In the web app, complete onboarding to: * Define your workspace context (team name, default channels) * Connect your first integrations (start with observability + messaging) * Verify access and permissions for each connector ## 3) Configure integrations for your org Enterprise integrations are configured in the web app and shared across your organization. Use the integration catalog for credential requirements: * [Integrations overview](/docs/integrations-overview) — local CLI and hosted setup paths * [Datadog](/docs/datadog) — API key + application key (hosted flow in Option 3) * [Grafana](/docs/grafana) — service account token + endpoint URL * [AWS](/docs/aws) — cross-cloud infrastructure mapping * [Slack](/docs/messaging/slack) — deliver investigation reports to channels Integrations ### Verification checklist After connecting each integration in the web app: 1. Confirm the connector shows **connected** in **Integrations** 2. Run a test investigation from the app or ask an org admin to trigger a sample alert 3. Confirm the report arrives in your configured Slack channel (see [Slack setup](/docs/messaging/slack)) For Slack-specific help, use **Support** in the app or follow the self-serve steps in [Slack integration](/docs/messaging/slack). ## 4) Run investigations with shared context After integrations are connected, OpenSRE investigations use the same organization-level connectors and policies. Engineers can also run local CLI investigations that reference the same integration catalog. ## Local CLI users in enterprise teams If your engineers also run local investigations: ```bash theme={null} opensre # inside the shell: onboard ``` This configures local LLM credentials while org-wide connectors remain in the web app. See [Install locally](/docs/install-local) for the full local path. # Local and on-prem setup Source: https://opensre.com/docs/install-local Run OpenSRE locally with the CLI binary and onboard integrations from your environment This guide is for teams running OpenSRE with the local `opensre` binary on developer machines, virtual machines, or on-prem hosts. ## What mode should I use? Use the mode that matches where your workloads and alerts live: * **Local laptop mode** for fast setup, testing, and debugging * **On-prem VM/server mode** for shared internal environments * **Container mode** for reproducible deployments in Docker Environment-specific install steps live under the `Installation` tab: * [macOS](/docs/environments/macos) * [Linux (local)](/docs/environments/linux-local) * [Windows (local)](/docs/environments/windows-local) * [Docker](/docs/environments/docker) ## Local setup workflow ### 1) Install the OpenSRE CLI ```bash theme={null} brew tap tracer-cloud/tap brew install tracer-cloud/tap/opensre # or curl -fsSL https://install.opensre.com | bash ``` The macOS/Linux installer does not require `sudo`. It uses a writable bin directory already on your `PATH` when possible; otherwise it installs to `~/.local/bin` and prints the command to apply the PATH update in your shell. The installer makes `opensre` available on your PATH without sudo: it installs into (or links the binary from) a user-writable directory that is already on your PATH, so the command works in the current terminal right away. Only when no such directory exists does it fall back to updating your shell profile for new terminals. When you run the installer from an interactive terminal — including the piped `curl … | bash` form above — it launches `opensre onboard` automatically once the install finishes, so you can set up your LLM provider and integrations right away. In non-interactive environments (CI, scripts, provisioning) the auto-launch is skipped and a "Next steps" hint is printed instead. To opt out of the auto-launch entirely: ```bash theme={null} curl -fsSL https://install.opensre.com | OPENSRE_AUTO_LAUNCH=0 bash ``` The curl installer also soft-installs the [GitHub CLI](https://cli.github.com/) (`gh`) when it is missing (via Homebrew or apt). The Windows PowerShell installer does the same via `winget install --id GitHub.cli`. OpenSRE's `github_cli` chat tools need `gh` on `PATH`. Skip that step with: ```bash theme={null} curl -fsSL https://install.opensre.com | OPENSRE_SKIP_GH_INSTALL=1 bash ``` Homebrew installs pull `gh` automatically via `depends_on "gh"` in the OpenSRE formula. ### 2) Enter the OpenSRE shell ```bash theme={null} opensre ``` ### 3) Run onboarding From inside the OpenSRE shell: ```bash theme={null} onboard ``` `onboard` helps you configure: * LLM provider (for investigation reasoning) * integration credentials * optional communication/reporting integrations ### 4) Verify integration health From inside the OpenSRE shell: ```bash theme={null} integrations verify ``` To verify one integration: ```bash theme={null} integrations verify ``` ### 5) Run your first investigation **From inside the OpenSRE shell** (after running bare `opensre`): ```bash theme={null} investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` **From your normal terminal** (direct investigation, no shell): ```bash theme={null} opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` ## Integration catalog The full integration catalog is maintained under the `Integrations` tab, not the `Installation` tab. Start here: * [Integrations overview](/docs/integrations-overview) Then open each provider page for credentials, minimum permissions, and setup examples. ## Related guides * [Quickstart](/docs/quickstart) * [Investigations overview](/docs/investigation-overview) # Integrations overview Source: https://opensre.com/docs/integrations-overview Connect observability, cloud, data, and messaging tools for automated RCA OpenSRE investigates alerts by querying the same tools your engineers use. Connect your stack so OpenSRE can pull logs, check recent deploys, map dependencies, and deliver findings where your team already works. ## Two setup paths | Path | Best for | How to configure | | -------------------------- | ---------------------------------------------- | ---------------------------------------------------------------- | | **Local CLI** | Self-hosted `opensre` on your laptop or server | `opensre onboard` or `opensre integrations setup`, plus env vars | | **Hosted (OpenSRE Cloud)** | Org-wide connectors shared across users | [app.tracer.cloud](https://app.tracer.cloud) web app | Most users start with the **local CLI** path. You need credentials for the tools you want to connect and permission to create API keys. ## Local CLI setup ```bash theme={null} opensre integrations setup ``` Or set environment variables — OpenSRE picks them up automatically on the next run. ### Verify ```bash theme={null} opensre integrations verify opensre integrations verify datadog ``` Inside the interactive shell you can also run `/integrations list`, `/integrations verify `, `/verify`, and `/health`. ## Integration catalog | Category | Integrations | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Observability and incidents** | [Alertmanager](/docs/alertmanager), [Azure Monitor](/docs/azure-monitor), [Better Stack](/docs/betterstack), [Coralogix](/docs/coralogix), [Datadog](/docs/datadog), [Grafana](/docs/grafana), [Hermes](/docs/hermes), [Honeycomb](/docs/honeycomb), [Incident.io](/docs/incident_io), [OpenObserve](/docs/openobserve), [OpsGenie](/docs/opsgenie), [Sentry](/docs/sentry), [SignOz](/docs/signoz), [Splunk](/docs/splunk), [Victoria Logs](/docs/victoria_logs) | | **Cloud and infrastructure** | [AWS](/docs/aws), [Argo CD](/docs/argocd), [Helm](/docs/helm), [Jenkins](/docs/jenkins), [Vercel](/docs/vercel) | | **Databases and data platforms** | [Azure SQL](/docs/azure-sql), [ClickHouse](/docs/clickhouse), [Kafka](/docs/kafka), [MariaDB](/docs/mariadb), [MongoDB](/docs/mongodb), [MongoDB Atlas](/docs/mongodb-atlas), [MySQL](/docs/mysql), [OpenSearch](/docs/opensearch), [PostgreSQL](/docs/postgresql), [RabbitMQ](/docs/rabbitmq), [RDS](/docs/rds), [Snowflake](/docs/snowflake), [Supabase](/docs/supabase) | | **Source control and collaboration** | [Bitbucket](/docs/bitbucket), [GitHub](/docs/github), [GitHub Actions](/docs/integrations/github-actions), [GitLab](/docs/gitlab), [Google Docs](/docs/google-docs), [Jira](/docs/jira), [Trello](/docs/integrations/trello) | | **Workflow orchestration** | [Airflow](/docs/airflow), [Dagster](/docs/dagster), [Prefect](/docs/prefect), [Temporal](/docs/temporal) | | **Messaging** | [Slack](/docs/messaging/slack), [Discord](/docs/messaging/discord), [Telegram](/docs/messaging/telegram), [Rocket.Chat](/docs/messaging/rocketchat), [WhatsApp](/docs/messaging/whatsapp), [Twilio SMS](/docs/messaging/twilio-sms) | | **AI coding assistants** | [OpenClaw](/docs/openclaw) | [Multi-instance integrations](/docs/multi-instance-integrations) let you register multiple clusters, regions, or accounts per provider (for example prod and staging Grafana). ## Hosted (OpenSRE Cloud) setup Enterprise connectors are configured through the OpenSRE web app at [app.tracer.cloud](https://app.tracer.cloud). Once connected, they are available to all users in your organization. | Category | Integrations | Setup doc | | ------------------ | ----------------------------------- | -------------------------------------------------------------- | | **Observability** | Grafana, Datadog, Sentry, Honeycomb | [Grafana](/docs/grafana) (Option 4), [Datadog](/docs/datadog) (Option 4) | | **Infrastructure** | AWS, Kubernetes | [AWS](/docs/aws), [Kubernetes](/docs/kubernetes) | | **Communication** | Slack, Discord, Telegram | [Messaging overview](/docs/messaging/index) | After connecting in the web app, confirm each connector shows **connected**, then run a test investigation or verify the same integration locally with `opensre integrations verify ` if your team also uses the CLI. See [Enterprise setup](/docs/install-enterprise) for the full org onboarding checklist. Integrations ### Example: Datadog in the web app 1. **Name** — distinguish between instances (for example `prod` vs `staging`) 2. **API key** — create in Datadog and paste into OpenSRE 3. **Application key** — create in Datadog and paste into OpenSRE Connect Datadog # Bash Source: https://opensre.com/docs/integrations/bash Using Tracer with Bash scripts Tracer can monitor any Bash-based pipeline, from small scripts to large chained commands, giving instant visibility into each command’s behavior. ## Why use Tracer in combination with Bash Many scientific or data pipelines rely on simple shell scripts that lack structured logging. Tracer automatically adds observability: * Traces every subprocess (even background jobs) * Captures CPU, memory, and I/O usage for each command * Provides a single timeline of all steps * Enables root-cause debugging for failures or stalls * Works for scripts, loops, and complex shell pipelines ## Getting started ### Prerequisites * Bash 4.0 or higher * Tracer installed on [your operating system](/docs/environments/overview) ### Just run your pipeline, Tracer will automatically attach If Tracer is already installed on your operating system, you only need to enable the Tracer agent for pipelines that have not been run with Tracer before.
In that case, run the following command: ```bash theme={null} sudo tracer init --token ``` Go to our [onboarding](https://app.tracer.cloud/dashboard) to get your own personal token When running this command, you will be asked to name your pipeline for clear labeling in the dashboard. ## Examples Run a Bash script under Tracer: ```bash theme={null} bash my_analysis.sh ``` or launch the Tracer demo workflow: ```bash theme={null} sudo tracer demo ``` Once the pipeline starts, open the Tracer dashboard, and you’ll see each command as a timeline step updating in real time. Tracer Logo Tracer Logo
Watch your pipeline run in the Tracer dashboard
View real-time metrics, resource usage, and performance insights for your pipeline runs.
# GitHub Actions Source: https://opensre.com/docs/integrations/github-actions # GitHub Actions OpenSRE’s GitHub Actions integration helps you trace incidents back to the workflow run that caused them. It is designed for situations where a failed deploy, a flaky test, or a broken secret rotation explains a production problem. ## What it provides * Recent workflow runs for a repository, including status and trigger * Job and step summaries for a workflow run * Failed job step log output * Currently active workflow runs ## Configuration This integration uses the same GitHub MCP credentials as the existing GitHub setup when available. ### Setup ```bash theme={null} opensre integrations setup github opensre integrations verify github ``` Ensure the GitHub MCP toolsets include `actions` (default: `repos,issues,pull_requests,actions`). ### Required * `GITHUB_MCP_AUTH_TOKEN` (or a GitHub token supplied via the GitHub MCP integration) ### Optional * `GITHUB_MCP_URL` to point at a custom MCP endpoint (defaults to `https://api.githubcopilot.com/mcp/`) * `GITHUB_MCP_MODE`, `GITHUB_MCP_COMMAND`, `GITHUB_MCP_ARGS` for stdio-based MCP setups If no token is configured, GitHub Actions tools will report that the GitHub integration is unavailable. ## Verify ```bash theme={null} opensre integrations verify github ``` Confirm the verify output mentions Actions tools. Then test from the REPL: ```text theme={null} > list recent failed workflow runs for Tracer-Cloud/opensre ``` ## Troubleshooting | Symptom | Fix | | ---------------------------------- | ------------------------------------------------------------------------------------------- | | **GitHub integration unavailable** | Run `opensre integrations setup github` and verify token scopes include `repo` | | **No workflow runs returned** | Confirm `owner`/`repo` match the repository; check token access to private repos | | **Step log empty** | Use `job_id` and exact `step_name` from `list_github_actions_run_jobs` | | **MCP connection failed** | Verify `GITHUB_MCP_URL` is reachable; see [GitHub troubleshooting](/docs/github#troubleshooting) | ## Available tools ### `list_github_actions_workflow_runs` List recent workflow runs for a repository. Example: ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre", "per_page": 10 } ``` ### `list_github_actions_active_runs` List queued and in-progress runs. Example: ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre" } ``` ### `list_github_actions_run_jobs` List jobs and step outcomes for a run. Example: ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre", "run_id": 123456789 } ``` ### `get_github_actions_step_log` Fetch the log output for a failed job step. Example: ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre", "run_id": 123456789, "job_id": 987654321, "step_name": "Deploy" } ``` ## Investigation workflow 1. Find the workflow run that happened right before the incident. 2. Open the job list and identify the failed job or step. 3. Pull the failed step log and look for the exact deployment/test error. 4. Use the run metadata to correlate the failure with commits, pull requests, or a secret/config change. ## Example RCA usage A failed deployment workflow often shows up as: * a run with `conclusion: failure` * a job named `deploy`, `release`, or `rollout` * a step such as `Deploy`, `Apply manifests`, or `Run migrations` That context is usually enough to connect the incident to a recent workflow change. # Integration Workflow Source: https://opensre.com/docs/integrations/integration-workflow Choose the right framework integration for your workflow Tracer seamlessly integrates with a wide array of workflow managers, schedulers, and scripting environments.
Choose the integration that best matches your pipeline infrastructure. ## Common Integration Steps Regardless of your framework, Tracer integration follows these simple steps: Start the Tracer agent with your organization token
`sudo tracer init --token ` Go to our [onboarding](https://app.tracer.cloud/dashboard) to get your own personal token Execute your pipeline as usual, Tracer automatically monitors all processes.
-> Monitor your pipeline run in [the Tracer dashboard](https://app.tracer.cloud/tracer-bioinformatics) ## Verify Delivery Check the [Tracer dashboard](https://app.tracer.cloud/) for real-time insights Tracer Dashboard ## Key Features Across All Integrations * **Zero Code Changes** - No instrumentation required, works at the system level * **Real-Time Monitoring** - See your pipeline execution as it happens * **Resource Tracking** - CPU, memory, I/O, and network metrics per process * **Cost Attribution** - Understand the cost of each pipeline step * **Execution Graphs** - Visualize dependencies and execution flow * **Performance Insights** - Identify bottlenecks and optimization opportunities # Nextflow Source: https://opensre.com/docs/integrations/nextflow Monitor Nextflow pipelines with Tracer Tracer integrates with Nextflow to provide step-level visibility into every process and container.
Once installed, it automatically detects all Nextflow processes and displays real-time metrics, performance traces, and cost data for each task. ## Why use Tracer when running Nextflow pipelines Many teams running Nextflow also rely on Seqera Platform to orchestrate and monitor workflows across different environments.
Because both tools appear in the same setups, we’ve prepared an objective comparison that explains how Tracer and Seqera differ in focus and how they can complement each other effectively. You can read the full comparison here: [Tracer vs Seqera](/docs/comparisons/Tracer_vs_Seqera) Nextflow orchestrates complex scientific workflows efficiently, but it doesn’t provide full visibility into why certain tasks run slowly, stall, or waste compute. Tracer fills that gap: * Real-time insight into every process, container, and tool * Performance metrics (CPU, memory, I/O, GPU) per task * Automatic detection of idle or blocked tasks * AI-native logging for easy debugging and root-cause analysis * Works across local, HPC, or cloud environments without changes to your Nextflow code ## Getting Started ### Prerequisites * Nextflow installed and working (`nextflow run hello` test passes) * Tracer installed on [your operating system](/docs/environments/overview) ### Just run your pipeline, Tracer will automatically attach If Tracer is already installed on your operating system, you only need to enable the Tracer agent for pipelines that have not been run with Tracer before.
In that case, run the following command: ```bash theme={null} sudo tracer init --token ``` Go to our [onboarding](https://app.tracer.cloud/dashboard) to get your own personal token When running this command, you will be asked to name your pipeline for clear labeling in the dashboard. ## Examples Launch the Tracer demo workflow: ```bash theme={null} sudo tracer demo ``` Or run any Nextflow pipeline as usual. Once the pipeline starts, open the Tracer dashboard, and you’ll see each Nextflow process as a timeline step updating in real time. Tracer Logo Tracer Logo
Watch your pipeline run in the Tracer dashboard
View real-time metrics, resource usage, and performance insights for your pipeline runs.
# Slurm Source: https://opensre.com/docs/integrations/slurm Monitor Slurm workloads with Tracer Tracer integrates with Slurm, providing detailed observability for batch jobs and array tasks. It works by running the Tracer agent on each compute node — no modification to job scripts required. ## Why use Tracer in combination with Slurm Slurm gives job-level scheduling visibility, but not what happens inside the job. Tracer adds that missing layer: * Per-process telemetry inside each job allocation * Job array correlation and node-level performance * Resource and cost insights across users and queues * Zero changes to job submission or scripts * Real-time updates in the Tracer dashboard ## Getting Started ### Prerequisites * Slurm cluster access with sudo or admin privileges for installation * Tracer installed on [your operating system](/docs/environments/overview) ### Just run your pipeline, Tracer will automatically attach If Tracer is already installed on your operating system, you only need to enable the Tracer agent for pipelines that have not been run with Tracer before.
In that case, run the following command: ```bash theme={null} sudo tracer init --token ``` Go to our [onboarding](https://app.tracer.cloud/dashboard) to get your own personal token When running this command, you will be asked to name your pipeline for clear labeling in the dashboard. ## Examples Run a Slurm pipeline under Tracer: ```bash theme={null} #!/bin/bash #SBATCH --job-name=test #SBATCH --cpus-per-task=8 #SBATCH --time=01:00:00 module load python python analysis.py ``` Submit this job as usual with: ```bashsbatch theme={null} my_job.sh ``` or launch the Tracer demo workflow: ```bash theme={null} sudo tracer demo ``` Once the pipeline starts, open the Tracer dashboard, and you’ll see each Slurm job as a timeline step updating in real time. Tracer Logo Tracer Logo
Watch your pipeline run in the Tracer dashboard
View real-time metrics, resource usage, and performance insights for your pipeline runs.
# Snakemake Source: https://opensre.com/docs/integrations/snakemake Monitor Snakemake workflows with Tracer Tracer integrates with Snakemake to provide pipeline-level observability. It captures per-rule execution times, resource usage, and cross-node performance metrics. Install the Tracer agent on the machines where Snakemake executes and it will automatically collect detailed metrics from every rule and job. You don’t need to modify your Snakefile or add wrappers because Tracer passively observes workflow execution at the system level using eBPF. Whether running locally, on HPC, or in the cloud, Tracer provides real-time visibility into resource usage, execution timelines, and performance bottlenecks across the workflow. ## Why use Tracer in combination with Snakemake Snakemake reports high-level progress, but Tracer explains why rules take time or fail: * View CPU, memory, and I/O usage per rule * Identify bottlenecks or inefficient resource requests * Attribute cost per sample or rule * Trace nested shell commands and containers automatically * Correlate logs and metrics across distributed environments ## Getting Started ### Prerequisites * Working Snakemake installation * Tracer installed on [your operating system](/docs/environments/overview) ### Just run your pipeline, Tracer will automatically attach If Tracer is already installed on your operating system, you only need to enable the Tracer agent for pipelines that have not been run with Tracer before.
In that case, run the following command: ```bash theme={null} sudo tracer init --token ``` Go to our [onboarding](https://app.tracer.cloud/dashboard) to get your own personal token When running this command, you will be asked to name your pipeline for clear labeling in the dashboard. ## Examples Run a Snakemake pipeline under Tracer: ```bash theme={null} snakemake -j 16 ``` or launch the Tracer demo workflow: ```bash theme={null} sudo tracer demo ``` Once the pipeline starts, open the Tracer dashboard, and you’ll see each Snakemake rule as a timeline step updating in real time. Tracer Logo Tracer Logo
Watch your pipeline run in the Tracer dashboard
View real-time metrics, resource usage, and performance insights for your pipeline runs.
Each rule in your Snakefile will appear as a timeline step, with detailed performance and cost breakdowns. # Trello Source: https://opensre.com/docs/integrations/trello Connect Trello so OpenSRE can view cards and boards during investigations and incident management When incidents happen, keeping everyone organized matters. OpenSRE connects to Trello so you can create incident cards, track investigation progress, and manage follow-up tasks — all without leaving your incident response workflow. ## What you need * A Trello account with access to the boards you want OpenSRE to use * A Trello API key and token * Optional board and list IDs for automatic card creation ## Setting up Trello ### Quick setup ```bash theme={null} opensre integrations setup ``` Choose **Trello** and enter your API key and token when prompted. ### Manual config: Environment variables Or add these to your `.env`: ```bash theme={null} TRELLO_API_KEY=your_api_key TRELLO_TOKEN=your_token TRELLO_BOARD_ID=board_id_optional TRELLO_LIST_ID=list_id_optional TRELLO_BASE_URL=https://api.trello.com/1 ``` | Variable | Default | Description | | ----------------- | -------------------------- | ------------------------------------------- | | `TRELLO_API_KEY` | — | **Required.** Trello API key | | `TRELLO_TOKEN` | — | **Required.** Trello API token | | `TRELLO_BOARD_ID` | *(empty)* | Optional default board ID for card creation | | `TRELLO_LIST_ID` | *(empty)* | Optional default list ID for card creation | | `TRELLO_BASE_URL` | `https://api.trello.com/1` | Trello API base URL | ### Persistent store You can also save your Trello configuration to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "trello-prod", "service": "trello", "status": "active", "credentials": { "api_key": "your_api_key", "token": "your_token", "board_id": "board_id_optional", "list_id": "list_id_optional" } } ] } ``` ## Getting your API key and token 1. Visit [https://trello.com/app-key](https://trello.com/app-key) 2. Copy your Trello API key 3. Generate a token from the same page 4. Authorize access when prompted 5. Store both values securely ## Optional: Finding board and list IDs If you'd like OpenSRE to create cards in a specific location, you'll need the board and list IDs. ### Using the API ```bash theme={null} curl "https://api.trello.com/1/members/me/boards?key=${TRELLO_API_KEY}&token=${TRELLO_TOKEN}" ``` ### Manual method 1. Open the board in Trello 2. Append `.json` to the board URL 3. Search the response for the board ID 4. Locate the desired list and copy its ID ## Token longevity Trello tokens can be created with different expiration periods. When generating a token, choose an expiration period that aligns with your organization's security requirements. For long-lived OpenSRE integrations, consider using an appropriately scoped token and rotating it regularly according to your security policies. ## Required permissions Your Trello token should have: * **read** — Allows OpenSRE to view boards, lists, and cards * **write** — Allows OpenSRE to create or update cards during investigations ## Usage Once configured, OpenSRE can: * Create Trello cards from investigation findings * Send incident titles and descriptions to a target list * Track follow-up tasks on your incident board Set `TRELLO_LIST_ID` to the list where new cards should appear. If omitted, card creation requires the list ID at investigation time. Trello does not currently support `opensre integrations verify trello`. Confirm your credentials during interactive setup or by creating a test card from an investigation. # WDL Source: https://opensre.com/docs/integrations/wdl Using Tracer with WDL workflows Tracer integrates with WDL (Workflow Description Language) pipelines executed through engines like Cromwell or miniWDL. It automatically traces all WDL tasks as they run, giving per-task performance data and cross-node visibility without modifying workflow definitions. ## Why use Tracer in combination with WDL While WDL focuses on portability and reproducibility, it lacks system-level telemetry. Tracer adds observability that helps teams debug and optimize large WDL workflows: * Step-level execution graphs for each WDL task * Real-time metrics for CPU, memory, I/O, and storage * Correlated logs and system traces even if task logs are minimal * Cost and resource attribution by sample, node, or step * Works locally, in HPC, or in cloud environments (AWS, GCP, Azure) ## Getting Started ### Prerequisites * WDL engine installed and working (e.g., Cromwell or miniWDL) * Tracer installed on [your operating system](/docs/environments/overview) ### Just run your pipeline, Tracer will automatically attach If Tracer is already installed on your operating system, you only need to enable the Tracer agent for pipelines that have not been run with Tracer before.
In that case, run the following command: ```bash theme={null} sudo tracer init --token ``` Go to our [onboarding](https://app.tracer.cloud/dashboard) to get your own personal token When running this command, you will be asked to name your pipeline for clear labeling in the dashboard. ## Examples Launch the Tracer demo workflow: ```bash theme={null} sudo tracer demo ``` Or run your WDL pipeline as usual. Once the pipeline starts, open the Tracer dashboard, and you’ll see each WDL task as a timeline step updating in real time. Tracer Logo Tracer Logo
Watch your pipeline run in the Tracer dashboard
View real-time metrics, resource usage, and performance insights for your pipeline runs.
# Interactive shell action policy Source: https://opensre.com/docs/interactive-shell-action-policy # Interactive Shell Action Policy (ADR) ## Status Superseded — Jun 18, 2026. The declarative-rule-pack deterministic mapper and the regex-based planner postprocessing overrides described in the original decision have been removed. See "Decision (current): LLM is the sole tool selector" below. The original decision is retained for historical context. ## Context The interactive-shell action policy had grown through layered heuristics in single modules: a regex/keyword deterministic mapper inferred tools from free-form text, and planner postprocessing rewrote the model's chosen actions with more regex. These heuristics competed with the LLM and caused misclassifications (e.g. "investigate a sample test alert?" being treated as an informational question instead of running the sample alert), and they were a recurring source of precedence drift. ## Decision (current): The shell action agent is the sole tool selector 1. There is no regex/keyword intent inference. Non-command turns are selected entirely by the shell action agent via native tool-calling. 2. Tool selection is driven by the action-agent system prompt (`core/agent_harness/prompts/action/assemble.py`) and the per-tool descriptions in the tool catalog (`tools/interactive_shell/*`). Keep both precise — they are the only selection signal. 3. The action path does not post-hoc rewrite the model's tool calls. Tool calls execute as first-class `AgentTool`s through the shared `core` tool-calling loop; argument shape and availability are enforced by the AgentTool runtime contract and per-tool gates. 4. When the action-agent prompt overflows the context window, the turn falls through to a conversational reply rather than guessing an action. When the action-agent LLM itself is unavailable, the REPL renders and persists a failed assistant turn so `/resume` can show the outage. 5. Literal `/slash` command text the user types verbatim is dispatched deterministically, without the action-agent LLM (see the "Deterministic literal-`/slash` dispatch" addendum below). This is an explicit-command bypass, not natural-language intent inference: free-form text is still selected entirely by the action agent. The runtime's literal-`/slash` detection in `runtime/utils/input_policy._literal_slash_command_text` remains terminal-UI policy (spinner suppression and exclusive-stdin gating); the execution-side deterministic dispatch lives in `core/agent_harness/turns/action_driver.py`. ## What this means for changes * To change how a phrasing maps to a tool, edit the action-agent system prompt and/or the relevant tool description — never add a regex. * To add a new tool, add it to the tool catalog with a clear, self-describing `description` and `input_schema`; the action agent selects it from that text and receives it as an AgentTool. * Live turn scenarios under `tests/core/agent/scenarios/` are the regression surface for action-agent behavior. Deterministic scenarios (`intent_class: deterministic`) assert literal command dispatch only. ## Original decision (historical, superseded) 1. Deterministic mapping was split into declarative rule packs with one explicit precedence table. 2. Rule matching windows were named typed strategies instead of inline numeric slices. 3. Planner postprocessing ran as pure transforms over a typed `PlannerState`. 4. Fail-closed policy transforms and normalization transforms were registered separately and executed in one ordered list. 5. Legacy planner-result tuple compatibility was collapsed behind a single adapter. 6. Planner contracts included policy-trace artifacts to detect silent precedence drift. ## Integration awareness and LLM-driven read-only discovery Addendum — Jun 18, 2026. Factual questions about live state (for example "is sentry installed?") are answered without adding keyword/regex rules. Two complementary mechanisms: 1. Context grounding (not action planning). At REPL boot, `run_repl_async` (`surfaces/interactive_shell/main.py`) hydrates `session.configured_integrations` from the shared `configured_integration_services()` helper in `integrations/catalog.py` (the same source the welcome banner uses, so they never diverge). The chat assistant prompt (`build_environment_block` in `core/agent_harness/prompts/assistant.py`) lists the configured set as facts, letting the model answer directly when state is already known. 2. LLM-driven discovery. The action-agent system prompt (`core/agent_harness/prompts/action/assemble.py`) lets the model, at its own discretion, emit a read-only discovery action (for example `slash_invoke("/integrations", ["list"])` or `["verify"]`) to discover the answer instead of deflecting. There is no keyword mapping for this — the LLM decides. Under the alpha allow-all policy every discovery action runs without confirmation (`execution_policy.allow_tool("slash")` returns `allow`); the former `ExecutionTier`/`resolve_slash_execution_tier` classification was removed because it gated nothing. No fail-closed regex rule is involved; the action agent decides whether to emit a discovery action. ### Observe→answer summary loop Addendum — Jun 18, 2026. When the action agent runs a read-only discovery command to answer a question (e.g. the user asks "is sentry installed?" and the model runs `/integrations`), the raw command output (a verification table) is not a direct answer on its own. The pipeline now follows up with a short assistant pass that summarizes that output: 1. Read-only discovery slash commands stash a compact text view of what they found on `session.agent.last_observation` (`_record_integrations_observation` in `surfaces/interactive_shell/command_registry/integrations.py`). 2. `run_agent_prompt` resets that field at the start of every action-agent turn and, when a discovery command produced an observation and succeeded, calls the conversational assistant with `tool_observation=...` (inside the handled-turn observation branch in `pipeline.py`). The assistant summarizes the output into a direct answer and is instructed not to emit further actions. This only fires when the action-agent tool path executes a read-only discovery command and records an observation. The pipeline no longer has a pre-agent deterministic dispatch branch. Discovery commands also no longer dump validator stack traces into the REPL: a vendor/config failure during verification (for example a GitHub MCP `401`) is logged as a one-line warning instead of a full traceback, because `report_validation_failure` now defaults to `include_traceback=False` while still capturing the exception to Sentry. ### Auto-launching interactive setup ("can you configure X?") Addendum — Jun 18, 2026. When the user asks to configure, connect, set up, or add an integration ("can you configure sentry?", "connect datadog"), the action agent does not just hand off to the conversational assistant — it launches the setup wizard for them. The action agent emits a `slash_invoke` tool call for `/integrations setup ` or `/mcp connect `. The model chooses the service; there is no per-vendor hardcoding. The setup wizard is a child process that needs exclusive stdin, so it cannot run inline mid-turn (the live prompt is competing for stdin). Instead `tools/interactive_shell/actions/slash.py` queues the command via `session.queue_auto_command(...)`, which prefills the next prompt and marks it for auto-submit. The prompt refresh hook (`wire_prompt_refresh` in `surfaces/interactive_shell/ui/input_prompt/refresh.py`) then submits it, so the command flows through the normal exclusive-stdin turn path of the REPL (`turn_needs_exclusive_stdin` recognizes `/integrations setup`) — the only place an interactive child process gets clean stdin. In a non-TTY/scripted context (no prompt to submit into), the slash command path degrades to normal non-interactive slash behavior. ### Removal of the planning-stage fail-closed safeguard (v0.1) Addendum — Jun 18, 2026. The action agent does not deny a turn. Previously, any clause the old planner could not map to an executable tool — flagged via the `mark_unhandled` tool, an `UNHANDLED:` text marker, or an unavailable tool call — collapsed the whole turn into a hard denial that printed *"I couldn't safely decide actions for that request."* In practice this fired on legitimate input (most often a conversational question that embedded a quoted, list-style directive such as *figure out why X is crashing by querying (a) sentry, (b) github, (c) posthog*), producing a dead end with no safety benefit. Every terminal action in v0.1 is **read-only**, so an unmatched, ambiguous, or chatty clause is not a safety risk. The action agent now: * runs every clause it *can* map to an executable action, and * lets everything else fall through to the conversational assistant (or simply drops a chatty clause in a compound request). Removed as part of this change: the `denied` field on `ActionPlanningDecision`, `enforce_plan_fail_closed_policy`, `normalize_terminal_plan`, `render_plan_denied`, the `mark_unhandled` planner tool, and the `UNHANDLED:` convention. The `fail_closed`, `has_unhandled_clause`, and `turn.expected_signals` fields were also removed from turn scenario fixtures, since the oracle never asserted on them; the fixture `policy` block now carries a single `executes_terminal_action` `boolean` (true only when a shell action AgentTool is expected to run). If write/mutating actions are introduced later, gate them with the execution-stage confirmation policy (`tools/interactive_shell/shared/execution_policy.py`), **not** an action-selection denial. ### Removal of the shell-command safety policy (alpha) Addendum — Jun 27, 2026. **Decision:** while OpenSRE is in **alpha**, the interactive REPL runs **every** shell command with **no guardrails**. The shell-command safety policy — the read-only / mutating / restricted classification, the command allowlist, and the hard `deny` floor — has been removed. This is a deliberate trade-off: alpha prioritizes developer velocity over command sandboxing, and the REPL already runs on the developer's own machine with their own privileges. What changed: * `shell_policy.py` (classification, allowlists, `classify_command`, `evaluate_policy`, `PolicyDecision`) was deleted. The pure parsing helpers it also contained moved to `tools/shell/parsing.py` (`parse_shell_command`, `argv_for_repl_builtin_detection`, `ParsedShellCommand`), alongside the shell execution policy in `tools/shell/policy.py`. * `tools.shell.policy.evaluate_shell_from_parsed` now returns `allow` for every command — read-only, mutating, `restricted` (`sudo`, `systemctl`, `kill`, `dd`, …), shell operators (`| && ; > <`), and command substitution (`` ` ``/`$(...)`). Commands that need a shell run through one automatically; the `!` prefix is still honored but no longer required to escape the old operator block. * The **only** remaining non-execution outcome is genuinely empty input (a bare `!` or whitespace), which is rejected as input validation, not as a guardrail. The `ask`/confirmation machinery (`trust_mode` plus the confirmation UX) is retained as an unused hook, split across two layers: the pure decision lives in `tools/interactive_shell/shared/execution_policy.py` (`resolve_confirmation`), and the terminal interaction (`execution_allowed` — console output, the `Proceed? [Y/n]` prompt, analytics) lives in `surfaces/interactive_shell/ui/execution_confirm.py`. If command guardrails are reintroduced after alpha, gate them here at the execution stage — never with an action-selection denial in the planner. ### Deterministic literal-`/slash` dispatch (no LLM) Addendum — Jun 28, 2026. **Decision:** input the user types as a literal `/slash` command is dispatched deterministically, **without consulting the action-agent LLM**. This supersedes the earlier "the literal-`/slash` detection must never become an action execution shortcut" wording. **Why:** all REPL turns previously routed through the action-agent LLM, so when that LLM was unavailable (a provider with no credit, a failed auth, an outage) every slash command failed — including the exact commands needed to recover (`/login`, `/auth`, `/onboard`, `/model`). That is a deadlock: you could not log in because logging in required the LLM you were trying to fix. Typed commands should not depend on a funded LLM. **Scope — the line that keeps the original concern intact.** The original ADR removed regex/keyword heuristics because they *inferred intent from natural language* and competed with the LLM. This bypass does the opposite: it fires **only** when the message text itself is a literal `/command` the user typed verbatim. There is no inference. Free-form natural language ("log me in", "show my integrations") is still selected entirely by the action agent. The line is "explicit typed command" vs "natural language", identical to the line the terminal-UI policy (`_literal_slash_command_text`) already draws. **How it works.** `core/agent_harness/turns/action_driver.ActionTurnRunner` recognizes literal `/slash` input and emits a deterministic `slash_invoke` tool call through the same static-LLM path as the explicit `!cmd` shell escape (`_StaticToolCallLLM`). Execution then flows through the normal `slash_invoke` AgentTool → `dispatch_slash`, so recording, execution policy, exclusive-stdin gating, pickers, and exit behavior are identical to an LLM-selected slash call — the only difference is the tool *selection* is deterministic instead of LLM-driven. The bypass no-ops (falls back to the LLM path) when `slash_invoke` is not an available tool that turn. **Consequences.** * Literal slash commands are faster, free, and reliable even with no LLM credit. * Compound requests that *start with* a literal slash are dispatched as that one command (a single `slash_invoke`); compound phrasing that does not start with a slash (e.g. `run /health and then investigate …`) is unaffected and still LLM-routed. No turn scenario uses a prompt beginning with a literal `/`. * A literal discovery command (e.g. `/integrations list`) shows its raw output directly; the optional LLM summary pass only runs if the action-agent LLM is available, and degrades cleanly (no summary) when it is not. **Still forbidden:** regex/keyword/fuzzy intent routing for natural language, post-hoc rewriting of LLM-selected tool calls, and any deterministic mapping from non-`/`-prefixed text to an action. Those compete with the LLM and were removed for good reason. # Assistant with Connected Tools Source: https://opensre.com/docs/interactive-shell-assistant How the interactive shell assistant pulls live data from your connected integrations to answer free-form questions When you ask the OpenSRE interactive shell a free-form question, the assistant can pull **live data from your connected integrations** before answering — instead of replying from text alone. If a question is answerable from an integration you have configured (GitHub, Datadog, Sentry, Grafana, cloud services, and more), the assistant transparently calls the same tools the investigation pipeline uses, gathers the results, and answers from that real data. This happens automatically. You do not run a slash command or pick a tool — just describe what you want to know. ## Example ```text theme={null} > can you check the github issues for the windows crashes in https://github.com/Tracer-Cloud/opensre? · gathering via GitHub · search issues — windows crashes… There are 3 open issues mentioning Windows crashes. The most recent, #482 "App crashes on launch (Windows 11)", reports a startup segfault after the 2.4.0 update and has 6 reactions… ``` You can name the repository inline (`… in https://github.com/org/repo`), mention it in an earlier turn, rely on `GITHUB_REPOSITORY`, or let OpenSRE infer it from the current git checkout's `origin` remote when GitHub is connected. The dim `· gathering via · — …` line shows which integration the assistant reached for while it works. When the same tool runs more than once, a `(2)`, `(3)`, … suffix and the query hint distinguish each call. ### End-to-end example ```text theme={null} > what do the datadog logs say about payment-service errors in the last hour? · gathering via Datadog · query logs — payment-service errors… Found 847 error log lines in the last hour for service:payment-service. Top pattern: ConnectionTimeout to postgres-primary (412 occurrences)… ``` Prerequisites: Datadog connected (`opensre integrations verify datadog`) and `LLM_PROVIDER` configured. Run `/health` if gathering does not start. More questions that trigger live gathering: | You ask | The assistant gathers from | | --------------------------------------------------------- | -------------------------- | | "check the GitHub issues for the Windows crashes" | GitHub issues | | "what do the datadog logs say about the payment service?" | Datadog logs | | "are there recent Sentry errors on checkout?" | Sentry | | "what's the recent error rate in grafana for prod-api?" | Grafana | ## How it works 1. You ask a question in plain language. 2. The assistant runs a short, bounded tool-gathering pass over the tools your **configured integrations** expose. 3. Tools are **read-only data fetches**, so they run autonomously — no per-call confirmation, just like an investigation. 4. The gathered results are folded into the answer the assistant writes back to you. If gathering is interrupted (Ctrl+C) or a tool is unavailable, the assistant falls back to a normal text answer. ## When nothing is configured If you have **no integrations configured**, behavior is unchanged: the assistant answers from text only, exactly as before. There is nothing to enable or turn on — connect an integration and the relevant questions start returning live data. Run `/health` to see which integrations are connected, and `/tools` to list the tools currently available to the assistant. ## Related docs * [Interactive Shell Commands](/docs/interactive-shell-commands) — slash commands and natural-language actions * [Integrations Overview](/docs/integrations-overview) — connecting data sources * [GitHub](/docs/github) — configuring the GitHub integration used by the issues example * [Investigation overview](/docs/investigation-overview) — the full RCA pipeline these tools also power # Interactive Shell Commands Source: https://opensre.com/docs/interactive-shell-commands Complete reference for every slash command in the OpenSRE REPL — session control, investigations, integrations, tasks, watchdogs, and more Start the interactive shell with `opensre` (TTY required). Type a slash command at the prompt, or describe what you want in plain language — the action agent can route intent to the right command. Run `/help` anytime for the live command list grouped by category. In a TTY, bare `/help` opens an interactive picker; selecting a command runs it directly. When you type `/` and browse completions with the arrow keys, the full description of the highlighted command appears in the hint line above the prompt. Commands marked **elevated** may prompt for confirmation unless [trust mode](#trust-mode-and-confirmations) is on. Non-TTY sessions fail closed on elevated actions. ## How the REPL works | Input type | What happens | | ------------------------------------ | ------------------------------------------------------------------------------------- | | Slash command (`/status`) | Routed through the action agent and executed via the `slash_invoke` AgentTool | | Plain language (`verify datadog`) | Routed by the action agent to slash commands, investigations, or doc-grounded answers | | Pasted alert JSON/text | Often starts an investigation without a slash command | | Shell one-liner (`kubectl get pods`) | Executed through shell policy when the action agent selects a shell action | **TTY vs non-TTY:** The full experience (interactive menus, confirmations, onboarding wizards) requires a real terminal. Piping input or running in CI sets non-interactive mode — elevated commands are rejected unless trust mode was enabled in a prior interactive session (prefer explicit CLI commands outside the REPL for automation). **Unknown commands:** Typos suggest the closest registered command (`Did you mean /integrations?`). Run `/help` for the authoritative list — it always matches your installed OpenSRE version. **Keyboard shortcuts:** | Key | Effect | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Up/Down** | Recall persisted command history (when enabled); during `/` completion browse, preview the highlighted command description in the hint line above the prompt | | **Ctrl+C** once | Cancel in-flight streaming work or interrupt the current prompt | | **Ctrl+C** twice within 2s | Exit the REPL (prints `/resume` hint) | | **Ctrl+D** | Exit when the prompt is empty (same resume hint as `/exit`) | *** ## Quick reference ### Help and exit | Command | What it does | | ------- | ------------------------------------------------------ | | `/help` | List commands or show help for one command or category | | `/?` | Shortcut for `/help` | | `/exit` | Leave the interactive shell | | `/quit` | Alias for `/exit` | ### Session | Command | What it does | | ---------------------- | -------------------------------------------------------------------- | | `/status` | Session summary — interactions, alerts, provider, effort, trust mode | | `/cost` | Token usage and LLM call count for the current session | | `/context` | Infra metadata accumulated during the session | | `/effort` | Set or show reasoning effort (`low` … `max`) | | `/trust` | Enable or disable trust mode (skip confirmation prompts) | | `/verbose` | Toggle verbose logging | | `/clear` | Clear the screen and re-render the banner | | `/compact` | Summarize older context into a replayable compaction entry | | `/sessions` | List recent REPL sessions on disk | | `/resume ` | Restore a past session's conversation context | | `/resume :` | Restore a specific session branch point | | `/new` | Start a new session file while keeping LLM context | ### Investigation | Command | What it does | | -------------- | ---------------------------------------------------- | | `/investigate` | Run an RCA from a file path or sample template | | `/template` | Print a starter alert JSON template | | `/last` | Reprint the most recent investigation report | | `/save ` | Export the last investigation to a file **elevated** | ### Integrations, models, and tools | Command | What it does | | --------------- | ---------------------------------------------------- | | `/health` | Integration and agent health check | | `/verify` | Verify integration connectivity (all or one service) | | `/integrations` | List, verify, show, setup, or remove integrations | | `/mcp` | List, connect, or disconnect MCP servers | | `/model` | Show or switch LLM provider and models | | `/tools` | List registered investigation and chat tools | ### Privacy and history | Command | What it does | | ---------- | ------------------------------------------------------- | | `/history` | Show or manage persisted command history | | `/privacy` | History persistence, redaction status, and threat model | ### Tasks and background work | Command | What it does | | -------------- | ----------------------------------------------------------------- | | `/background` | Run investigations async; track RCAs and completion notifications | | `/loops` | List, create, run once, and debug recurring prompt loops | | `/tasks` | List recent and in-flight background tasks | | `/work` | List, add, complete, and prioritize human work items | | `/cancel ` | Cancel a running task by id **elevated** | | `/stop` | Guidance for stopping investigations and background work | ### Watchdog | Command | What it does | | --------------- | ---------------------------------------------------------------------------------- | | `/watch` | Watch a process and send threshold alarms via Telegram or Rocket.Chat **elevated** | | `/watches` | List active watchdog tasks with latest samples | | `/unwatch ` | Stop a watchdog task by id **elevated** | | `/watchdog` | CLI-parity wrapper for the watchdog monitor | ### Agents and alerts | Command | What it does | | --------- | ---------------------------------------- | | `/fleet` | View and manage the local AI agent fleet | | `/alerts` | Alert listener inbox status | ### CLI parity These commands delegate to the same Click CLI you would run outside the REPL. | Command | What it does | | ------------- | ---------------------------------------------------------------------------------------- | | `/auth` | Log in to LLM providers, show provider auth status, or clear credentials | | `/login` | Shortcut for `/auth login` (`/login chatgpt`, `/login claude`, `/login deepseek`) | | `/onboard` | Interactive onboarding wizard | | `/remote` | Connect to and operate remote deployed agents | | `/config` | Show or edit `~/.opensre/config.yml` | | `/cron` | Manage scheduled delivery jobs | | `/sentry` | Schedule automated Sentry morning digests or uptime watches (Telegram or Slack required) | | `/messaging` | Telegram pairing and allowlist | | `/hermes` | Hermes log tailing and incident escalation | | `/guardrails` | Sensitive-information guardrail rules | | `/tests` | Browse and run inventoried tests | | `/update` | Check for updates and upgrade OpenSRE | | `/debug` | Targeted runtime diagnostics | | `/uninstall` | Remove OpenSRE and local data **elevated** | ### System | Command | What it does | | ---------- | --------------------------------- | | `/doctor` | Full local environment diagnostic | | `/version` | OpenSRE, Python, and OS versions | ### Commands with interactive menus (TTY) Bare invocation opens a picker or submenu: | Command | Menu behavior | | -------------------- | -------------------------------------------------- | | `/help` | Browse categories; Enter runs the selected command | | `/integrations` | list / verify / show / setup / remove | | `/mcp` | list / connect / disconnect | | `/model` | show / set / restore / toolcall | | `/investigate` | Demo alerts, templates, custom file path | | `/template` | Template type picker | | `/trust`, `/verbose` | on / off | | `/history` | show / clear / off / on / retention | | `/resume` | Numbered list of recent sessions | *** ## Using `/help` ```text theme={null} /help /help /model /help investigation /help tasks /help all ``` | Form | Result | | ------------------ | ------------------------------------------------------------------ | | `/help` | Interactive picker in a TTY; category index otherwise | | `/help ` | Detailed usage for one command (e.g. `/help /watch`) | | `/help ` | All commands in a section (`investigation`, `session`, `tasks`, …) | | `/help all` | Full command index | Help categories mirror the REPL grouping: Quick Access, Session, Integrations/Models/Tools, Investigation, Privacy, Tasks, Agents, Alerts, CLI parity, and System. **Quick Access** duplicates frequently used commands (`/investigate`, `/integrations`, `/model`, `/health`, `/watch`, `/status`, `/help`) for faster discovery in the picker. *** ## Session commands ### `/status` Shows a snapshot of the **current** REPL session: ```text theme={null} /status ``` Typical fields: | Field | Meaning | | --------------------- | -------------------------------------------------------------------------- | | `interactions` | Count of recorded turns (chat, slash, alerts) | | `incoming alerts` | Alerts received by the local listener this session, plus age of the latest | | `last investigation` | `yes` if `/investigate` or a streamed investigation completed | | `trust mode` | `on` / `off` | | `reasoning effort` | Current `/effort` level (OpenAI/Codex only) | | `provider` | Active `LLM_PROVIDER` | | `grounding … cache` | Doc/source cache stats used for help answers | | `accumulated context` | Comma-separated keys from prior investigations (e.g. `service`, `cluster`) | Use `/status` for **session** state. Use `/health` for **integration connectivity**. ### `/cost` Shows LLM usage tracked **locally** for the current REPL session. This is not cloud-provider billing and does not read your vendor invoice. ```text theme={null} /cost ``` Example output (measured + estimated mix): ```text theme={null} Session cost (includes estimates) history entries 12 llm calls 8 input tokens 4,210 (measured: 1,200 · estimated: 3,010) output tokens 890 (estimated: 890) ``` | Field | Meaning | | -------------------------------- | ----------------------------------------------------- | | `history entries` | Lines recorded in this session's in-memory history | | `llm calls` | LLM turns counted (planner + chat + help + follow-up) | | `input tokens` / `output tokens` | Running totals for prompt and completion usage | **Measured vs estimated:** | Code path | Token source | | ------------------------------------- | ---------------------------------------------------------- | | Planner `invoke()` (action selection) | Provider usage metadata when the API returns it | | Streaming chat, help, follow-up | Estimated from character length (\~4 characters per token) | When any estimate is included, the table title notes `(includes estimates)` and rows show measured vs estimated splits. **What increments the counter:** Normal LLM chat turns and planner calls. **What does not:** Non-LLM command handling such as history recall or prompt rendering. Token totals reset on `/new` or when session identity is rotated. They do **not** carry across `/resume`. See [Session History](/docs/sessions). ### `/context` Displays key/value infra metadata collected during investigations and chat — typically `service`, `cluster`, `region`, or similar fields extracted from alert text and investigation state. ```text theme={null} /context ``` Context is **inherited** across investigations in the same session (and restored by `/resume`). It is passed as overrides to subsequent `/investigate` runs so the pipeline does not re-ask for environment details you already established. ### `/effort` Set reasoning depth for **OpenAI and Codex** providers in this REPL session only. ```text theme={null} /effort high /effort ``` | Level | When to use | | -------------- | ----------------------------------------------------------------------------------------------- | | `low` | Fast triage, simple lookups, lower token cost | | `medium` | Default balance for most incident work | | `high` | Deeper RCA when latency is acceptable | | `xhigh`, `max` | Maximum reasoning; best with newer GPT-5 or Codex models — older models may reject these levels | Bare `/effort` prints the current level, the config default for your provider/model, and supported choices. `/status` includes the same field. Other providers (Anthropic, Ollama, etc.) ignore `/effort`; the shell prints a hint suggesting `/model set openai` or `/model set codex`. Set `OPENSRE_REASONING_EFFORT` in the environment for non-interactive defaults. See [LLM providers](/docs/llm-providers). ### `/trust` Trust mode skips execution confirmation prompts for **elevated** commands (`/save`, `/watch`, `/cancel`, `/uninstall`, integration remove, etc.). ```text theme={null} /trust on /trust off /trust ``` In a TTY, bare `/trust` opens an interactive on/off menu. Trust mode is a **session preference** — it is not restored by `/resume`. During watchdog demos or repeated `/save` exports, `/trust on` avoids confirmation fatigue. Turn it off before running destructive commands you might mistype. ### `/verbose` Toggle `TRACER_VERBOSE` logging for the REPL process (deeper internal logs to stderr). ```text theme={null} /verbose on /verbose off /verbose ``` ### `/clear` Clears the terminal and re-renders the OpenSRE banner. Does **not** reset session state, token usage, LLM conversation context, or accumulated infra context. ```text theme={null} /clear ``` ### `/compact` Summarizes older conversation context, keeps recent messages, and persists a `compaction` entry in the session file. Future `/resume` calls replay the summary before the kept messages. ```text theme={null} /compact ``` If there are not enough messages to compact, the command reports nothing to compact. OpenSRE also compacts automatically before a shell turn when the replayed branch context exceeds the runtime threshold. ### `/sessions` List up to **20** recent sessions stored on disk, newest first: ```text theme={null} /sessions ``` ```text theme={null} Recent sessions # Session ID Name Started Duration Turns Investigations ───────────────────────────────────────────────────────────────────────────────────────────── 1 3f8a1c2d (current) Jun 17 10:00 42m 8 2 2 9b2e4f7a why is CPU spiking on prod-api Jun 16 14:30 1h 5m 15 4 ``` | Column | Meaning | | ---------------- | ------------------------------------------------------------ | | `Session ID` | First 8 characters of the UUID (enough for `/resume`) | | `Name` | Derived from the first user message or resumed session label | | `Turns` | Total chat + slash + alert turns recorded | | `Investigations` | Count of investigation turns | The current row updates **duration** live. Sessions with no name show `(current)` or `↩ ` when applicable. ### `/resume` Restore LLM conversation context and accumulated infra context from a previous session. ```text theme={null} /resume 9b2e4f7a /resume 9b2e4f7a:abc123 /resume redis /resume ``` | Form | Behavior | | ----------------- | ------------------------------------------------------------------------------- | | ID prefix | Match session UUID by prefix (must be unique) | | ID + entry prefix | Replay the branch ending at the matching entry ID prefix | | Name substring | Match when ≥3 characters and exactly one session matches (e.g. `/resume redis`) | | Bare `/resume` | Interactive numbered picker of recent sessions (TTY) | **Current session:** If the ID prefix matches the session you are already in, `/resume` is a no-op — OpenSRE prints a hint to pick a previous session from `/sessions`. When you resume a **different** session, OpenSRE: 1. Switches the active session file to the target session 2. Restores `cli_agent_messages` so the assistant remembers prior turns 3. Restores `accumulated_context` keys 4. Reprints conversation history (user prompts, assistant replies, slash commands) Compaction entries are replayed as summary messages before the kept branch messages. **Restored vs not restored:** | | `/resume` | | ------------------------------ | ---------------------------------------------------- | | Conversation context | Yes | | Infra context keys | Yes | | Trust mode, reasoning effort | No — per-session preferences | | Token usage / `/cost` counters | No — fresh counters for the resumed session identity | | In-memory history count | Yes — turn stubs from the saved session | **Warning:** Resuming a different session replaces the current session's LLM context if messages already exist. Session replay reads the current version-2 session tree format. Older session files are ignored by `/sessions` and `/resume`. See [Session History](/docs/sessions). ### `/new` Rotate to a **new session file** while keeping the current LLM conversation thread and accumulated context. ```text theme={null} /new ``` ```text theme={null} new session started — conversation context carried forward. 14 messages in context · type to continue ``` | Command | Session file | LLM context | Token counters | | --------- | -------------- | -------------------- | ---------------------- | | `/clear` | Same | Kept | Kept | | `/new` | New UUID | Kept | Reset | | `/resume` | Target session | Replaced with target | Reset for that session | Use `/new` after a long `/resume` so `/sessions` stays tidy without losing your place in the conversation. ### Exit paths `/exit`, `/quit`, double **Ctrl+C**, or **Ctrl+D** (empty prompt) all print: ```text theme={null} Resume this session with: /resume 3f8a1c2d opensre --resume 3f8a1c2d goodbye. ``` *** ## Investigation commands Three ways to start RCA from the REPL: | Method | Example | Best for | | -------------------------- | -------------------------------- | ---------------------------- | | Plain language | `checkout returns 502 on prod` | Quick starts, pasted context | | Slash + file/template | `/investigate alert.json` | Repeatable runs, CI fixtures | | Slash + interactive picker | `/investigate` → choose template | Demos, first-time users | Investigations inherit [accumulated context](#context) from earlier runs in the same session. ### `/investigate` ```text theme={null} /investigate alert.json /investigate generic /investigate sample:datadog /investigate ./alerts/checkout-502.json /investigate ``` | Target | Behavior | | ------------------------------------------------------------------- | -------------------------------------------------- | | `alert.json` | Bundled demo alert file shipped with OpenSRE | | `generic`, `datadog`, `grafana`, `honeycomb`, `coralogix`, `splunk` | Built-in sample templates (runs immediately) | | `sample:` or `template:` | Explicit template prefix | | File path | Read alert text from disk (`.json`, `.md`, `.txt`) | | Bare `/investigate` | Interactive picker in a TTY | Template names win over same-named files in the working directory. Force file mode: `/investigate ./generic` **During a run:** * Output streams to the terminal like chat * **Ctrl+C** cancels and marks the task cancelled * `/tasks` shows the investigation task id and status * On completion, `last_state` is updated for `/last` and `/save` * Infra fields from the result merge into accumulated context **Errors:** Missing files, unreadable paths, and pipeline failures print actionable messages; failed runs do not update `last_state`. ### `/template` Print starter alert JSON to stdout — copy, edit, save, then `/investigate your-file.json`. ```text theme={null} /template generic /template datadog /template ``` | Template | Typical use | | ----------- | ----------------------------- | | `generic` | Minimal portable alert shape | | `datadog` | Datadog monitor-style payload | | `grafana` | Grafana alerting format | | `honeycomb` | Honeycomb trigger shape | | `coralogix` | Coralogix alert JSON | | `splunk` | Splunk notable-event style | ### `/last` Reprint the root cause and report sections from the most recent successful investigation in **this session**. ```text theme={null} /last ``` Sections rendered when present: **Root Cause**, **Report** (from `problem_md` or `slack_message`). If no investigation ran yet, prints a dim empty-state message. ### `/save` Write the last investigation to disk. Requires confirmation unless trust mode is on. ```text theme={null} /save report.md /save out.json /save ./rca/checkout-502.json ``` | Extension | Output | | ------------------------ | ------------------------------------------------------ | | `.json` | Full investigation state object (machine-readable) | | Other (`.md`, `.txt`, …) | Markdown with `## Root Cause` and `## Report` sections | Parent directories must exist or be creatable; write failures print the underlying error. *** ## Integrations, models, and tools ### Choosing the right diagnostic command | Question | Command | | --------------------------------------------- | -------------------------------------------------------------- | | Are my integrations configured and reachable? | `/verify`, `/health`, or `/integrations verify` | | Verify one integration by name? | `/verify datadog` or `/integrations verify datadog` | | Show one integration's credentials/endpoints? | `/integrations show datadog` | | Is Python/Docker/venv OK on this machine? | `/doctor` | | What integrations exist in OpenSRE generally? | Ask in plain language (docs answer) — not `/integrations list` | ### `/health` Read-only pass/fail report for the local OpenSRE agent, LLM connectivity, and each configured integration. ```text theme={null} /health ``` Runs live verification against the integration store. Use before incidents to confirm Datadog/Grafana/K8s credentials still work. For integration-only checks without the full agent/LLM report, prefer `/verify`. ### `/verify` Shortcut for integration connectivity checks (same as `opensre integrations verify` / `make verify-integrations`). ```text theme={null} /verify /verify datadog /verify telegram ``` | Form | Behavior | | ------------------- | ------------------------------------------------------------------------------------------------------------ | | Bare `/verify` | Verify all integrations; prints status table plus `all integrations ok` or `N integration(s) need attention` | | `/verify ` | Verify one named integration | ### `/integrations` ```text theme={null} /integrations /integrations list /integrations verify /integrations verify datadog /integrations show datadog /integrations setup datadog /integrations remove datadog ``` | Subcommand | Behavior | | ------------------ | -------------------------------------------------------------- | | `list` (default) | Verify all configured integrations and render status table | | `verify` | Verify all integrations; prints summary line | | `verify ` | Verify one integration (same as `/verify `) | | `show ` | Verify one service and print key/value config (masked secrets) | | `setup ` | Launch setup wizard via CLI subprocess | | `remove ` | Remove from local store **elevated** | Bare `/integrations` opens an interactive menu in a TTY with service pickers for show/remove. ### `/mcp` MCP-capable integrations only (subset of the full integration catalog). ```text theme={null} /mcp list /mcp connect github /mcp disconnect github ``` | Subcommand | Maps to | | --------------------- | ----------------------------------------------- | | `list` | Table of MCP servers from verified integrations | | `connect ` | `opensre integrations setup ` | | `disconnect ` | `opensre integrations remove ` | ### `/model` Show or change LLM provider and models. Updates project `.env` (default: repository root `.env`, override with `OPENSRE_PROJECT_ENV_PATH`), `~/.opensre/opensre.json`, and resets in-process LLM caches. ```text theme={null} /model show /model set openai gpt-4.1 /model set anthropic claude-sonnet-4-20250514 --toolcall-model claude-sonnet-4-20250514 /model set claude-sonnet-4-20250514 /model restore /model restore anthropic /model toolcall set gpt-4.1-mini /model ``` | Subcommand | Behavior | | ----------------------------------------------- | --------------------------------------------------------- | | `show` (default) | Provider, reasoning model env var, toolcall model env var | | `set [model] [--toolcall-model ]` | Switch provider; optional per-slot models | | `set ` (no provider) | Update reasoning model for **current** provider only | | `restore [provider]` | Reset to provider's default reasoning model | | `toolcall set ` | Set investigation tool-call model for active provider | **Credential guard:** Switching to a provider without prompt-safe auth status fails fast with setup instructions. Run `/auth login ` or export the provider API key before switching. If status is stale, run `/auth verify `. **Reasoning vs toolcall model:** Reasoning model drives chat and planning; toolcall model drives investigation tool invocation when the provider exposes a separate slot (common on Anthropic/OpenAI). Interactive `/model` menu flow: pick provider → reasoning model (or default) → optional toolcall model (`keep`, `match-reasoning`, or explicit). See [LLM providers](/docs/llm-providers). ### `/tools` List tools available to investigation and chat surfaces in this build (name, source, surfaces). ```text theme={null} /tools /tools list ``` There is no global `/list` command — use domain-specific list commands (see [Natural language actions](#natural-language-actions)). *** ## Privacy and history Command **history** (up-arrow recall) is separate from **session** history (`/sessions`). Full redaction patterns and env vars: [Interactive Shell Privacy](/docs/interactive-shell-privacy). ### `/history` ```text theme={null} /history /history clear /history off /history on /history retention 1000 ``` | Subcommand | Behavior | | --------------- | ----------------------------------------------------------------------------- | | (none) | Numbered list of persisted prompt lines | | `clear` | Delete `~/.opensre/interactive_history`; up-arrow recall empty on next launch | | `off` | Pause disk writes for this session (in-memory recall still works) | | `on` | Resume persistence | | `retention ` | Cap file entries; prunes immediately (requires redacting backend) | Bare `/history` opens an interactive menu in a TTY (presets: 100, 500, 1000, 5000 for retention). Redaction applies to the **history file**, not necessarily to what is sent to the LLM. Treat shared machines accordingly — run `/history clear` after sensitive sessions. ### `/privacy` ```text theme={null} /privacy ``` Shows persistence on/off, redaction on/off, retention cap, history file path, built-in pattern count, and a short threat-model reminder (unencrypted local disk). *** ## Tasks and background work Long-running work is tracked in a per-session task registry surfaced by `/tasks`. Human todos and reminders are tracked separately by `/work`. ### Task kinds | Kind | Source | | ---------------- | ------------------------------------------------- | | `investigation` | `/investigate`, streamed free-text investigations | | `watchdog` | `/watch` | | `synthetic_test` | `/tests synthetic`, CloudOpsBench | | `cli_command` | Other `/tests` runs, delegated CLI subprocesses | | `code_agent` | Code-agent integrations (when used) | ### Task statuses | Status | Meaning | | ----------- | ------------------------------- | | `running` | In progress; `/cancel` eligible | | `completed` | Finished successfully | | `cancelled` | User or `/cancel` stopped it | | `failed` | Non-zero exit or pipeline error | | `pending` | Created but not yet started | ### `/tasks` ```text theme={null} /tasks ``` Example: ```text theme={null} Tasks id kind status started (UTC) duration detail abc12 investigation running 2026-06-17 10:01 45.2s streaming… def34 watchdog completed 2026-06-17 09:55 120.0s pid=12345 max_cpu=80% ``` Shows up to **50** recent tasks, newest first. Use the **id** column (short prefix) with `/cancel` or `/unwatch`. ### `/cancel` ```text theme={null} /cancel abc12 ``` * Matches task id by **prefix** (must be unique among active/recent tasks) * Only **running** tasks accept cancellation * Investigations: signals cancel; press **Ctrl+C** if streaming continues * Watchdogs: prefer `/unwatch` for watchdog-specific messaging ### `/stop` Prints guidance only — does not kill processes: ```text theme={null} /stop ``` | Situation | Action | | --------------------------- | --------------------------------- | | Streaming investigation | **Ctrl+C** | | Background test or CLI task | `/tasks` → `/cancel ` | | Watchdog | `/unwatch ` or `/cancel ` | ### `/background` Session-local async investigation mode. Launch an RCA, keep using the shell, and get the finished report delivered to email, Telegram, Rocket.Chat, or Buzz. | Subcommand | What it does | | ----------------------------------------------- | -------------------------------------------------------------------------- | | `/background on` \| `off` | Enable or disable async investigation launches | | `/background status` | Mode, tracked job count, and active notify channels | | `/background list` | Tracked jobs with status and root cause | | `/background show ` | Full RCA summary plus per-channel notification result | | `/background use ` | Promote a completed RCA into active follow-up context | | `/background notify list` | Show the current completion channels | | `/background notify set ` | Set channels — `email`, `telegram`, `rocketchat`, `buzz`, or a combination | ```text theme={null} /background on /background notify set email,telegram /investigate datadog /background show ``` Channels are **off by default** and reset on REPL restart. `email` requires the [`smtp`](/docs/smtp) integration; `telegram` requires [`telegram`](/docs/messaging/telegram); `rocketchat` requires [`rocketchat`](/docs/messaging/rocketchat); `buzz` requires [`buzz`](/docs/messaging/buzz). Interactive shell only — there is no `opensre background …` CLI command. Full guide: [Background investigations](/docs/background-investigations). *** ## Watchdog commands Monitor a local process and send **Telegram** alarms when CPU, memory, or runtime thresholds breach. Configure Telegram credentials before use (via `/onboard` or env — see messaging docs). ### `/watch` ```text theme={null} /watch 12345 --max-cpu 80 --max-rss 512M --max-runtime 30m --cooldown 5m --interval 2s --once ``` | Flag | Meaning | Default | | --------------- | ----------------------------------------------- | ---------- | | `` | Process to watch (required first arg) | — | | `--max-cpu` | CPU percent threshold (max ≈ `100 × cpu_count`) | none | | `--max-rss` | Resident memory cap (`512M`, `1G`, `1.5Gib`) | none | | `--max-runtime` | Wall-clock limit (`30s`, `5m`, `1h`) | none | | `--cooldown` | Min seconds between repeat alarms | `300` (5m) | | `--interval` | Sample period | `2s` | | `--once` | At most one alarm per threshold type | off | On start: ```text theme={null} task abc12 started. ``` When a threshold fires (Telegram configured): ```text theme={null} [task abc12] alarm fired: max_cpu … (telegram delivered) ``` **Typical demo workflow:** 1. `/trust on` (optional — skips `/watch` confirmation) 2. `/watch --max-cpu 80` — use a real PID (e.g. the REPL's Python process) 3. `/watches` — confirm `running` status and threshold string 4. `/unwatch abc12` — request stop; `/watches` should show `cancelled` Quoted values are supported: `/watch 12345 --max-cpu "80"`. ### `/watches` ```text theme={null} /watches ``` Watchdog-only view with columns: id, pid, status, started, thresholds (from command summary), **last sample** (live CPU/RSS from progress line). ### `/unwatch` ```text theme={null} /unwatch abc12 ``` Cancels a **watchdog** task by task id. Using `/cancel` also works; `/unwatch` validates the task kind. ### `/watchdog` CLI-parity syntax (subprocess to `opensre watchdog`): ```text theme={null} /watchdog --pid 12345 --max-rss 1G --max-cpu 80 ``` Prefer `/watch` inside the REPL — it registers tasks for `/watches` and `/tasks` automatically. *** ## Agents and alerts Fleet coordination is documented further on [Agents](/docs/fleet). Register discovered agents with `opensre fleet scan --register` before `/fleet trace` targets them. ### `/fleet` ```text theme={null} /fleet /fleet budget /fleet budget cursor-agent 5.00 /fleet bus /fleet claim feature-x my-agent /fleet release feature-x /fleet conflicts /fleet kill 12345 /fleet kill 12345 --force /fleet trace 12345 /fleet wait 12345 --on 67890 /fleet graph ``` | Subcommand | Behavior | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | (none) | Dashboard: registered + discovered agents (Cursor, Claude Code, Codex, Aider, …) with pid, uptime, CPU, tokens/min, \$/hr, status | | `budget` | View hourly budgets from `~/.opensre/agents.yaml` | | `budget ` | Set `hourly_budget_usd` for one agent | | `bus` | Live-tail cross-agent context bus until Ctrl+C | | `claim ` | Exclusive branch claim (agent must be in registry) | | `release ` | Release a claim | | `conflicts` | File-write conflict report between agents | | `kill [--force]` | SIGTERM → wait → SIGKILL **elevated**; refuses self-PID | | `trace ` | Live stdout tail of agent process | | `wait --on ` | Record wait dependency for graph | | `graph` | Tree of wait-on relationships | **Kill confirmation:** Without `--force`, prompts `[y/N]`. `--force` skips prompt (still elevated tier unless trust mode). ### `/alerts` ```text theme={null} /alerts ``` When the alert listener is active: | Field | Meaning | | ------------- | ----------------------------------- | | `status` | `listening` | | `queue depth` | Alerts waiting for REPL consumption | | `dropped` | Alerts dropped when queue was full | | `recent` | Up to 5 latest alert names/snippets | If the listener is inactive, prints a warning. Incoming alerts also appear in `/status` as `incoming alerts`. Post alerts to your configured webhook/listener URL during setup; the REPL can start investigations from plain language. *** ## CLI parity commands These spawn `opensre …` subprocesses. Output is captured into the REPL buffer for non-interactive subcommands; wizards attach to the real TTY. ### `/onboard` ```text theme={null} /onboard /onboard local_llm ``` First-run setup: LLM keys, integrations, Telegram, alert listener. Requires exclusive stdin — the REPL tears down prompt\_toolkit before the wizard runs. ### `/remote` ```text theme={null} /remote health /remote investigate /remote ops /remote pull /remote trigger ``` Operate remote deployed OpenSRE agents (EC2, Nitro, hosted runtime). See [Remote runtime investigation](/docs/remote-runtime-investigation). ### `/config` ```text theme={null} /config show /config set interactive.history.max_entries 1000 ``` Read/write `~/.opensre/config.yml`. Prefer env vars for secrets; use config for structured interactive-shell settings. ### `/cron` ```text theme={null} /cron list /cron add /cron remove /cron run /cron logs ``` Scheduled investigation or report delivery. See [Cron](/docs/cron). ### `/loops` ```text theme={null} /loops /loops active /loops all /loops add --name "Morning ops" --time 08:30 --prompt "Check open incidents and summarize production risk" --run-now /loops run /loops stop /loops start /loops delete /loops next /loops messages ``` List recurring loops from the scheduler store, create manual prompt loops, run a loop once immediately, stop or restart a loop, delete a loop, and debug the next fire time. Onboarding creates draft examples such as a weekday morning report; active loops run through the cron scheduler when the gateway or `opensre cron start` is running. You can also ask naturally: ```text theme={null} Set up a manual loop called Morning ops at 08:30 UTC to check open incidents and summarize production risk, and run it once now. ``` By default, new loops send to every configured default handle OpenSRE can reach: Telegram, Slack, and the local interactive-shell inbox. ### `/sentry` ```text theme={null} /sentry digest run /sentry digest schedule list /sentry digest schedule add --cron "0 8 * * *" --provider telegram --chat-id /sentry digest schedule run /sentry digest schedule remove /sentry uptime check /sentry uptime watch list /sentry uptime watch add --cron "*/5 * * * *" --provider telegram --chat-id /sentry uptime watch run /sentry uptime watch remove ``` Automated Sentry morning digest delivery (#10) and uptime watch notifications (#4032). Requires Sentry plus Telegram or Slack configured for the chosen provider. See [Sentry](/docs/sentry). ### `/messaging` ```text theme={null} /messaging pair /messaging allow /messaging revoke /messaging status ``` Telegram bot pairing and sender allowlist — required for `/watch` alarms and some delivery features. ### `/hermes` ```text theme={null} /hermes watch ``` Tail Hermes logs and escalate classified incidents. See [Hermes](/docs/hermes) and [Hermes runbook](/docs/hermes_runbook). ### `/guardrails` ```text theme={null} /guardrails audit /guardrails init /guardrails rules /guardrails test ``` | Subcommand | Purpose | | ---------- | ------------------------------------------ | | `init` | Scaffold local guardrail config | | `rules` | List active masking rules | | `test` | Run sample text through rules | | `audit` | Scan recent content for sensitive patterns | Related: [Masking](/docs/masking). ### `/tests` ```text theme={null} /tests /tests list /tests run /tests synthetic /tests cloudopsbench /tests synthetic --scenario ``` | Form | Behavior | | ----------------------------------- | --------------------------------------------------------------- | | Bare `/tests` | Interactive multi-select picker; chosen tests run in background | | `list` | Print inventoried tests (captured output) | | `run`, `synthetic`, `cloudopsbench` | Background task — monitor with `/tasks` | | Flags after subcommand | Passed through to CLI | Synthetic tests can run for a long time; default timeout is generous — use `/cancel` to stop. ### `/update` ```text theme={null} /update ``` Checks PyPI and upgrades OpenSRE in-place (5-minute network timeout). Non-zero exit prints CLI error code. ### `/debug` ```text theme={null} /debug sentry ``` Targeted smoke tests (Sentry, etc.) — subcommands match `opensre debug --help`. ### `/uninstall` ```text theme={null} /uninstall ``` Removes OpenSRE and local data. **Destructive** — confirmation required unless trust mode. Delegates to interactive CLI uninstall flow. *** ## System commands ### `/doctor` Environment diagnostic distinct from `/health`: ```text theme={null} /doctor ``` Checks Python version, venv, config paths, Docker availability, credential presence, and related local prerequisites. Each row: `ok`, `warn`, or `error` with detail text. Use **`/doctor`** before first install debugging; use **`/health`** before an incident to verify integrations. ### `/version` ```text theme={null} /version ``` | Field | Example | | --------- | ---------------------------- | | `opensre` | Package version from install | | `python` | `3.12.x` | | `os` | `darwin (arm64)` | ### `/exit` and `/quit` ```text theme={null} /exit /quit ``` Clean shutdown with `/resume` hint. Prefer over killing the terminal so session files flush cleanly. *** ## Trust mode and confirmations | Tier | Examples | Confirmation | | -------- | --------------------------------------------------------------------------------- | ------------------------------ | | Exempt | `/help`, `/exit`, `/trust` | Never prompts | | Safe | `/status`, `/health`, `/investigate`, `/integrations list` | Read-only or standard risk | | Elevated | `/save`, `/watch`, `/cancel`, `/integrations remove`, `/fleet kill`, `/uninstall` | Prompt `[y/N]` unless trust on | Non-TTY: elevated commands **fail closed** (error message, no side effect). ```text theme={null} /trust on # skip prompts for this session /trust off # restore prompts ``` *** ## Natural language actions You do not need to memorize every slash command. Examples: | You type | Typical action | | ------------------------------------- | ----------------------------------------- | | "what's my token usage?" | `/cost` | | "verify datadog" | `/verify datadog` | | "show datadog config" | `/integrations show datadog` | | "switch to openai gpt-4.1" | `/model set openai gpt-4.1` | | "run the generic sample alert" | `/investigate generic` | | "list my past sessions" | `/sessions` | | "check health then list integrations" | `/health` then `/integrations list` | | "how do I configure Datadog?" | Doc-grounded answer (no mutating command) | **List intents** — there is no global `/list`: | Intent | Command | | ------------------------- | ------------------------ | | Connected integrations | `/integrations list` | | Tools | `/tools` | | Background tasks | `/tasks` | | Human todos and reminders | `/work` | | MCP servers | `/mcp list` | | Cron jobs | `/loops` or `/cron list` | | Past REPL sessions | `/sessions` | | Watchdog tasks | `/watches` | When the planner is uncertain, it asks for clarification or falls back to help — it does not silently run elevated commands. Compound requests ("health then integrations") execute as an ordered sequence of slash actions. *** ## Common workflows ### First-time setup ```text theme={null} /onboard /health /verify /model show ``` ### Incident triage (local) ```text theme={null} /status /alerts /health ``` Paste alert text or: ```text theme={null} /investigate ./alerts/prod-502.json /last /save ./rca/prod-502.md ``` ### Pick up yesterday's thread ```text theme={null} /sessions /resume 9b2e4f7a ``` Continue chatting, then optionally: ```text theme={null} /new ``` ### Switch model mid-session ```text theme={null} /model set anthropic claude-sonnet-4-20250514 /effort high /cost ``` ### Monitor a runaway process ```text theme={null} /watch --max-cpu 90 --max-rss 2G --cooldown 10m /watches /unwatch ``` ### Debug integration failures ```text theme={null} /integrations show datadog /doctor /verbose on ``` *** ## Troubleshooting | Symptom | Try | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `unknown command` | `/help`; check spelling; use suggested correction | | Elevated command blocked in CI | Use CLI equivalent (`opensre investigate …`) or run interactively | | `/cost` shows zero | No LLM turns yet this session; chat once and retry | | `/resume` not found | Run `/sessions`; use longer ID prefix | | `/resume` ambiguous prefix | Add more characters from Session ID column | | `/model set` missing credential | `/auth login ` or export `ANTHROPIC_API_KEY` / etc.; use `/auth verify ` for stale metadata | | `/watch` Telegram error | Configure messaging via `/messaging status` | | `/integrations show` not found | Run `/integrations list` for exact service slug | | History pause not working | Requires redacting backend — see `/privacy` | | Garbage in next prompt after table | Known TTY issue — upgrade OpenSRE; report if persistent | *** ## Related docs * [Session History](/docs/sessions) — persistence format, privacy, `/new` vs `/clear` * [Interactive Shell Privacy](/docs/interactive-shell-privacy) — redaction, env vars, threat model * [Investigation overview](/docs/investigation-overview) — RCA workflow, plain-language actions, CLI investigate * [LLM providers](/docs/llm-providers) — `/model` and `/effort` * [Agents](/docs/fleet) — fleet dashboard, bus, trace, budgets * [Cron](/docs/cron) — `/cron` scheduled deliveries * [Hermes](/docs/hermes) — `/hermes watch` * [Remote runtime investigation](/docs/remote-runtime-investigation) — `/remote` # Interactive Shell Privacy Source: https://opensre.com/docs/interactive-shell-privacy How OpenSRE redacts secrets from your command history and LLM prompt/response logs, and the controls you have over persistence and cloud telemetry ## Overview The OpenSRE interactive shell persists every line you type to a history file so up-arrow recall and `/history` work across sessions, and separately records each LLM prompt/response turn for local debugging and `/resume`. Incident prompts can include sensitive identifiers and tokens, so the shell: * redacts known token shapes before each entry is written to disk * supports disabling persistence entirely (memory-only mode) * caps how many entries are kept (oldest pruned) * offers a one-shot `/history clear` to wipe the file on demand The history file lives at `~/.opensre/interactive_history`. See [Prompt and response logging](#prompt-and-response-logging) below for the separate LLM turn log and its PostHog forwarding behavior. ## Defaults | Setting | Default | Effect | | ------------- | ---------------- | ------------------------------------------------------------------------ | | Persistence | **on** | Lines you type are appended to the history file. | | Redaction | **on** | Known token shapes are replaced with `[REDACTED:]` before writing. | | Retention cap | **5000 entries** | Older entries are pruned when the cap is exceeded. | ## Redaction patterns The built-in pattern set targets token shapes that are unique enough to keep false positives on natural-language incident text very low. Each match is replaced with a labeled placeholder. | Kind | Examples | | --------------- | ---------------------------- | | `aws_key` | `AKIA…`, `ASIA…` | | `aws_secret` | `aws_secret_access_key=…` | | `github_pat` | `ghp_…`, `github_pat_…` | | `anthropic_key` | `sk-ant-…` | | `openai_key` | `sk-…` | | `slack_token` | `xoxb-…`, `xoxp-…`, `xoxa-…` | | `stripe_key` | `sk_live_…`, `sk_test_…` | | `bearer` | `Bearer ` headers | | `jwt` | `eyJ…` three-segment tokens | | `password` | `--password=…`, `password=…` | | `private_key` | PEM-encoded private keys | Redaction applies only to **persistent history**. The line you typed is still passed to OpenSRE's normal pipeline as you typed it. ## Slash commands | Command | Effect | | ------------------------ | ---------------------------------------------------------------------------- | | `/history` | Show all persisted entries. | | `/history clear` | Wipe the history file. Up-arrow recall resets on next launch. | | `/history off` | Pause persistence for this session. New entries are not written. | | `/history on` | Resume persistence for this session. | | `/history retention ` | Keep at most N entries on disk. Prunes immediately. | | `/privacy` | Show current persistence + redaction state, retention cap, and threat model. | ## Configuration Settings resolve from (highest wins): 1. Environment variables 2. The `interactive.history` block in `~/.opensre/config.yml` 3. Built-in defaults ### Environment variables | Variable | Default | Effect | | ----------------------------- | ------- | ----------------------------------------------------------------------- | | `OPENSRE_HISTORY_ENABLED` | `1` | Set to `0`/`false`/`off` to skip persistence entirely (in-memory only). | | `OPENSRE_HISTORY_REDACT` | `1` | Set to `0`/`false`/`off` to disable redaction (raw `FileHistory`). | | `OPENSRE_HISTORY_MAX_ENTRIES` | `5000` | Non-negative integer. `0` disables the cap (unlimited). | ### Config file ```yaml theme={null} interactive: history: enabled: true redact: true max_entries: 5000 ``` ## Prompt and response logging Separately from typed-command history, the interactive shell records each LLM turn — the full prompt sent and the full response received — for chat and follow-up routes. This log is richer than command history (it includes model output, not just what you typed) and is used for two purposes: 1. **Local debugging / `/resume`**: appended as JSON Lines to `~/.opensre/prompt_log.jsonl`, and folded into the session file so `/resume` can restore conversation context. 2. **Product analytics**: forwarded to PostHog as an `$ai_generation` event (model, provider, latency, token counts, and the prompt/response text) so we can track usage and quality of the AI features. Investigation turns additionally include the model/provider/token usage of the investigation run and an `investigation_id` join key; failed turns carry structured error properties (`$ai_is_error`, `$ai_error`, `error_kind`). Turns handled entirely by terminal tools or slash commands report `$ai_model` / `$ai_provider` as `no_conversational_agent`; when a conversational reply was intended but the LLM provider failed (missing API key, model access, quota, auth), the event instead reports the attempted model (or `unknown`) plus an `ai_error_kind` bucket (`not_configured`, `quota`, `auth`, or `provider_error`). ### Defaults | Setting | Default | Effect | | ------------------ | ------- | ------------------------------------------------------------------------------------------------------------------- | | Logging | **on** | Each LLM turn is recorded. | | Local JSONL file | **on** | Turns are appended to `~/.opensre/prompt_log.jsonl`. | | PostHog forwarding | **on** | Turns are also sent as a PostHog `$ai_generation` event. | | Redaction | **on** | Known token shapes (same patterns as command history) are stripped from the prompt and response before either sink. | ### Environment variables | Variable | Default | Effect | | ----------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------- | | `OPENSRE_PROMPT_LOG_DISABLED` | `0` | Set to `1` to disable prompt/response logging entirely (both local file and PostHog). | | `OPENSRE_PROMPT_LOG_LOCAL_DISABLED` | `0` | Set to `1` to skip the local JSONL file while leaving PostHog forwarding (if enabled) unaffected. | | `OPENSRE_PROMPT_LOG_REDACT` | `1` | Set to `0` to log/send raw prompt and response text without redaction. | | `OPENSRE_PROMPT_LOG_PATH` | `~/.opensre/prompt_log.jsonl` | Override the local JSONL file path. | The separate operations log at `~/.opensre/operations_log.jsonl` records agent and scheduler lifecycle breadcrumbs such as loop start, iteration outcome, loop creation, and delivery status. It does not store prompt or response bodies. PostHog forwarding for this event additionally honors the global telemetry opt-outs: set **`OPENSRE_NO_TELEMETRY=1`**, `OPENSRE_ANALYTICS_DISABLED=1`, or `DO_NOT_TRACK=1` to stop all PostHog traffic (including `$ai_generation`) without touching the local JSONL file. See [Environment Variables](/docs/configuration/environment-variables#telemetry-monitoring). ### Config file ```yaml theme={null} interactive: prompt_log: posthog_enabled: true redact: true max_chars: 32000 path: ~/.opensre/prompt_log.jsonl ``` Redaction here uses the same built-in pattern set as command history (see [Redaction patterns](#redaction-patterns) above) — it catches known secret shapes, not arbitrary sensitive content. Raw incident details, hostnames, or business context in a prompt are not redacted; only credential-shaped substring patterns are. ## Threat model The history file is **plain text on local disk** at `~/.opensre/interactive_history`, with the user's default file permissions. Built-in redaction targets common token shapes only — it is not a substitute for proper secret handling. Treat the file as confidential and be aware: * A determined attacker with read access to your home directory can still read pre-existing entries written before redaction was enabled. * Redaction cannot detect tokens that look like normal text (for example a natural-language password). Don't paste secrets you wouldn't be comfortable seeing in a system log. * Custom redaction patterns are not yet supported in v1. If you need to redact internal token shapes, use `/history off` for that session and run `/history clear` afterwards. The prompt/response log carries the same caveat, plus one more: with PostHog forwarding on (the default), redacted prompt/response text leaves your machine. If you discuss confidential systems or data in the interactive shell, set `OPENSRE_NO_TELEMETRY=1` (or `OPENSRE_PROMPT_LOG_DISABLED=1` to also stop the local file) rather than relying on redaction alone. For the strongest posture: set `OPENSRE_HISTORY_ENABLED=0` and `OPENSRE_NO_TELEMETRY=1`, accept the loss of cross-session up-arrow recall and `/resume` context, and rely on the in-memory ring instead. # Tracer Monitoring Source: https://opensre.com/docs/introduction-monitoring Tracer brings complete observability to scientific and high-compute workflows built for scientists and researchers. It automatically tracks performance, cost, and resources across every process, tool, and node. ## One line of code and Tracer is installed ```bash theme={null} curl -sSL https://install.tracer.cloud | sh -s ``` Once installed, Tracer begins tracing all jobs and processes in real time, providing complete visibility into performance and efficiency. Tracer Tool Visualiser ## Interesting links Get to know Tracer. Integrate Tracer. See instances. Local on mac. ## Mini History Tracer was founded in 2023 by Vincent Hus and Laura Bogaert to transform how scientists understand and manage computational workloads. Frustrated by legacy tools, broken pipelines, and infrastructure bottlenecks, they built the world's first observability platform purpose-built for scientific computing. # Investigations overview Source: https://opensre.com/docs/investigation-overview Start investigations from the interactive shell or CLI, review artifacts, and use plain-language REPL actions Use this workflow when running OpenSRE via the local `opensre` binary. You can work in two ways: * **Interactive prompt shell** — run `opensre` with no subcommand (TTY) to enter the REPL: describe incidents conversationally, stream investigations, and use slash commands. * **Direct investigation** — run `opensre investigate` from your terminal with `-i` pointing at an alert payload (or `--interactive` to pick a file in the UI). The process runs and exits when the investigation completes. ## 1) Start an investigation ### Interactive shell (`opensre`) From a terminal with stdin and stdout attached (`opensre` detects a TTY), run: ```bash theme={null} opensre ``` Then describe the incident or paste alert context at the prompt. Use `/help` for slash commands and `/exit` when finished. See [Interactive Shell Commands](/docs/interactive-shell-commands) for the full slash-command reference (`/cost`, `/status`, `/investigate`, and every other REPL command). When `LLM_PROVIDER` is `openai` or `codex`, `/effort` sets how much reasoning the model applies for that REPL session (`low`, `medium`, `high`, `xhigh`, or `max`). Run `/effort` with no arguments to print the current level and usage; `/status` includes the same field. Other providers ignore this setting (the shell prints a hint). You can also set `OPENSRE_REASONING_EFFORT` to `low`, `medium`, `high`, or `xhigh` in the environment for non-interactive defaults. ### Plain language and compound requests You do not need to memorize every slash command. Describe what you want in natural language and the REPL planner maps intent to the right actions — often a sequence of slash commands executed in order. Examples: | You type | Typical action | | ----------------------------------------------------------------- | --------------------------------------------------- | | "check the health of my opensre and then show connected services" | `/health`, then `/integrations list` | | "verify datadog" | `/verify datadog` or `/integrations verify datadog` | | "run the sample alert investigation" | `/investigate` with a built-in template | | "connect to my remote EC2 instance and send it hello world" | `/remote` subcommands, then a remote investigation | List-style questions map to **per-domain** commands (there is no global `/list`): | Intent | Command | | ---------------------- | -------------------- | | Connected integrations | `/integrations list` | | Investigation tools | `/tools` | | Background tasks | `/tasks` | | MCP servers | `/mcp list` | | Cron deliveries | `/cron list` | | Past REPL sessions | `/sessions` | For procedural questions ("how do I configure Datadog?"), the assistant answers from docs without running mutating commands unless you ask it to. ### Direct investigation (`opensre investigate`) Pass an alert payload to `opensre investigate`: ```bash theme={null} opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` You can also use `--interactive` to pick an input file from your terminal UI. For deployed services by name, see [Remote runtime investigation](/docs/remote-runtime-investigation). ## 2) Review investigation artifacts A local run produces structured RCA artifacts such as: * `problem.md` for incident framing and initial hypothesis * `theory/hypothesis_*.md` for each hypothesis tested during the run * `report.md` for final root-cause summary and next steps If you want a single machine-readable output file, pass: ```bash theme={null} opensre investigate -i --output ./rca.json ``` ## 3) Understand what OpenSRE analyzed Each run captures: * the original alert payload * extracted context and normalized evidence * tool outputs collected from connected integrations * final diagnosis and recommended remediation steps ## Session continuity REPL sessions are persisted under `~/.opensre/sessions/`. When you exit, the shell prints a `/resume ` hint so you can pick up later. See [Session history](/docs/sessions). ## Chat For local binary usage, the primary workflow is file-based (`problem.md`, `report.md`, and optional JSON output). You can open these artifacts in your editor and iterate from there (for example, by asking your editor's AI chat to drill into specific hypotheses or evidence sections). ## Slack reports If you've configured the Slack integration, OpenSRE can publish a concise incident summary into Slack after the local investigation completes. Slack Alert # Investigation pipeline architecture Source: https://opensre.com/docs/investigation-pipeline-architecture # Investigation pipeline architecture Contributor guide to how a single investigation runs end-to-end: the six-stage pipeline, the ReAct evidence-gathering loop, and the guardrails that keep it bounded. Companion to [`investigation-tool-calling.md`](investigation-tool-calling.md), which covers tool schema / LLM invoke payload mechanics specifically — this doc covers the pipeline and loop control flow around that. ## Where code lives | Concern | Location | | ------------------------------- | ----------------------------------------------------------------------------------------- | | Stage ordering | `tools/investigation/lifecycle.py` (`run_connected_investigation`) | | Public runner entrypoint | `tools/investigation/capability.py` | | Integration discovery | `tools/investigation/stages/resolve_integrations/node.py` | | Alert classification/extraction | `tools/investigation/stages/intake/node.py` | | Pre-loop tool planning | `tools/investigation/stages/plan_evidence/node.py`, `core/domain/alerts/tool_planning.py` | | ReAct loop (the agent) | `tools/investigation/stages/gather_evidence/{agent,loop,tools,prompt}.py` | | Diagnosis parsing | `tools/investigation/stages/diagnose/node.py`, `core/domain/diagnosis` | | Report delivery | `tools/investigation/reporting/` | | Shared state contract | `core/state/` (`AgentState`, `InvestigationState`, `EvidenceEntry`) | | Context budget enforcement | `core/context_budget.py` | ## Pipeline overview Each stage is a pure function — `(state) -> dict of updates`, merged into a shared `AgentState` via `apply_state_updates`. A stage exception is reported to Sentry and then re-raised; the pipeline never silently swallows a failure. ```mermaid theme={null} flowchart TD A[raw_alert] --> B[resolve_integrations] B --> C[extract_alert] C -->|is_noise = true| Z[Stop — no investigation] C -->|is_noise = false| D[plan_actions] D --> E["ReAct loop\n(ConnectedInvestigationAgent.run)"] E --> F[diagnose] F --> G[deliver] G --> H[Slack / GitLab / report.md] ``` ## Stage by stage ### 1. `resolve_integrations` — what tools exist Looks up which vendor integrations (Datadog, Grafana, EKS, …) this org has connected and credentialed. Not alert-specific — establishes the universe of tools everything downstream can draw from. ### 2. `extract_alert` — is this worth investigating One LLM call classifies the raw alert: noise (chat, greetings, replies in an existing thread) short-circuits the pipeline immediately with no tools run. A real alert gets structured fields extracted — `alert_name`, `severity`, `alert_source`, namespace, error message — plus a computed `incident_window`. ### 3. `plan_actions` — what to check first Scores every available tool against the alert (`score_tools`, source match + tool metadata) and keeps the top `tool_budget` (default 10) as `planned_actions`, with a written rationale. Advisory: if nothing scores confidently, the loop falls back to its own relevance ranking instead of an empty plan. ### 4. The ReAct loop — the core evidence-gathering agent `ConnectedInvestigationAgent.run()` in `tools/investigation/stages/gather_evidence/agent.py`. Before the model's first turn: the tool set is narrowed to a hard cap (`select_investigation_tools`, `MAX_AGENT_TOOL_SCHEMAS = 32`) using the plan from stage 3 if present, otherwise alert-source relevance ranking. A handful of "obviously needed" tools may fire as deterministic **seed calls** before the LLM gets a turn at all, so the loop starts with free evidence already in hand. ```mermaid theme={null} flowchart TD S0["Select ≤32 relevant tools\n(select_investigation_tools)"] --> S1[Build system prompt + alert context] S1 --> S2{Seed calls for\nthis alert_source?} S2 -->|yes| S3["Execute seed tool calls\nrecord as evidence"] S2 -->|no| S4 S3 --> S4["Loop: iteration 0..19\n(MAX_INVESTIGATION_LOOPS)"] S4 --> S5["Enforce context budget\n(evict/truncate low-value evidence)"] S5 --> S6[llm.invoke] S6 --> S7{Tool calls\nreturned?} S7 -->|no| S8{Accept\nconclusion?} S8 -->|yes| S9[Done — exit loop] S8 -->|no: nudge| S4 S7 -->|yes| S10{Identical to a\nprior call?} S10 -->|yes| S11["Replay cached result,\ntell LLM not to repeat"] S10 -->|no| S12[Execute tool, record evidence] S11 --> S13{Any fresh calls\nthis iteration?} S12 --> S13 S13 -->|yes| S4 S13 -->|no| S15["Send stagnation nudge\n(stagnant_iterations += 1)"] S15 --> S16{stagnant_iterations\n>= 2?} S16 -->|yes| S14["Strip tool access;\nloop back for one\nfinal llm.invoke"] S16 -->|no| S4 S14 --> S4 ``` Guardrails inside the loop: * **Duplicate detection** (`InvestigationToolCallCache`) — identical tool name + args is served from cache instead of re-executed, and the LLM is told explicitly it already has that result. * **Stagnation breaker** — any iteration where every tool call was a replayed duplicate (no fresh evidence) appends a nudge telling the model to stop repeating itself and try something different. After `MAX_STAGNANT_ITERATIONS = 2` such iterations in a row (two nudges), tool access is stripped on the next turn to force a text-only conclusion rather than burning the rest of the loop budget. * **CLI-backed models** (Codex, Claude Code CLI) use a subclass, `CLIBackedInvestigationAgent`, that overrides conclusion acceptance to refuse an early stop until every planned tool has been called — these models tend to write a final answer as soon as they see *some* results. * **LLM invoke failures** degrade to a partial "investigation failed" state (`degraded_investigation_from_llm_failure`) instead of crashing, preserving whatever evidence was already gathered. ### 5. `diagnose` — structure the conclusion The loop's final free-text answer is unstructured. A separate LLM call (structured output) parses it into `root_cause`, `root_cause_category`, `causal_chain`, `validated_claims` / `non_validated_claims`, `remediation_steps`, and a `validity_score`, with a legacy regex-based fallback (`parse_root_cause`) if structured parsing fails. ### 6. `deliver` — publish it Formats and ships the report to the destinations configured in `state` — Slack, GitLab writeback, local `report.md`, etc. See [`tools/investigation/reporting/`](https://github.com/Tracer-Cloud/opensre/tree/main/tools/investigation/reporting/). ## Guardrails at a glance | Guardrail | Constant | Defined in | Purpose | | ---------------------- | ------------------------------------ | ----------------------------------------------------- | ------------------------------------------------------------------------ | | Tool schema cap | `MAX_AGENT_TOOL_SCHEMAS = 32` | `tools/investigation/stages/gather_evidence/tools.py` | Bounds per-turn schema payload regardless of registry size. | | Secondary tool reserve | `MAX_SECONDARY_FALLBACK_TOOLS = 3` | `tools/investigation/stages/gather_evidence/tools.py` | Guarantees cheap reasoning/knowledge tools survive the cap. | | Loop iteration cap | `MAX_INVESTIGATION_LOOPS = 20` | `config/constants/investigation.py` | Worst-case runtime bound for the ReAct loop. | | Stagnation breaker | `MAX_STAGNANT_ITERATIONS = 2` | `tools/investigation/stages/gather_evidence/tools.py` | Stops the loop from spinning on duplicate-only iterations. | | Context budget | `context_budget_ceiling_for_model()` | `core/context_budget.py` | Evicts/truncates lowest-value evidence before the model's context limit. | | Pre-loop plan size | `tool_budget` (default 10) | `tools/investigation/stages/plan_evidence/node.py` | Shortlist size the plan hands the loop before it even starts. | ## Related docs * [`investigation-tool-calling.md`](investigation-tool-calling.md) — tool schema / LLM invoke payload mechanics, per provider. * [`AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/AGENTS.md) — "Changing the investigation pipeline" entry point and checklist for making changes here. # Investigation tool calling Source: https://opensre.com/docs/investigation-tool-calling Contributor guide for investigation ReAct tool schemas, LLM invoke payloads, and message shapes This page is a **contributor reference** for how OpenSRE shapes tool-calling during investigations. End users do not need to configure any of this — see [How an investigation works](/docs/how-investigations-work) and [Investigations overview](/docs/investigation-overview) instead. The investigation agent does **not** call integration APIs through the LLM directly. The flow is: 1. **Tools** — `get_registered_tools("investigation")`, filtered with `tool.is_available(...)`. 2. **Schemas** — `llm.tool_schemas(tools)` from the active provider client. 3. **Invoke** — `llm.invoke(messages, system=..., tools=tool_schemas)`; the model returns tool calls. 4. **Execute** — Tools run locally; results append as turns for the next invoke. 5. **Seed path** — `_build_seed_calls` may inject deterministic tool runs before the loop. ## Where code lives | Concern | Location | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | Provider routing | `core/llm/factory.py`, `core/llm/client_builders.py` | | Investigation loop | `tools/investigation/stages/gather_evidence/agent.py` | | Tool registry | `tools/` and `integrations//tools/` | | Pipeline stages | [Investigation pipeline architecture](https://github.com/Tracer-Cloud/opensre/blob/main/docs/investigation-pipeline-architecture.md) | ## Full reference The complete schema normalization rules, provider-specific adapters, and message-shape tables live in the repository source file: [docs/investigation-tool-calling.md](https://github.com/Tracer-Cloud/opensre/blob/main/docs/investigation-tool-calling.md) When changing investigation tools or LLM adapters, read that file and run the registry contract tests described in [adding tools and integrations](https://github.com/Tracer-Cloud/opensre/blob/main/docs/adding-tools-and-integrations.md). # Jenkins Source: https://opensre.com/docs/jenkins # Jenkins Integration Correlate failed builds and deployments with incidents. OpenSRE reads your [Jenkins](https://www.jenkins.io) server over its REST API to answer: *"was there a recent build or deployment that coincides with this alert?"* ## Commands | Command | What it does | | ------------------------------------- | ------------------------------------------------ | | `opensre integrations setup jenkins` | Store your Jenkins URL, username, and API token. | | `opensre integrations verify jenkins` | Check connectivity to the Jenkins server. | | `opensre integrations show jenkins` | Show the configured Jenkins connection. | | `opensre integrations remove jenkins` | Remove the stored Jenkins credentials. | ## What the agent can do During an investigation (or in chat), the agent can call these tools: * **`list_jenkins_builds`** — Recent builds for a job with status (SUCCESS / FAILURE / RUNNING / ABORTED) and timestamp. * **`get_jenkins_build_log`** — The console log for a specific build, to read the failing step or error. * **`get_jenkins_pipeline_stages`** — Per-stage status and duration for a Pipeline build (empty for freestyle jobs). * **`list_jenkins_jobs`** — All jobs with their last-build status. * **`list_jenkins_running_builds`** — Builds currently in progress across all jobs. ## Setup ### 1. Create a Jenkins API token In Jenkins: click your **username** (top-right) → **Security** → **API Token** → **Add new Token** → **Generate**. Copy the token — Jenkins shows it only once. API access uses HTTP Basic auth with your **username** and this **token** (not your password). ### 2. Configure credentials Either run the setup wizard: ```bash theme={null} opensre integrations setup jenkins ``` You'll be prompted for the Jenkins URL, username, and API token. Or use environment variables: ```bash theme={null} export JENKINS_URL="http://localhost:8080" export JENKINS_USER="your-username" # required — Basic auth is username:token export JENKINS_API_TOKEN="" ``` > Folder-organized jobs are supported — pass the full path, e.g. `team/payment-service`. ### 3. Verify connectivity ```bash theme={null} opensre integrations verify jenkins ``` A successful check reports the server it reached, e.g. `Jenkins connectivity successful at http://localhost:8080 (node: built-in)`. ## Example investigation Provide the affected job name so the agent can pull its recent builds and logs: ```bash theme={null} opensre investigate --input-json '{ "alert_name": "PaymentServiceErrors", "pipeline_name": "payment-service", "severity": "critical", "commonAnnotations": { "summary": "Error rate spiked shortly after a deploy", "job_name": "payment-service-deploy" } }' ``` The agent lists recent builds for the job, spots the failed one near the alert time, fetches its console log, and surfaces the failing step in the RCA. ## API reference | Purpose | Endpoint | | ------------------ | ------------------------------------------------------------------------------------------------- | | Connectivity check | `GET {JENKINS_URL}/api/json` | | Recent builds | `GET {JENKINS_URL}/job//api/json?tree=builds[number,result,timestamp,duration,url,building]` | | Build console log | `GET {JENKINS_URL}/job///consoleText` | | Pipeline stages | `GET {JENKINS_URL}/job///wfapi/describe` (Pipeline Stage View) | | Job list | `GET {JENKINS_URL}/api/json?tree=jobs[name,url,color,lastBuild[...]]` | ## Troubleshooting | Symptom | Fix | | ------------------------------- | ------------------------------------------------------------------------------- | | `Jenkins base URL is required` | Set `JENKINS_URL` or run `opensre integrations setup jenkins`. | | `Jenkins API token is required` | Generate an API token (User → Security → API Token) and configure it. | | `HTTP 401` on verify | Check the username and regenerate the API token; passwords are not accepted. | | `HTTP 403` on verify | The user lacks Overall/Read permission, or CSRF/crumb settings block the token. | | No builds returned | Confirm the job name is correct and has at least one build. | # Jira Source: https://opensre.com/docs/jira Connect Jira so OpenSRE can create and update incident tickets automatically OpenSRE connects to Jira to create incident tickets, update existing issues, and add investigation findings as comments — keeping your team's workflow in sync with automated investigations. ## Prerequisites * Jira Cloud account (Jira Server/Data Center also supported) * API token and project access ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Jira** when prompted and provide your base URL, email, API token, and project key. ### Option 2: Persistent store Add to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "jira-prod", "service": "jira", "status": "active", "credentials": { "base_url": "https://your-org.atlassian.net", "email": "you@example.com", "api_token": "your-api-token", "project_key": "OPS" } } ] } ``` | Field | Description | | ------------- | ----------------------------------------------------------------------------- | | `base_url` | **Required.** Your Jira instance URL (e.g., `https://your-org.atlassian.net`) | | `email` | **Required.** Email address associated with the API token | | `api_token` | **Required.** Jira API token | | `project_key` | **Required.** Default project key for new issues (e.g., `OPS`, `SRE`) | ## Creating a Jira API token 1. Go to [id.atlassian.com/manage-profile/security/api-tokens](https://id.atlassian.com/manage-profile/security/api-tokens) 2. Click **Create API token** 3. Give it a label (e.g., `opensre`) 4. Copy the token For Jira Data Center/Server, use a personal access token from **Profile** → **Personal Access Tokens** instead. ## Verify ```bash theme={null} opensre integrations verify jira ``` Expected output on success: ``` Service: jira Status: passed Detail: Configured for Jira at https://your-org.atlassian.net. ``` This is a configuration-presence check (`base_url`, `email`, and `api_token` are all set) rather than a live API call. ## Troubleshooting | Symptom | Fix | | -------------------- | ------------------------------------------------------------ | | **401 Unauthorized** | Check email and API token combination | | **404 Not Found** | Verify `base_url` (include `https://`) and `project_key` | | **403 Forbidden** | The user may lack permission to create issues in the project | ## Security best practices * Use a **dedicated Jira user** for OpenSRE with only the permissions it needs (create and comment on issues). * Store credentials in `~/.opensre/integrations.json`, not in source code or environment variables. * Rotate the API token periodically. # Kafka Source: https://opensre.com/docs/kafka Connect Kafka so OpenSRE can inspect topic health and consumer group lag during investigations OpenSRE queries Kafka to retrieve topic partition health, consumer group lag, and broker metadata — helping diagnose lag spikes, under-replicated partitions, and consumer group failures during incidents. ## Prerequisites * Apache Kafka cluster (2.x or later) * Network access from the OpenSRE environment to the Kafka brokers ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Kafka** when prompted and provide your bootstrap servers. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} KAFKA_BOOTSTRAP_SERVERS=broker1:9092,broker2:9092 KAFKA_SECURITY_PROTOCOL=PLAINTEXT # or SASL_SSL, SSL, SASL_PLAINTEXT KAFKA_SASL_MECHANISM=PLAIN # optional, for SASL KAFKA_SASL_USERNAME=your-username # optional KAFKA_SASL_PASSWORD=your-password # optional ``` | Variable | Default | Description | | ------------------------- | ----------- | ------------------------------------------------------------------- | | `KAFKA_BOOTSTRAP_SERVERS` | — | **Required.** Comma-separated broker addresses | | `KAFKA_SECURITY_PROTOCOL` | `PLAINTEXT` | Security protocol: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL` | | `KAFKA_SASL_MECHANISM` | — | SASL mechanism: `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` | | `KAFKA_SASL_USERNAME` | — | SASL username | | `KAFKA_SASL_PASSWORD` | — | SASL password | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "kafka-prod", "service": "kafka", "status": "active", "credentials": { "bootstrap_servers": "broker1:9092,broker2:9092", "security_protocol": "SASL_SSL", "sasl_mechanism": "PLAIN", "sasl_username": "your-username", "sasl_password": "your-password" } } ] } ``` ## Common configurations **MSK (AWS Managed Kafka) with IAM:** ```bash theme={null} KAFKA_BOOTSTRAP_SERVERS=b-1.your-cluster.kafka.us-east-1.amazonaws.com:9098 KAFKA_SECURITY_PROTOCOL=SASL_SSL KAFKA_SASL_MECHANISM=AWS_MSK_IAM ``` **Confluent Cloud:** ```bash theme={null} KAFKA_BOOTSTRAP_SERVERS=pkc-xxxxx.us-east-1.aws.confluent.cloud:9092 KAFKA_SECURITY_PROTOCOL=SASL_SSL KAFKA_SASL_MECHANISM=PLAIN KAFKA_SASL_USERNAME=your-api-key KAFKA_SASL_PASSWORD=your-api-secret ``` ## Investigation tools When OpenSRE investigates a Kafka-related alert, two diagnostic tools are available: * **Topic health** — lists topic partition metadata: leader, replicas, ISR status, and under-replicated partitions * **Consumer group lag** — retrieves committed offsets vs high watermarks per partition for a specific consumer group All operations are read-only. ## Verify ```bash theme={null} opensre integrations verify kafka ``` Expected output: ``` Service: kafka Status: passed Detail: Connected to Kafka cluster with 3 broker(s) and 42 topic(s) ``` ## Troubleshooting | Symptom | Fix | | ------------------------- | --------------------------------------------------------------------- | | **Connection timeout** | Check broker hostnames, ports, and firewall rules | | **Authentication failed** | Verify SASL credentials and mechanism match the broker config | | **SSL handshake error** | Ensure the broker's TLS certificate is trusted or configure a CA cert | | **Leader not available** | Broker may be restarting — wait and retry | ## Security best practices * Use **SASL\_SSL** in production — avoid `PLAINTEXT` outside of local development. * Create a dedicated Kafka user with **Describe** permissions only — no produce or consume. * Store credentials in `.env`, not in source code. # Kubernetes Source: https://opensre.com/docs/kubernetes Connect any Kubernetes cluster via kubeconfig so OpenSRE can investigate pods, deployments, and events OpenSRE connects to Kubernetes using a standard kubeconfig. Works with any cluster — GKE, AKS, EKS, on-prem, kind, or minikube — without requiring AWS credentials or cloud-specific tooling. ## Prerequisites * A kubeconfig file (typically `~/.kube/config`) or its raw YAML content * Read access to the namespaces you want to investigate (`get`/`list` on pods, deployments, events, logs) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup kubernetes ``` The wizard will ask for: 1. **kubeconfig** — paste the raw kubeconfig YAML, or provide a file path that will be read and stored inline 2. **Context** — optional; leave blank to use the kubeconfig's current context 3. **Default namespace** — defaults to `default` ### Option 2: Environment variables ```bash theme={null} # Point to a kubeconfig file KUBECONFIG=/home/user/.kube/config # Or provide the raw YAML directly (useful in CI / Docker) KUBECONFIG_CONTENT="$(cat ~/.kube/config)" # Optional overrides KUBECONFIG_CONTEXT=my-context KUBECONFIG_NAMESPACE=production ``` ### Option 3: Verify connectivity ```bash theme={null} opensre integrations verify kubernetes ``` A successful verification lists the namespaces visible to the configured credentials. ## Available tools | Tool | What it does | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kubernetes_list_pods` | List pods in a namespace with phase, readiness, and restart counts | | `kubernetes_get_pod_logs` | Fetch recent log lines from a pod container | | `kubernetes_list_deployments` | List deployments with desired/ready/available replica counts | | `kubernetes_get_events` | List cluster events — crash loops, OOM kills, scheduling failures | | `kubernetes_describe_pod` | Fetch full spec, status, and container states for a single pod | | `kubernetes_list_nodes` | List cluster nodes with conditions, capacity, and allocatable resources | | `kubernetes_list_services` | List services with their type, clusterIP, ports, and selector | | `kubernetes_list_statefulsets` | List StatefulSets with desired/ready/current/updated replica counts | | `kubernetes_list_daemonsets` | List DaemonSets with desired/current/ready/available counts | | `kubernetes_list_ingresses` | List Ingress resources with routing rules, host→service path mappings, and TLS config | | `kubernetes_list_configmaps` | List ConfigMaps with their key-value data | | `kubernetes_get_resource` | Fetch a single named resource by type and name (pods, deployments, statefulsets, daemonsets, services, ingresses, configmaps, replicasets, pvcs, nodes) | ## Configuration reference | Field | Environment variable | Default | Description | | ------------ | ---------------------------------------------- | --------------- | ---------------------------------- | | `kubeconfig` | `KUBECONFIG_CONTENT` or read from `KUBECONFIG` | — | Raw kubeconfig YAML (required) | | `context` | `KUBECONFIG_CONTEXT` | current context | Kubeconfig context to activate | | `namespace` | `KUBECONFIG_NAMESPACE` | `default` | Default namespace for tool queries | ## Example: investigating a crash-looping pod ``` > What's wrong with the payments-api pod in the production namespace? ``` OpenSRE will: 1. Call `kubernetes_list_pods` scoped to `production` to find pods with high restart counts 2. Call `kubernetes_get_pod_logs` to read recent container output 3. Call `kubernetes_get_events` with `involvedObject.name=payments-api` to surface Warning events 4. Correlate the evidence into a root-cause summary ## Permissions The kubeconfig's service account or user needs these Kubernetes RBAC permissions: ```yaml theme={null} rules: - apiGroups: [""] resources: ["pods", "pods/log", "events", "namespaces", "nodes", "services", "configmaps", "persistentvolumeclaims"] verbs: ["get", "list"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "daemonsets", "replicasets"] verbs: ["get", "list"] - apiGroups: ["networking.k8s.io"] resources: ["ingresses"] verbs: ["get", "list"] ``` # LLM Providers Source: https://opensre.com/docs/llm-providers Supported LLM APIs and CLIs, environment variables, and how to switch between them. OpenSRE is provider-agnostic: bring your own model. Selection is controlled by the `LLM_PROVIDER` environment variable, with `LLM_AUTH_METHOD` selecting API-key or OAuth auth where both are supported. Defaults are tracked in [`config/config.py`](https://github.com/Tracer-Cloud/opensre/blob/main/config/config.py) and routing lives in [`core/llm/factory.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/factory.py). ## Quick reference | Provider | `LLM_PROVIDER` | Auth | Reasoning model default | Toolcall model default | | --------------------------- | --------------------------------------- | --------------------------------------------------------- | -------------------------------------------- | --------------------------------------------- | | Anthropic API key | `anthropic` + `LLM_AUTH_METHOD=api_key` | `ANTHROPIC_API_KEY` | `claude-sonnet-4-6` | `claude-haiku-4-5-20251001` | | Anthropic OAuth | `anthropic` + `LLM_AUTH_METHOD=oauth` | Onboarding launches `claude auth login` | Claude Code CLI default | Claude Code CLI default | | OpenAI API key | `openai` + `LLM_AUTH_METHOD=api_key` | `OPENAI_API_KEY` | `gpt-5.4-mini` | `gpt-5.4-mini` | | OpenAI OAuth | `openai` + `LLM_AUTH_METHOD=oauth` | OpenSRE opens `localhost:1455` for Codex-compatible OAuth | Codex CLI default | Codex CLI default | | OpenRouter | `openrouter` | `OPENROUTER_API_KEY` | `openrouter/auto` | `openrouter/auto` | | DeepSeek | `deepseek` | `DEEPSEEK_API_KEY` | `deepseek-v4-pro` | `deepseek-v4-flash` | | Google Gemini API key | `gemini` | `GEMINI_API_KEY` | `gemini-3.1-pro-preview` | `gemini-3.1-flash-lite-preview` | | Google Gemini CLI | `gemini-cli` | `gemini` interactive login or API key env | Gemini CLI default | Gemini CLI default | | Google Antigravity CLI | `antigravity-cli` | `agy` browser OAuth / OS keyring | Antigravity CLI configured model | same as reasoning model | | NVIDIA NIM | `nvidia` | `NVIDIA_API_KEY` | `meta/llama-3.1-405b-instruct` | `meta/llama-3.1-8b-instruct` | | MiniMax | `minimax` | `MINIMAX_API_KEY` | `MiniMax-M3` | `MiniMax-M2.7-highspeed` | | Amazon Bedrock | `bedrock` | AWS IAM (`AWS_REGION`) | `us.anthropic.claude-sonnet-4-6` | `us.anthropic.claude-haiku-4-5-20251001-v1:0` | | Google Vertex AI | `vertex-ai` | Google ADC (`VERTEX_AI_PROJECT`) | `gemini-2.5-pro` | `gemini-2.5-flash-lite` | | Ollama (local) | `ollama` | None (local daemon) | `llama3.2` | `llama3.2` | | GitHub Copilot CLI | `copilot` | `copilot login` or `gh auth login` (CLI) | Copilot CLI default | Copilot CLI default | | xAI Groq API key | `groq` | `GROQ_API_KEY` | `llama-3.3-70b-versatile` | `llama-3.1-8b-instant` | | Azure OpenAI | `azure-openai` | `AZURE_OPENAI_API_KEY` + resource URL | `gpt-5.4-mini` (deployment name) | `gpt-5.4-mini` (deployment name) | | Custom OpenAI-compatible | `custom-openai` | `CUSTOM_OPENAI_API_KEY` + `CUSTOM_OPENAI_BASE_URL` | your `CUSTOM_OPENAI_MODEL` | your `CUSTOM_OPENAI_MODEL` | | Custom Anthropic-compatible | `custom-anthropic` | `CUSTOM_ANTHROPIC_API_KEY` + `CUSTOM_ANTHROPIC_BASE_URL` | your `CUSTOM_ANTHROPIC_MODEL` | your `CUSTOM_ANTHROPIC_MODEL` | | xAI Grok Build CLI | `grok-cli` | `grok login` (CLI) | Grok Build CLI default | Grok Build CLI default | | Pi CLI (BYOK) | `pi` | provider API key env or `pi` → `/login` | Pi configured model (`PI_MODEL` to override) | same as reasoning model | OpenSRE distinguishes two model slots per provider: * **Reasoning model** — full-capability model used for diagnosis, claim validation, and multi-step analysis. * **Toolcall model** — lightweight, lower-cost model used for tool selection and routing. ## Selecting a provider Set `LLM_PROVIDER` (default: `anthropic`) in your environment or `.env` file: ```bash theme={null} export LLM_PROVIDER=openai export OPENAI_API_KEY=sk-... ``` Or run the onboarding wizard, which writes the same values to `.env`: ```bash theme={null} opensre onboard ``` When a provider has more than one supported auth route, onboarding asks for the provider first, then the auth method. For example, choose **Anthropic** and then **OAuth** to use a Claude subscription through the onboarding flow, or choose **API key** to paste `ANTHROPIC_API_KEY`. OpenSRE keeps the provider as `anthropic` or `openai`; `LLM_AUTH_METHOD=oauth` selects the OAuth-backed runtime. OAuth browser login, token storage, refresh, and logout are delegated to the vendor CLI that owns that account session. OpenSRE owns the onboarding UX and does not persist OAuth tokens directly. In the interactive shell, `/model` shows curated quick-pick choices for common models. Providers with fast-changing or account-gated catalogs (OpenAI, OpenRouter, Gemini, NVIDIA, Bedrock, local CLIs, Ollama, and DeepSeek) also accept custom model IDs: ```bash theme={null} /model set openai gpt-5.6-sol /model set openai gpt-5.6-terra --toolcall-model gpt-5.6-luna /model set openai gpt-5.5 --toolcall-model gpt-5.4-mini ``` The GPT-5.6 family has three tiers: `gpt-5.6-sol` (flagship), `gpt-5.6-terra` (balanced), and `gpt-5.6-luna` (cost-efficient). The bare `gpt-5.6` alias is routed to Sol by OpenAI. Override the default model for a slot via env vars: ```bash theme={null} export OPENAI_REASONING_MODEL=gpt-5.4-mini export OPENAI_TOOLCALL_MODEL=gpt-5.4-mini ``` A shared `LLM_MAX_TOKENS` (default `4096`) controls the response token budget for every provider. ## LiteLLM transport OpenSRE can route **hosted API providers** through [LiteLLM](https://docs.litellm.ai/) instead of native vendor SDKs. This is opt-in for most providers and **required** for Azure OpenAI. | Command / variable | What it does | | ------------------------------------------- | ---------------------------------------------------- | | `export OPENSRE_LLM_TRANSPORT=litellm` | Route API providers through LiteLLM | | unset or `export OPENSRE_LLM_TRANSPORT=sdk` | Use native SDK clients (default) | | `opensre onboard` → **Azure OpenAI** | Writes `OPENSRE_LLM_TRANSPORT=litellm` automatically | **CLI-backed providers** (`codex`, `claude-code`, `copilot`, `pi`, etc.) always use their subprocess path — LiteLLM does not affect them. ### Providers supported via LiteLLM When `OPENSRE_LLM_TRANSPORT=litellm` (or when using `LLM_PROVIDER=azure-openai`), OpenSRE builds investigation tool schemas the same way as the SDK path and passes them to `litellm.completion(..., tools=..., tool_choice="auto")`. LiteLLM handles provider routing; OpenSRE keeps schema normalization, retries, and message replay. | `LLM_PROVIDER` | Native SDK path (default) | LiteLLM path | Notes | | -------------- | ----------------------------------- | -------------------------------------- | ------------------------------------------ | | `anthropic` | Anthropic SDK | `anthropic/` | Opt-in via `OPENSRE_LLM_TRANSPORT=litellm` | | `openai` | OpenAI SDK | `openai/` | Opt-in | | `bedrock` | boto3 / AnthropicBedrock / Converse | `bedrock/` | Opt-in; uses AWS credential chain | | `openrouter` | OpenAI-compatible SDK | `openai/` + OpenRouter base URL | Opt-in | | `deepseek` | OpenAI-compatible SDK | `openai/` + DeepSeek base URL | Opt-in | | `gemini` | OpenAI-compatible SDK | `openai/` + Gemini base URL | Opt-in | | `nvidia` | OpenAI-compatible SDK | `openai/` + NVIDIA NIM base URL | Opt-in | | `minimax` | OpenAI-compatible SDK | `openai/` + MiniMax base URL | Opt-in | | `groq` | OpenAI-compatible SDK | `openai/` + Groq base URL | Opt-in | | `ollama` | OpenAI-compatible SDK | `openai/` + `${OLLAMA_HOST}/v1` | Opt-in | | `azure-openai` | — | `azure/` | **Always** via LiteLLM | | `vertex-ai` | — | `vertex_ai/` | **Always** via LiteLLM; uses Google ADC | The native OpenAI SDK transport uses the Responses API for GPT-5.6 agent tool calls, including replaying reasoning and function-call items between tool steps. Older OpenAI models and OpenAI-compatible providers continue to use Chat Completions. For providers beyond this list, LiteLLM supports [100+ backends](https://docs.litellm.ai/docs/providers). OpenSRE only wires the slugs above today — use one of them, or open an issue if you need another first-class provider. ## Login and secret storage Use `opensre auth` for provider login without writing secrets to `.env`: | Command | What it does | | ------------------------------ | --------------------------------------------------------------------------------------------------------------- | | `opensre auth` | Show auth status for subscription and API-key providers | | `opensre auth login deepseek` | Open DeepSeek setup guidance, validate `DEEPSEEK_API_KEY`, store it in the system keychain, and select DeepSeek | | `opensre auth login claude` | Configure the `claude-code` provider through Claude Code CLI subscription login | | `opensre auth login chatgpt` | Configure the `codex` provider through OpenSRE-managed ChatGPT OAuth | | `opensre auth verify deepseek` | Intentionally resolve DeepSeek credentials and refresh stale local metadata | | `opensre auth logout deepseek` | Remove OpenSRE-managed DeepSeek credentials and metadata | `opensre auth login` never reads browser cookies, browser profiles, browser local storage, or IndexedDB. API-key providers use hidden paste prompts plus keyring storage. OpenAI OAuth is handled by OpenSRE's local Codex-compatible callback server; other subscription providers delegate OAuth/session handling to the vendor CLI that owns the browser login flow. `opensre auth` and `/auth status` are prompt-safe: they do not read API-key secrets from Keychain. For API-key providers they inspect environment variables plus non-secret metadata in `~/.opensre/llm-auth.json`. If a key was deleted directly from Keychain, status may show the old metadata until you run `opensre auth verify ` or start a request; that verification marks the provider `stale` when the secret is gone. For Codex CLI auth, status checks do not run `codex login status` by default, because some Codex versions can open browser OAuth while checking a session. Run `/login chatgpt` or `opensre auth login chatgpt` from an interactive terminal when you need to refresh the browser login. OpenSRE starts its own temporary callback server on `http://localhost:1455/auth/callback`, exchanges the short-lived OAuth code, and writes Codex-compatible tokens to the local Codex auth store before redirecting the browser to the Codex-style `/success?id_token=...` completion page. If a browser flow reaches `/success` with token material directly, OpenSRE stores that token material instead of dropping the callback. Use `codex login` only as a direct CLI fallback. Inside the interactive shell, use the same flows through `/auth` or `/login`: ```bash theme={null} /login chatgpt /login claude /login deepseek /auth status ``` ## API providers ### Anthropic ```bash theme={null} export LLM_PROVIDER=anthropic export ANTHROPIC_API_KEY=sk-ant-... # Optional overrides: export ANTHROPIC_REASONING_MODEL=claude-sonnet-4-6 export ANTHROPIC_TOOLCALL_MODEL=claude-haiku-4-5-20251001 ``` The default. Uses the Anthropic Python SDK directly. Get an API key at [console.anthropic.com](https://console.anthropic.com/). Claude Fable 5 (`claude-fable-5`), Anthropic's most capable model, is also selectable (`/model set claude-fable-5`, or via the onboarding wizard and the Claude Code CLI provider). It is priced above the Opus tier, so the defaults stay unchanged — opt in explicitly when you want it. ### OpenAI ```bash theme={null} export LLM_PROVIDER=openai export OPENAI_API_KEY=sk-... # Optional overrides: export OPENAI_REASONING_MODEL=gpt-5.4-mini export OPENAI_TOOLCALL_MODEL=gpt-5.4-mini ``` Uses the OpenAI SDK. Reasoning models (`o1`, `o3`, `o4`, `gpt-5*`) automatically use `max_completion_tokens` instead of `max_tokens`. ### Azure OpenAI Azure OpenAI routes through LiteLLM. Model env vars hold **deployment names** from your Azure resource, not public OpenAI model IDs. ```bash theme={null} export LLM_PROVIDER=azure-openai export OPENSRE_LLM_TRANSPORT=litellm # set automatically by `opensre onboard` export AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com export AZURE_OPENAI_API_KEY=... # Optional override; defaults to 2024-10-21 when unset: # export AZURE_OPENAI_API_VERSION=2024-10-21 # Deployment names (must exist in your Azure resource): export AZURE_OPENAI_REASONING_MODEL=gpt-5.4-mini export AZURE_OPENAI_TOOLCALL_MODEL=gpt-5.4-mini export AZURE_OPENAI_CLASSIFICATION_MODEL=gpt-5.4-mini ``` Quick setup: ```bash theme={null} opensre onboard # choose Azure OpenAI; paste resource URL, API key, deployment ``` Onboarding asks for your **resource URL**, **API key**, then lists **deployments** from that resource for you to pick. OpenSRE sets `AZURE_OPENAI_API_VERSION=2024-10-21` and `OPENSRE_LLM_TRANSPORT=litellm` automatically unless you override them in `.env`. In the REPL, switch provider or deployment like any other API provider: ```bash theme={null} /model set azure-openai gpt-5.4-mini /model set azure-openai gpt-5.4-mini --toolcall-model gpt-5.4-nano ``` If deployment discovery fails during onboarding, enter the deployment name manually — it must match a deployment in your Azure resource, not a model ID from `/openai/models`. ### Custom OpenAI-/Anthropic-compatible endpoints Point OpenSRE at an arbitrary base URL — a LiteLLM proxy, vLLM, LocalAI, or an internal model gateway — for self-hosted, proxied, and on-prem deployments where direct calls to the public APIs are not allowed. Two providers cover the two API shapes: **`custom-openai`** reuses the OpenAI-compatible client: ```bash theme={null} export LLM_PROVIDER=custom-openai export CUSTOM_OPENAI_BASE_URL=http://localhost:4000/v1 # LiteLLM, vLLM, LocalAI, … export CUSTOM_OPENAI_API_KEY=your-key export CUSTOM_OPENAI_MODEL=gpt-5.4 # applies to every tier # Optional per-tier overrides: # export CUSTOM_OPENAI_TOOLCALL_MODEL=gpt-5.4-mini ``` **`custom-anthropic`** uses the Anthropic SDK with a base-URL override: ```bash theme={null} export LLM_PROVIDER=custom-anthropic export CUSTOM_ANTHROPIC_BASE_URL=https://proxy.example.com export CUSTOM_ANTHROPIC_API_KEY=your-key export CUSTOM_ANTHROPIC_MODEL=claude-opus-4-7 ``` Both require the base URL and a model — onboarding and startup fail loudly if either is missing, rather than reaching a wrong endpoint mid-investigation. * **The base URL is used verbatim.** Include the API path yourself (e.g. `/v1` for OpenAI-compatible gateways). OpenSRE never appends a path, so `http://host/v1` stays `http://host/v1` — it does not become `/v1/v1`. * **`custom-anthropic` is SDK-only.** It ignores `OPENSRE_LLM_TRANSPORT=litellm` by design and errors if you force it; use `custom-openai` for a LiteLLM-proxied OpenAI-compatible endpoint. * Run with `opensre --debug` (or set `TRACER_VERBOSE=1`) to print the resolved provider, redacted base URL (host only — never a token), and model for each call when debugging a gateway. Quick setup: ```bash theme={null} opensre onboard # choose the custom provider; paste base URL, API key, model ``` ### OpenRouter ```bash theme={null} export LLM_PROVIDER=openrouter export OPENROUTER_API_KEY=sk-or-... # Optional override (single value applies to both slots if set): export OPENROUTER_MODEL=openrouter/auto # Or per-slot: export OPENROUTER_REASONING_MODEL=anthropic/claude-sonnet-4-6 export OPENROUTER_TOOLCALL_MODEL=openai/gpt-4o-mini ``` OpenAI-compatible proxy — pick any model on [openrouter.ai/models](https://openrouter.ai/models). Base URL: `https://openrouter.ai/api/v1`. ### DeepSeek ```bash theme={null} export LLM_PROVIDER=deepseek export DEEPSEEK_API_KEY=sk-... # Optional override (single value applies to all slots if set): export DEEPSEEK_MODEL=deepseek-v4-pro # Or per-slot: export DEEPSEEK_REASONING_MODEL=deepseek-v4-pro export DEEPSEEK_TOOLCALL_MODEL=deepseek-v4-flash ``` Uses DeepSeek's official OpenAI-compatible API endpoint at `https://api.deepseek.com`. Run `opensre auth login deepseek` for browser-assisted key setup and secure local storage. ### Google Gemini ```bash theme={null} export LLM_PROVIDER=gemini export GEMINI_API_KEY=... # Optional override: export GEMINI_MODEL=gemini-3.1-pro-preview # Or per-slot: export GEMINI_REASONING_MODEL=gemini-3.1-pro-preview export GEMINI_TOOLCALL_MODEL=gemini-3.1-flash-lite-preview ``` Uses Google's OpenAI-compatible endpoint at `https://generativelanguage.googleapis.com/v1beta/openai/`. Get an API key at [aistudio.google.com](https://aistudio.google.com/app/apikey). ### NVIDIA NIM ```bash theme={null} export LLM_PROVIDER=nvidia export NVIDIA_API_KEY=nvapi-... # Optional override: export NVIDIA_MODEL=meta/llama-3.1-405b-instruct # Or per-slot: export NVIDIA_REASONING_MODEL=meta/llama-3.1-405b-instruct export NVIDIA_TOOLCALL_MODEL=meta/llama-3.1-8b-instruct ``` Uses NVIDIA's OpenAI-compatible API at `https://integrate.api.nvidia.com/v1`. Browse available models on [build.nvidia.com](https://build.nvidia.com/). ### MiniMax ```bash theme={null} export LLM_PROVIDER=minimax export MINIMAX_API_KEY=... # Optional override (single value applies to both slots if set): export MINIMAX_MODEL=MiniMax-M3 # Or per-slot: export MINIMAX_REASONING_MODEL=MiniMax-M3 export MINIMAX_TOOLCALL_MODEL=MiniMax-M2.7-highspeed ``` OpenAI-compatible endpoint at `https://api.minimax.io/v1`. Temperature is fixed to `1.0` to match MiniMax recommendations. ### Groq ```bash theme={null} export LLM_PROVIDER=groq export GROQ_API_KEY=gsk_... # Optional override: export GROQ_MODEL=llama-3.3-70b-versatile # Or per-slot: export GROQ_REASONING_MODEL=llama-3.3-70b-versatile export GROQ_TOOLCALL_MODEL=llama-3.1-8b-instant ``` Uses Groq's OpenAI-compatible API at `https://api.groq.com/openai/v1`. ### Amazon Bedrock ```bash theme={null} export LLM_PROVIDER=bedrock export AWS_REGION=us-east-1 # Optional overrides: export BEDROCK_REASONING_MODEL=us.anthropic.claude-sonnet-4-6 export BEDROCK_TOOLCALL_MODEL=us.anthropic.claude-haiku-4-5-20251001-v1:0 ``` No API key — auth uses the AWS credential chain (environment variables, shared credentials file, or IAM role). Your principal needs permission to invoke the model IDs you configure (for example Bedrock `InvokeModel` / Converse access scoped to those resources in IAM). **Model routing:** * **Anthropic Claude** on Bedrock (`anthropic.claude-*`, `us.anthropic.claude-*`, and foundation-model ARNs that contain `anthropic.claude`) use the existing **AnthropicBedrock** SDK path. * **Other Bedrock foundation models** (for example Mistral, Meta Llama, Amazon Titan IDs you enable in your account) use the **Bedrock Converse** API via `boto3`, so you can set `BEDROCK_REASONING_MODEL` to a non-Claude model ID when your use case requires it. * **Application inference profile** ARNs (`…:application-inference-profile/…`) do not encode the vendor in the ID; those are always sent through **Converse**, which works for any backing model in the profile. Defaults in `config/config.py` are US cross-region inference profile IDs for Anthropic Claude; override with IDs or ARNs that are **inference-access enabled** in your account and region. ### Google Vertex AI ```bash theme={null} export LLM_PROVIDER=vertex-ai export VERTEX_AI_PROJECT=my-gcp-project # Optional (defaults to us-central1): export VERTEX_AI_LOCATION=us-central1 # Optional overrides: export VERTEX_AI_REASONING_MODEL=gemini-2.5-pro export VERTEX_AI_TOOLCALL_MODEL=gemini-2.5-flash-lite ``` No API key — auth uses Google Application Default Credentials (ADC): run `gcloud auth application-default login`, or set `GOOGLE_APPLICATION_CREDENTIALS` to a service-account key file, or rely on the ambient GCE/GKE metadata server. Your principal needs the Vertex AI User IAM role (or equivalent) in the configured project. Always routed through LiteLLM (like Azure OpenAI) as `vertex_ai/`. Curated model choices are Gemini models served via Vertex; any other Vertex-supported model ID also works by typing it directly (`allow_custom_models`). The default (`gemini-2.5-pro`/`-flash`/`-flash-lite`) is the GA Gemini generation. Gemini 3.x (`gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-3.1-flash-lite-preview`) is selectable in the wizard but is Preview-only in Vertex Model Garden as of mid-2026 — expect availability and pricing to change. ### Ollama (local) ```bash theme={null} export LLM_PROVIDER=ollama # Optional overrides: export OLLAMA_HOST=http://localhost:11434 export OLLAMA_MODEL=llama3.2 ``` Run any local model exposed by an [Ollama](https://ollama.com/) daemon. No API key required — OpenSRE talks to Ollama's OpenAI-compatible endpoint at `${OLLAMA_HOST}/v1`. ## CLI providers (subprocess) CLI-backed providers shell out to a vendor CLI instead of an HTTP API during inference. OpenSRE detects the binary on `PATH` (or via an explicit env var) and reuses the existing session. OpenAI OAuth is stored by OpenSRE in Codex-compatible auth format; other CLI-backed providers authenticate via the vendor's own login command. **Investigation timeouts:** Each ReAct turn runs one full CLI subprocess with the system prompt, tool schemas, and conversation history. The shared default subprocess budget is **300 seconds** (Python adds a small buffer). Override per provider when needed, for example `GEMINI_CLI_TIMEOUT_SECONDS`, `CLAUDE_CODE_TIMEOUT_SECONDS`, or `ANTIGRAVITY_CLI_TIMEOUT_SECONDS` (clamped 30–600 where the adapter supports it). ### OpenAI OAuth backend ```bash theme={null} export LLM_PROVIDER=openai export LLM_AUTH_METHOD=oauth # Authenticate through onboarding or `/login chatgpt`: opensre auth login chatgpt # Optional overrides (all blank-by-default): export CODEX_MODEL= export CODEX_BIN= ``` Requires the [OpenAI Codex CLI](https://github.com/openai/codex). If `CODEX_MODEL` is unset, OpenSRE omits `-m` so `codex exec` uses the CLI's currently configured model. If `CODEX_BIN` is unset, the binary is resolved via `PATH` and known install locations. Run `opensre onboard`, `/login chatgpt`, or `opensre auth login chatgpt` to launch OpenSRE-managed Codex browser login on `localhost:1455` and persist `LLM_PROVIDER=openai` with `LLM_AUTH_METHOD=oauth`. Existing `LLM_PROVIDER=codex` configs still work for backward compatibility. ### Anthropic OAuth backend ```bash theme={null} export LLM_PROVIDER=anthropic export LLM_AUTH_METHOD=oauth # Authenticate through onboarding, `/login claude`, or the Claude Code CLI directly: claude auth login # Optional overrides (all blank-by-default): export CLAUDE_CODE_MODEL= export CLAUDE_CODE_BIN= ``` Requires the [Claude Code CLI](https://github.com/anthropics/claude-code) (`npm i -g @anthropic-ai/claude-code`). If `CLAUDE_CODE_MODEL` is unset, OpenSRE omits the `--model` flag and the CLI uses its configured default. If `CLAUDE_CODE_BIN` is unset, the binary is resolved via `PATH` and known install locations. Run `opensre onboard`, `/login claude`, or `opensre auth login claude` to launch Claude browser login when needed and persist `LLM_PROVIDER=anthropic` with `LLM_AUTH_METHOD=oauth`. Existing `LLM_PROVIDER=claude-code` configs still work for backward compatibility. ### GitHub Copilot ```bash theme={null} export LLM_PROVIDER=copilot # Authenticate the Copilot CLI separately. Either flow works — the adapter # detects both. The interactive `/login` slash command inside `copilot` writes # to the platform credential store; `gh auth login` is an equivalent path that # Copilot CLI delegates to automatically. copilot login # OAuth device flow; preferred CLI-first onboarding # or: gh auth login # logs you into the gh CLI; Copilot will use that token # Optional overrides (all blank-by-default): export COPILOT_MODEL= export COPILOT_BIN= # Optional auth bypass for automation (only used when no CLI login is detected): # export COPILOT_GITHUB_TOKEN= # export GH_TOKEN= # export GITHUB_TOKEN= ``` Requires the [GitHub Copilot CLI](https://docs.github.com/copilot/how-tos/use-copilot-agents/use-copilot-cli) (`npm i -g @github/copilot`). Login uses the interactive `/login` slash command or `copilot login`. OpenSRE detects auth in this order: (1) `COPILOT_GITHUB_TOKEN` / `GH_TOKEN` / `GITHUB_TOKEN` env, (2) [`gh auth status`](https://docs.github.com/en/copilot/how-tos/copilot-cli/set-up-copilot-cli/authenticate-copilot-cli#authenticating-with-github-cli) when `gh` is on `PATH` (including `✓ Logged in to github.com account …`, `- Active account: true`, or a supported `- Token:` prefix: `gho_`, `github_pat_`, `ghu_` per Copilot docs — not `ghp_`), with `gh auth status --hostname …` when `COPILOT_GH_HOST` or `GH_HOST` targets a non-`github.com` host. It does **not** read plaintext `$COPILOT_HOME/config.json` (keychain-backed installs may omit it; mis-parsing arbitrary JSON risks false positives). If nothing matches, detection reports `logged_in=None` and the runner verifies at invoke time. If `COPILOT_MODEL` is unset, OpenSRE omits `--model`. Invocations run as `copilot -p PROMPT --no-color --no-ask-user --silent` so they never block on user input. **BYOK / `COPILOT_OFFLINE`:** GitHub auth may be unnecessary; a `None` probe can still be fine if Copilot is configured for offline or external providers only. ### Google Antigravity CLI ```bash theme={null} export LLM_PROVIDER=antigravity-cli # Authenticate the Antigravity CLI separately (browser OAuth on first run): agy # interactive launch triggers Google Sign-In; token cached by OS keyring # Stay current — 1.0.0 had OAuth hangs (fixed in 1.0.1): agy update # Optional overrides (all blank-by-default): export ANTIGRAVITY_CLI_BIN= export ANTIGRAVITY_CLI_TIMEOUT_SECONDS=300 # default 300; clamped 30–600; maps to `--print-timeout {N}s` # Note: ANTIGRAVITY_CLI_MODEL is registered for forward-compat but currently no-op # (agy v1.0.2 does not expose --model in headless `-p` mode). Each invocation uses # whatever model is persisted in agy's local config; switch it interactively with # `/models` inside the `agy` REPL. The wizard's model picker is a forward-compat # catalog: once Google ships `--model` in headless, picking a value here will start # being forwarded to agy via a one-line change in the adapter. ``` Antigravity CLI (`agy`) is Google's successor to Gemini CLI. Install via `curl -fsSL https://antigravity.google/cli/install.sh | bash`, then run `agy install` to configure your shell `PATH`. The minimum tested version is **1.0.1** — older builds log a warning via the probe and direct you to `agy update`. **Why two Google providers?** Google's [transition announcement](https://developers.googleblog.com/an-important-update-transitioning-gemini-cli-to-antigravity-cli/) states that **on 2026-06-18** Gemini CLI stops serving Pro/Ultra and free users. Paid Gemini Code Assist licences keep Gemini CLI indefinitely. OpenSRE keeps both `gemini-cli` (deprecated alias with a probe-time notice) and `antigravity-cli` so either group can run without surprises. As a best-effort fallback, the probe treats explicit `GEMINI_API_KEY` / `GOOGLE_API_KEY` / `GOOGLE_APPLICATION_CREDENTIALS` env credentials as authenticated (mirroring the Gemini CLI adapter), so users migrating across the two CLIs can keep their existing env-var-based auth without re-running the browser flow. Invocations run as `agy -p PROMPT --print-timeout {N}s`. The adapter never passes `--continue` / `--conversation` / `--sandbox` / `--dangerously-skip-permissions`, keeping every opensre call ephemeral. ### xAI Grok Build CLI ```bash theme={null} export LLM_PROVIDER=grok-cli # Authenticate the Grok Build CLI separately. Either path works: grok login # OAuth sign-in with a SuperGrok / X Premium+ account # ...or, for headless / CI runs, use an API key instead of a browser login: export XAI_API_KEY=xai-... # get one from the xAI console # Optional overrides (all blank-by-default): export GROK_CLI_MODEL= # e.g. grok-build; unset → CLI configured default export GROK_CLI_BIN= # explicit path to the `grok` binary export GROK_CLI_TIMEOUT_SECONDS=300 # default 300; clamped 30-600 ``` Requires the [xAI Grok Build CLI](https://x.ai/cli) (binary: `grok`). Install with `curl -fsSL https://x.ai/cli/install.sh | bash` (macOS/Linux) or `irm https://x.ai/cli/install.ps1 | iex` (Windows). If `GROK_CLI_MODEL` is unset, OpenSRE omits `-m` and the CLI uses its configured default. The wizard populates the model list live from `grok models` at onboarding time so newly released models appear without an OpenSRE update. Invocations run as `grok -p PROMPT --output-format plain`, so each opensre call is a single non-interactive turn. The adapter deliberately omits `--always-approve`: OpenSRE drives its own tools, so Grok is used purely as a text responder and never auto-executes shell commands or file edits. **Auth detection:** auth is probed via `grok models` (\~0.5 s, no LLM call), which prints "You are logged in" on success. `XAI_API_KEY` is treated as an authenticated fallback for headless / CI runs even when the probe result is unclear. `XAI_API_KEY` is forwarded **only** to the Grok subprocess (never via the shared CLI env allowlist), so it cannot leak into other CLI adapters. > **Not to be confused with `groq`.** The `grok-cli` provider is xAI's Grok Build CLI. The > separate `groq` provider is the Groq HTTP API (a different company); the two are unrelated. ### Pi CLI ```bash theme={null} export LLM_PROVIDER=pi # Authenticate Pi separately. Either path works — the adapter detects both: pi # then run /login for an OAuth subscription or to store a key # …or export a provider API key Pi understands (BYOK), e.g. for Gemini: export GEMINI_API_KEY=... export PI_MODEL=google/gemini-2.5-flash-lite # provider/model; unset → Pi configured default export PI_BIN= # explicit path to the `pi` binary (optional) ``` Requires the [Pi CLI](https://pi.dev) (`npm i -g @earendil-works/pi-coding-agent`). Pi is bring-your-own-key across \~30 providers, so `PI_MODEL` uses the `provider/model` form (for example `google/gemini-2.5-flash-lite`, `anthropic/claude-haiku-4-5`, `openai/gpt-4o-mini`); run `pi --list-models` for the full catalog. If `PI_MODEL` is unset, OpenSRE omits `--model` and Pi uses its configured default. If `PI_BIN` is unset, the binary is resolved via `PATH` and known install locations. Invocations run as `pi -p PROMPT` (non-interactive print mode), so each OpenSRE call is a single headless turn with no TTY. **Auth detection:** Pi has no non-interactive auth-status command, so OpenSRE detects auth from state: (1) a supported provider API key in the environment (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, …) → authenticated; (2) otherwise, credentials stored in `~/.pi/agent/auth.json` (written by `pi`'s `/login`, covering OAuth subscriptions and stored keys) → authenticated; (3) neither → not authenticated. Provider API keys are forwarded **only** to the Pi subprocess, never via the shared CLI env allowlist, so they cannot leak into other CLI adapters. See [`integrations/llm_cli/AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/integrations/llm_cli/AGENTS.md) for the adapter pattern used to add new CLI providers. ## Reasoning effort (interactive shell) In the TTY REPL (`opensre` with no subcommand), `/effort` stores a **session** preference for how strongly reasoning models should think before answering. It applies only when `LLM_PROVIDER` is **`openai`** (HTTP API) or **`codex`** (Codex CLI); other providers ignore the setting and the shell notes that. | Input | Sent to the model | | -------------------------------- | ----------------- | | `low`, `medium`, `high`, `xhigh` | same string | | `max` | `xhigh` | Run `/effort` alone to show the current choice (or `(default)` when unset) and the usage line. `/new` starts a fresh session but **keeps** `/effort` (and trust mode), consistent with other session prefs. Outside the REPL, optional defaults use the environment variable: ```bash theme={null} export OPENSRE_REASONING_EFFORT=high # low | medium | high | xhigh ``` Session `/effort` overrides this for interactive runs. Implementation: [`config/llm_reasoning_effort.py`](https://github.com/Tracer-Cloud/opensre/blob/main/config/llm_reasoning_effort.py). ## Provider diagnostics OpenSRE does not silently switch LLM providers when the provider in `LLM_PROVIDER` is missing credentials. It keeps the configured provider selected and reports missing or stale auth status before starting LLM work. * **`opensre auth` and `/auth status`** show prompt-safe status from environment variables, provider metadata, CLI probes, or ambient/local config. * **`opensre auth verify `** intentionally checks request-time credentials and refreshes metadata. * **`opensre config llm` and `opensre doctor`** report the configured provider plus credential status without resolving secrets. * **Provider errors** are prefixed with the configured provider that served the request: ``` [LLM provider: openai] Missing credential for LLM provider 'openai'. Set OPENAI_API_KEY or run `opensre auth login openai`. ``` If credentials are missing, set the provider's API-key environment variable, run `opensre auth login `, or change `LLM_PROVIDER` to a provider you have configured. ## Switching providers at runtime OpenSRE caches LLM clients on first use. To switch providers within a single process (tests, benchmarks), call `reset_llm_clients()` from `core.llm.factory` after updating the env vars; otherwise a fresh process picks up the new `LLM_PROVIDER` automatically. ## Where this lives in the code * Provider literals and defaults: [`config/config.py`](https://github.com/Tracer-Cloud/opensre/blob/main/config/config.py) (`LLMProvider`, `LLMSettings`). * Runtime routing: [`core/llm/factory.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/factory.py) (`resolve_llm_route`, `get_llm`) and client construction in [`core/llm/client_builders.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/client_builders.py). * LiteLLM routing (when enabled): [`core/llm/transports/litellm/routing.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/transports/litellm/routing.py). * Investigation tool-calling adapters: [Investigation tool calling](/docs/investigation-tool-calling). * API-backed provider guide: [`core/llm/AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/AGENTS.md). * CLI-backed provider guide: [`integrations/llm_cli/AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/integrations/llm_cli/AGENTS.md). # MariaDB Source: https://opensre.com/docs/mariadb Connect MariaDB so OpenSRE can diagnose database issues during investigations OpenSRE uses MariaDB diagnostics to investigate database-related alerts — checking server health, finding slow queries, monitoring replication, and analyzing active threads and InnoDB engine state. ## Prerequisites * MariaDB 10.5+ (10.11 LTS or 11.x recommended) * Network access from the OpenSRE environment to your MariaDB instance * A database user with at least `SELECT` + `PROCESS` privileges (and `SELECT` on `performance_schema` for slow-query insights) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup mariadb ``` You will be prompted for host, port, database, username, password, and whether to enable SSL. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} MARIADB_HOST=db.example.com MARIADB_PORT=3306 MARIADB_DATABASE=production MARIADB_USERNAME=opensre_ro MARIADB_PASSWORD=... MARIADB_SSL=true ``` | Variable | Default | Description | | ------------------ | --------- | ----------------------------------------------------------------------------------------- | | `MARIADB_HOST` | — | **Required.** MariaDB server hostname or IP | | `MARIADB_PORT` | `3306` | MariaDB server port | | `MARIADB_DATABASE` | — | **Required.** Target database for slow-query analysis | | `MARIADB_USERNAME` | — | **Required.** Database user | | `MARIADB_PASSWORD` | *(empty)* | Database password; required unless the user is configured for passwordless authentication | | `MARIADB_SSL` | `true` | Use TLS with certificate verification | ### Option 3: Persistent store Credentials are automatically persisted to `~/.opensre/integrations.json` with `0o600` permissions: ```json theme={null} { "version": 1, "integrations": [ { "id": "mariadb-prod", "service": "mariadb", "status": "active", "credentials": { "host": "db.example.com", "port": 3306, "database": "production", "username": "opensre_ro", "password": "...", "ssl": true } } ] } ``` ## Recommended user setup Create a dedicated read-only user for OpenSRE so it cannot modify data: ```sql theme={null} CREATE USER 'opensre_ro'@'%' IDENTIFIED BY 'strong-password'; GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO 'opensre_ro'@'%'; GRANT SELECT ON performance_schema.* TO 'opensre_ro'@'%'; FLUSH PRIVILEGES; ``` The `PROCESS` privilege lets OpenSRE read `information_schema.PROCESSLIST`. `REPLICATION CLIENT` enables `SHOW ALL SLAVES STATUS` / `SHOW SLAVE STATUS`. `SELECT` on `performance_schema` is only needed if you want slow-query insights. ## TLS configuration SSL is enabled by default and uses the system CA bundle to verify the server certificate. Set `MARIADB_SSL=false` only in trusted local networks (development). ## Investigation tools When OpenSRE investigates a MariaDB-related alert, five diagnostic tools are available: ### Process list Retrieves active threads from `information_schema.PROCESSLIST`, excluding sleeping connections. Results are sorted by duration so long-running queries appear first. ### Global status Returns a curated set of key metrics from `SHOW GLOBAL STATUS` — thread counts, connection totals, slow query count, InnoDB buffer pool statistics, row lock waits, and uptime. ### InnoDB status Runs `SHOW ENGINE INNODB STATUS` and returns the engine status text, truncated to 4000 characters with a truncation marker appended when shortening occurs. Useful for investigating deadlocks, buffer pool pressure, and I/O patterns. ### Slow queries Reads `performance_schema.events_statements_summary_by_digest` to list statements by average execution time. Requires `performance_schema` to be enabled. If `performance_schema` is disabled, the tool returns an informative note instead of failing. Enable it in `my.cnf` with `performance_schema=ON`. ### Replication status Runs `SHOW ALL REPLICAS STATUS` (MariaDB multi-source replication; alias: `SHOW ALL SLAVES STATUS` on older builds) with a fallback to `SHOW REPLICA STATUS`. Returns all configured replication channels, each with I/O thread state, SQL thread state, seconds behind primary, last error, and log positions. ## Verify ```bash theme={null} opensre integrations verify mariadb ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL mariadb local env passed Connected to MariaDB 11.8.6-MariaDB; target database: production. ``` ## Troubleshooting | Symptom | Fix | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | | **Connection refused** | Verify host/port, check firewall rules, and confirm MariaDB is listening on the network interface (`bind-address` in `my.cnf`). | | **Access denied for user** | Confirm the username/password and that the user is granted access from the OpenSRE host (`'opensre_ro'@'%'` or a specific IP). | | **SSL: CERTIFICATE\_VERIFY\_FAILED** | The server certificate is not trusted by the system CA bundle. Install the correct CA or set `MARIADB_SSL=false` in trusted networks. | | **performance\_schema is disabled** | Slow-query tool returns an empty list with a note. Enable in `my.cnf`: `performance_schema=ON`. | | **SELECT command denied on performance\_schema** | Grant `SELECT` on `performance_schema.*` to the user. | | **This server is not configured as a replica** | Expected on standalone instances — replication tool returns an empty channel list, other tools still work. | ## Security best practices * Use a **dedicated read-only** user — never `root` or an admin account. * Always enable **TLS** in production (`MARIADB_SSL=true`, which is the default). * Keep passwords out of source control — use `.env` or the persistent store. * Rotate credentials periodically and scope them to specific hosts where possible. # Masking Sensitive Identifiers Source: https://opensre.com/docs/masking Reversible masking of pod, cluster, and account identifiers before external LLM calls ## Overview OpenSRE can mask sensitive infrastructure identifiers (pod names, cluster names, hostnames, account IDs, service names, IP addresses, emails) **before** sending text to external LLMs, and restore the originals in any user-facing output (Slack report, problem MD, ingest). This lets teams use external models while keeping raw identifiers private to the investigation runtime. Masking is **off by default**. Enable it per investigation via environment variables — no code changes required. ## How it works 1. When masking is enabled, the investigation step replaces sensitive identifiers in collected evidence with stable placeholders like ``, ``, ``. The placeholder→original map is stored in investigation state. 2. The diagnosis model receives masked evidence, so raw identifiers never hit the external LLM. 3. After the model returns its root-cause analysis, OpenSRE restores real identifiers in downstream state and display output. 4. Report delivery (for example Slack) runs a final unmask pass before sending, as defence in depth. The same identifier always maps to the same placeholder within a single investigation, so the LLM's reasoning about `` remains coherent. ## Environment variables | Variable | Default | Description | | -------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OPENSRE_MASK_ENABLED` | `false` | Master switch. Set to `true` / `1` / `yes` / `on` to activate masking. | | `OPENSRE_MASK_KINDS` | `pod,namespace,cluster,hostname,account_id,ip_address,email,service_name` | Comma-separated list of identifier kinds to mask. Unknown kinds are ignored with a warning. Empty value uses all defaults. | | `OPENSRE_MASK_EXTRA_REGEX` | *(empty)* | Optional JSON object mapping a label → regex for custom identifiers. Example: `'{"jira_key": "\\\\b[A-Z]+-\\\\d+\\\\b"}'`. Group 1 of the regex, if present, defines the span to mask. | Policies are read fresh from the environment at the start of each investigation — changes take effect on the next run without a restart. ## Built-in identifier kinds | Kind | Example input | Example placeholder | | -------------- | ------------------------------------------------- | ------------------------------ | | `pod` | `etl-worker-7d9f8b-xkp2q` | `` | | `namespace` | `kube_namespace:tracer-test` | `kube_namespace:` | | `cluster` | `eks_cluster:prod-us-east-1` | `eks_cluster:` | | `service_name` | `service:checkout-api` | `service:` | | `hostname` | `kind-control-plane`, `ip-10-0-1-23.ec2.internal` | `` | | `account_id` | `123456789012` | `` | | `ip_address` | `192.168.1.50` | `` | | `email` | `alice@example.com` | `` | ## Round-trip guarantee For the built-in detectors and extra regex patterns, `mask → unmask` round-trips the original payload byte-for-byte. See `tests/masking/test_integration_with_k8s_fixture.py` for a worked example against a realistic Datadog k8s alert. ## Relationship to guardrails The masking layer is complementary to the one-way `GuardrailEngine`. Guardrails handle hard-block rules (credit cards, API keys) and replace matches with `[REDACTED]` irreversibly. Masking handles infrastructure identifiers reversibly so they can be restored for user-facing output. Both can be active together: guardrails apply first at the LLM client layer, then masking at the node layer. ## Example ```bash theme={null} export OPENSRE_MASK_ENABLED=true export OPENSRE_MASK_KINDS=pod,namespace,cluster,hostname opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` During the investigation the LLM sees masked evidence; the final Slack report shows the original pod, namespace, and cluster names. # Long-Term Memory Source: https://opensre.com/docs/memory The agent remembers who you are and how your infrastructure works — durable facts persist across sessions as plain local files you control. ## Commands at a glance | Command | What it does | | ----------------------- | ------------------------------------------ | | `/memory` | List everything the agent remembers | | `/memory show ` | Print one memory in full | | `/memory forget ` | Delete one memory | | `/memory path` | Print the folder where memories are stored | *** ## What the agent remembers OpenSRE keeps durable knowledge across sessions so you never repeat yourself: * **Who you are** — your name, role, and how you like to work * **Your infrastructure** — cluster names, naming conventions, known-flaky services * **Your preferences** — report formats, notification choices, defaults * **Investigation learnings** — root causes and lessons worth keeping The memory index is shown to the agent at the start of every conversation, so facts saved last week ground answers today. ## Saving memories Talk normally — no special phrasing: ``` our prod cluster is eks-prod-eu-1 ``` ``` my name is Vaibhav, I'm on the platform team ``` When the agent judges a fact useful for future sessions, it stores it (and updates an existing entry instead of duplicating). After **every chat turn**, a background pass also scans for durable facts so knowledge lands even if the agent didn't call a memory tool mid-turn. On shell exit that pass runs to completion so nothing is dropped mid-quit. ## Viewing and deleting memories ``` /memory ``` ``` Long-term memory name type description updated ───────────────────────────────────────────────────────────────────────────────────────── prod-cluster-conventions infrastructure Prod EKS clusters: eks-prod- 2026-07-09 user-profile user Vaibhav, platform team 2026-07-09 ``` Delete one with `/memory forget prod-cluster-conventions`, or ask the agent: "forget what you know about the prod cluster". ## Where memories live On a laptop, memories are plain markdown files under `~/.opensre/memory/` — one file per memory plus a generated `MEMORY.md` index. OpenSRE creates that folder (and an empty index) the first time memory is used in a chat or via `/memory`. They are stored **locally and unencrypted**. On a Slack silo each member instead gets a private memory folder inside the organization's mounted context volume (`/users//memory/`), not a shared host folder. Every chat and action turn includes the stored facts (summaries plus bodies, within a size budget) in prompts sent to your configured LLM provider. You can open, edit, or delete the files directly at any time; the index rebuilds on the next write. ## Turning memory off | Environment variable | Effect | | --------------------------------------- | ------------------------------------------------------------- | | `OPENSRE_MEMORY_DISABLED=1` | Disable long-term memory entirely | | `OPENSRE_MEMORY_AUTOEXTRACT_DISABLED=1` | Keep memory, but skip the automatic per-turn extraction | | `OPENSRE_MEMORY_GATEWAY_ENABLED=1` | Opt in on Slack/Telegram gateway (off by default — see below) | The per-turn extraction makes one lightweight LLM call when durable facts may have appeared (coalesced across rapid turns) and saves at most five memories per pass. On shell exit that pass runs to completion after resources are released so durable facts are not dropped. Never share secrets or credentials expecting them to be remembered — the memory ingestion path blocks obvious tokens, passwords, private keys, and credential-shaped values, and redacts matching spans before the extraction LLM call. ## Gateway (Slack / Telegram) On Slack, memory is **per-user scoped** — each member's facts live under their own `users//memory/` folder inside the org volume, so one user's memories cannot appear in another's prompts or channel replies. Telegram turns are unbound and stay **host-global**. Because of that (and to keep memory off by default on shared hosts), the gateway path is **off by default**; opt in with `OPENSRE_MEMORY_GATEWAY_ENABLED=1`. # Buzz Source: https://opensre.com/docs/messaging/buzz Deliver investigation findings to a Buzz (block/buzz) channel. OpenSRE's Buzz integration delivers investigation findings to a channel in [Buzz](https://github.com/block/buzz), Block's self-hostable, Nostr-based workspace where humans and AI agents share rooms. Start the interactive shell with `opensre` (no subcommand). Slash commands below are run from that REPL. Steps 1–3 give you **outbound delivery** (investigation reports, watchdog alarms, and agent-requested messages posted to a channel). To `@mention` the agent from a Buzz channel and get replies, also do [Step 4](#step-4-enable-two-way-chat). *** ## Prerequisites * A running Buzz relay (self-hosted) and its URL, e.g. `https://buzz.example.com` (defaults to `http://localhost:3000` for local dev). * The **`buzz` CLI** on `PATH`. It is not published to any package registry — build it from the [block/buzz](https://github.com/block/buzz) repo: ```bash theme={null} cargo install --path crates/buzz-cli ``` If you can't put it on `PATH`, point OpenSRE at the binary with `BUZZ_PATH=/path/to/buzz` or the `buzz_path` field during setup. * An agent identity (a Nostr keypair). Generate one with the relay's admin tool: ```bash theme={null} buzz-admin generate-key ``` This prints a public key (`npub...`) and a private key (`nsec...` or hex). The private key is what OpenSRE needs — treat it like a password; it's routed to the OS keyring, never plain `.env`. * If your relay has `BUZZ_REQUIRE_RELAY_MEMBERSHIP` enabled, the agent's public key must be registered as a relay member before it can post — ask your Buzz admin to add it, or messages will fail silently. *** ## Step 1: Find a channel Buzz channels are identified by **UUID**, not by name. List the channels your agent identity can see: ```bash theme={null} BUZZ_PRIVATE_KEY= BUZZ_RELAY_URL= buzz channels list ``` Copy the UUID of the channel investigation reports should land in. *** ## Step 2: Configure the integration ### Option A: Onboarding wizard (recommended) Interactive shell: ```text theme={null} /onboard ``` CLI: ```bash theme={null} opensre onboard ``` Choose **Buzz** from the integration list. The wizard prompts for: * **Relay URL** (`BUZZ_RELAY_URL`, defaults to `http://localhost:3000`) * **Private key** (OS keyring via `sync_env_secret`, not plain `.env`) * **Default channel** — the UUID from Step 1 (`BUZZ_DEFAULT_CHANNEL`) * **Auth tag** — optional NIP-OA JSON for owner attestation (`BUZZ_AUTH_TAG`) * **CLI binary path** — only if `buzz` isn't on `PATH` (`BUZZ_PATH`) Credentials are also saved to `~/.opensre/integrations.json` via `upsert_integration("buzz", ...)`. ### Option B: Environment variables Set in `.env` (private key can also live in the keyring after wizard setup): ```bash theme={null} BUZZ_RELAY_URL=https://buzz.example.com BUZZ_PRIVATE_KEY= BUZZ_DEFAULT_CHANNEL= # BUZZ_AUTH_TAG= BUZZ_PATH=buzz ``` | Variable | Description | | ------------------------------------ | ----------------------------------------------------------------------------------------------- | | `BUZZ_RELAY_URL` | Buzz relay base URL. Defaults to `http://localhost:3000`. | | `BUZZ_PRIVATE_KEY` | Agent's Nostr private key (hex or `nsec1...`). Required. Resolved via env then keyring. | | `BUZZ_DEFAULT_CHANNEL` | Default delivery destination — a channel UUID from `buzz channels list`. Required for delivery. | | `BUZZ_AUTH_TAG` | Optional NIP-OA owner-attestation JSON, injected into every signed event. | | `BUZZ_PATH` | Override when the `buzz` binary isn't on `PATH`. Defaults to `buzz`. | | `BUZZ_ALLOWED_PUBKEYS` | Two-way chat only. Comma-separated 64-char hex pubkeys allowed to talk to the agent. | | `BUZZ_GATEWAY_POLL_INTERVAL_SECONDS` | Two-way chat only. How often to check for mentions. Defaults to `15`. | | `BUZZ_GATEWAY_MAX_CONCURRENT` | Two-way chat only. Turns run in parallel. Defaults to `4`. | | `BUZZ_GATEWAY_AUTO_START` | Two-way chat only. Set `false` to keep the gateway from starting Buzz polling. | OpenSRE picks these up at startup and registers Buzz as an active integration. **Credential resolution.** `BUZZ_PRIVATE_KEY` resolves via store → env → OS keyring. The rest (relay URL, default channel, auth tag, binary path) stay plain env / store. *** ## Step 3: Verify Interactive shell: ```text theme={null} /integrations verify buzz ``` CLI: ```bash theme={null} opensre integrations verify buzz ``` This resolves the `buzz` binary, then runs `buzz channels list` against the configured relay. A missing binary or key reports **missing** with an install/setup hint; an unreachable relay or rejected key reports **failed**. You can also trigger a real delivery test against a bundled fixture: ```bash theme={null} opensre investigate --input tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` Findings should appear in the configured channel. Long reports are truncated to 4,096 characters. **Verify does not test two-way chat.** It confirms the relay accepts your key; it does not start a listening process. If you `@mention` the agent at this point, nothing will answer. Complete Step 4 for that. *** ## Step 4: Enable two-way chat > **Skip this step if you only need outbound delivery.** Steps 1–3 are sufficient for that. ### 4a — Give the agent a name and add it to the channel Run both with the agent's own `BUZZ_PRIVATE_KEY`: ```bash theme={null} buzz users set-profile --name opensre buzz channels join --channel ``` The name is what makes `@opensre` work: Buzz resolves `@Name` against the channel's member profiles to get a pubkey. Without one the agent shows as a raw hex key and is effectively unmentionable. Skipping the join is the most common cause of "the gateway is running but nothing happens" — the agent only sees messages that mention it, and only members can be mentioned. This is **not** the same as a Buzz *managed agent* (`buzz agents draft-create`). Those are agents Buzz itself runs — the record holds a system prompt, provider, and model, and Buzz drives the loop. OpenSRE runs its own agent loop and uses Buzz purely as a chat surface, so it joins as a normal identity. Mention delivery works the same either way. ### 4b — Allow your own pubkey ```text theme={null} /messaging allow -p buzz -u ``` CLI: ```bash theme={null} opensre messaging allow -p buzz -u ``` Or set in `.env`: ```bash theme={null} BUZZ_ALLOWED_PUBKEYS= # comma-separated for multiple people ``` Use the **64-character hex** public key, not the `npub1...` form. Inbound authorization matches the hex key that Nostr events actually carry, so an `npub` entry can never match a real sender. `opensre messaging allow -p buzz` rejects anything that isn't 64 hex characters. Your own hex key is the `pubkey` field of `buzz users get` (run with your key, no `--pubkey` flag), or your key generator's hex output. ### 4c — Start the gateway daemon ```bash theme={null} opensre gateway start ``` Then `@mention` the agent in the channel. Follow-up messages **in the same channel do not need a mention** — the agent keeps your session and picks up the thread. Each person gets their own private session, so two people in one channel never see each other's context. | You type | What happens | | ------------------------------------------ | -------------------------------------------------------- | | `@agent why is checkout-service erroring?` | Starts (or continues) your session | | `/new` | Drops your session history and starts fresh | | `/help` | Built-in command list | | `/pair ` | Completes pairing when your pubkey isn't allowlisted yet | Manage the daemon with `opensre gateway status`, `opensre gateway logs -f`, and `opensre gateway stop`. Buzz has no push socket here — the gateway polls the relay every 15 seconds (`BUZZ_GATEWAY_POLL_INTERVAL_SECONDS`). Expect up to that long before the agent starts typing. Restarting the gateway can re-deliver the single most recent mention; it never replays your whole history. ### Approving write actions Tools that change something ask before running. The agent posts a prompt in the channel and you answer by **replying to that message** with `approve` or `deny` — no mention needed, and no buttons (Buzz has none). * **Only the person whose request triggered it can answer**, in the channel it was posted to. Another channel member replying `approve` is ignored, even if they are allowlisted. * A reply that is neither approve nor deny (`hold on, checking`) leaves the prompt open rather than counting as a refusal. * Unanswered prompts expire after 3 minutes and the action is skipped. *** ## Watchdog alarms Buzz is a supported watchdog delivery provider alongside Telegram and Rocket.Chat: ```text theme={null} /watch --provider buzz --max-cpu 90 ``` Or from the CLI: ```bash theme={null} opensre watchdog --pid --provider buzz --max-cpu 90 ``` Alarms use the same per-threshold cooldown as the other providers (default 300s) and deliver to `BUZZ_DEFAULT_CHANNEL` unless `--chat-id ` overrides it. # Discord Source: https://opensre.com/docs/messaging/discord Two-way OpenSRE chat in Discord DMs, channels, and threads. OpenSRE's Discord integration supports **two-way gateway chat** (DMs, @mentions, active threads) and the **`/investigate` slash command**. The gateway connects over Discord's Gateway WebSocket with `discord.py` — you do **not** need a public `/discord/interactions` URL for normal operation. *** ## Prerequisites * A Discord server where you can add applications (**Manage Server**). * An OpenSRE host running `opensre gateway start` (local or deployed). * Discord **Privileged Gateway Intent: Message Content** enabled for the bot (required to read @mentions and thread replies in servers). *** ## Step 1: Create a Discord application and bot 1. Go to the [Discord Developer Portal](https://discord.com/developers/applications) and click **New Application**. 2. Open the **Bot** tab: * Copy the **Token** (`DISCORD_BOT_TOKEN`). * Enable **Message Content Intent** under **Privileged Gateway Intents**. 3. On **General Information**, copy **Application ID** and **Public Key** (still used by onboarding and optional HTTP tooling). *** ## Step 2: Invite the bot with the right permissions In **OAuth2 → URL Generator**, select scopes `bot` and `applications.commands`, then bot permissions: * **View Channels** * **Read Message History** * **Send Messages** * **Send Messages in Threads** * **Embed Links** * **Add Reactions** * **Use Application Commands** Open the generated URL and add the bot to your server. *** ## Step 3: Configure OpenSRE Run onboarding and select **Discord**, or set env vars: ```bash theme={null} opensre onboard ``` | Variable | Description | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `DISCORD_BOT_TOKEN` | Bot token | | `DISCORD_APPLICATION_ID` | Used to register `/investigate` | | `DISCORD_PUBLIC_KEY` | Stored for integrations; not required for gateway chat | | `DISCORD_DEFAULT_CHANNEL_ID` | Optional fallback channel for outbound-only delivery | | `DISCORD_ALLOWED_USERS` | Comma-separated Discord user snowflakes allowed to chat | | `DISCORD_ALLOW_OPEN_GUILD` | Set to `1` to allow any member in server channels (dogfood only). DMs still require the allowlist | | `DISCORD_SILO_GUILD_IDS` | Comma-separated server ids allowed to use this deployment's organization. Unset serves every server the bot is in | Discord turns are billed and stored against the organization in `ORGANIZATION_ID`. Without it every turn is refused — the bot appears online and silently does nothing, and the reason is only in the gateway log. Set `DISCORD_SILO_GUILD_IDS` on any deployment holding real credentials. Leaving it unset means any server that adds the bot inherits that organization's integrations. Approving a write action always requires `DISCORD_ALLOWED_USERS`. `DISCORD_ALLOW_OPEN_GUILD` opens chat, never approvals, so with no allowlist every Approve/Deny click is ignored. Allow yourself explicitly: ```bash theme={null} opensre messaging allow -p discord -u ``` Start the gateway: ```bash theme={null} opensre gateway start ``` When configured, status includes `discord: connected via gateway`. *** ## Step 4: Chat and investigate * **DM** the bot, **@mention** it in a channel, or reply in a **thread** where it is already engaged. * Slash command (registered at onboarding): ``` /investigate alert: ``` Gateway turns use the same agent session model as Slack and Telegram: `/new`, `/help`, and `/pair ` work in Discord text. Each member gets their own conversation, even in a shared channel: your history and `/new` never affect a teammate's. Integrations you connect are shared by the whole organization. *** ## Troubleshooting **Bot online but ignores channel messages** Enable **Message Content Intent** in the Developer Portal and restart the gateway. Ensure your user id is on the allowlist (`opensre messaging list` or `DISCORD_ALLOWED_USERS`). **`/investigate` missing** Re-run `opensre onboard` to re-register commands. Global commands can take up to an hour to propagate. **Gateway shows Discord not configured** Set `DISCORD_BOT_TOKEN` and at least one allowed user (or `DISCORD_ALLOW_OPEN_GUILD=1`). # Messaging Source: https://opensre.com/docs/messaging/index Deliver investigation findings and trigger investigations from Slack, Discord, Telegram, Rocket.Chat, WhatsApp, or Twilio SMS. OpenSRE can deliver investigation findings — and accept investigation triggers — through six messaging platforms. Pick the one that matches where your team already responds to incidents. ## Pick a platform | Platform | Best for | Trigger investigations from chat? | Configured by | Setup time | | ---------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------ | ---------- | | [**Slack**](/docs/messaging/slack) | Teams already paged in Slack channels | ✅ Socket Mode chat via `opensre gateway start` (+ webhook delivery) | `opensre integrations setup slack` | \~10 min | | [**Discord**](/docs/messaging/discord) | Communities and teams using Discord servers | ✅ Gateway chat + `/investigate` via `opensre gateway start` | `opensre onboard` or `opensre integrations setup discord` | \~10 min | | [**Telegram**](/docs/messaging/telegram) | Mobile-first and on-call rotations | ✅ DM text chat via `opensre gateway start` | `opensre onboard` or `opensre integrations setup telegram` | \~5 min | | [**Rocket.Chat**](/docs/messaging/rocketchat) | Self-hosted and open-source workspaces | ❌ delivery only | `opensre onboard` or `opensre integrations setup rocketchat` | \~5 min | | [**WhatsApp**](/docs/messaging/whatsapp) | Mobile alerting via Twilio WhatsApp | ❌ delivery only | `opensre integrations setup whatsapp` | \~5 min | | [**Twilio SMS**](/docs/messaging/twilio-sms) | SMS paging via Twilio (separate from WhatsApp) | ❌ delivery only | `opensre integrations setup twilio` | \~5 min | If you're not sure, start with Slack — it has the simplest setup. Pick Discord if you want chat-driven investigations with threaded follow-ups. Pick WhatsApp or Twilio SMS when your team already pages via Twilio. You can configure more than one. Each is independent and uses its own credentials. ## What every guide covers Each platform guide walks you through: 1. **Prerequisites** — what you need before you start (workspace/server admin rights, a phone number for Telegram, a Twilio account for WhatsApp, etc.). 2. **Create the bot/app** — the platform-side setup (BotFather, Discord Developer Portal, Slack App). 3. **Credentials** — the exact tokens, IDs, and keys to copy. 4. **OpenSRE wizard** — the `opensre onboard` flow, what it asks for, and what it writes to `.env`. 5. **Verify** — how to confirm delivery is working. 6. **Troubleshooting** — common failures and how to read the error. *** ## Common environment variables Discord and Telegram credentials can also be set directly in `.env` instead of (or alongside) the CLI configuration: ```bash theme={null} # Slack — webhook delivery and/or Socket Mode gateway SLACK_WEBHOOK_URL= SLACK_BOT_TOKEN= SLACK_APP_TOKEN= SLACK_ALLOWED_USERS= # Discord DISCORD_BOT_TOKEN= DISCORD_APPLICATION_ID= DISCORD_PUBLIC_KEY= DISCORD_DEFAULT_CHANNEL_ID= # Telegram TELEGRAM_BOT_TOKEN= TELEGRAM_DEFAULT_CHAT_ID= TELEGRAM_ALLOWED_USERS= # Rocket.Chat (token mode and/or webhook mode) ROCKETCHAT_SERVER_URL= ROCKETCHAT_AUTH_TOKEN= ROCKETCHAT_USER_ID= ROCKETCHAT_DEFAULT_CHANNEL= ROCKETCHAT_WEBHOOK_URL= # WhatsApp (Twilio) TWILIO_ACCOUNT_SID= TWILIO_AUTH_TOKEN= TWILIO_WHATSAPP_FROM= WHATSAPP_DEFAULT_TO= # Twilio SMS (shares TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN) TWILIO_SMS_FROM= TWILIO_SMS_MESSAGING_SERVICE_SID= TWILIO_SMS_DEFAULT_TO= ``` After editing `.env`, run `opensre integrations verify ` (e.g. `opensre integrations verify telegram`, `opensre integrations verify whatsapp`, or `opensre integrations verify twilio`) to confirm the credentials work. Note that `opensre doctor` only inspects the integrations store (`~/.opensre/integrations.json`), so it does **not** detect env-var-only configurations like Telegram, WhatsApp, Twilio SMS, or Slack-via-`SLACK_WEBHOOK_URL`. *** ## Re-running the wizard You can re-run the Discord wizard at any time to update credentials: ```bash theme={null} opensre onboard ``` Existing values are pre-populated, so you can press Enter to keep what you have. Slack, WhatsApp, and Twilio SMS use `opensre integrations setup slack`, `opensre integrations setup whatsapp`, and `opensre integrations setup twilio`. Telegram does not currently have a wizard step that persists credentials — use `.env` for Telegram (see [#1481](https://github.com/Tracer-Cloud/opensre/issues/1481)). # Rocket.Chat Source: https://opensre.com/docs/messaging/rocketchat Deliver investigation findings to a Rocket.Chat channel. OpenSRE's Rocket.Chat integration delivers investigation findings to any channel your user or bot account can post to — useful for teams running self-hosted or cloud Rocket.Chat workspaces. Start the interactive shell with `opensre` (no subcommand). Slash commands below are run from that REPL. Rocket.Chat support is **outbound delivery only** (investigation reports posted to a channel). Triggering investigations from Rocket.Chat chat is not supported yet. *** ## Prerequisites * A Rocket.Chat workspace (self-hosted or cloud) and its base URL, e.g. `https://chat.example.com`. * An account allowed to post in the destination channel. A dedicated bot account is recommended so findings are not attributed to a personal user. * The **Personal Access Tokens** feature enabled on the server (`Admin → Settings → Accounts → Personal Access Tokens`, on by default in most installs) — or admin access to create an **incoming webhook**. *** ## Pick a mode | Mode | Credentials | Destination | Best for | | --------------------------------------- | --------------------------------------- | ------------------------------------------------ | ------------------------------------------------------ | | **Personal Access Token** (recommended) | `server_url` + `auth_token` + `user_id` | Any channel, chosen per message | Dynamic channel targeting, verifiable via `/api/v1/me` | | **Incoming webhook** | single `webhook_url` | Fixed channel chosen when the webhook is created | Simplest setup; no bot account needed | You can configure both — when both are present, delivery prefers the webhook. *** ## Step 1 (token mode): Create a Personal Access Token 1. Sign in to Rocket.Chat with the account that should post findings. 2. Open **My Account → Personal Access Tokens** (avatar menu → **My Account**). 3. Enter a token name (e.g. `opensre`) and select **Add**. Leave *Ignore Two Factor Authentication* checked unless your policy requires otherwise. 4. Rocket.Chat shows the **token** and your **user ID** once. Copy both — the token is your `auth_token` and the ID is your `user_id`. Treat the token like a password. If you missed the user ID, it is also shown when you regenerate the token, or under `Admin → Users` for admins. *** ## Step 1 (webhook mode): Create an incoming webhook 1. As an admin, open **Administration → Workspace → Integrations → New → Incoming**. 2. Enable the integration, set a name (e.g. `opensre`), pick the destination **channel**, and choose the user the messages post as. 3. Select **Save**. Rocket.Chat shows the **Webhook URL** (`https:///hooks//`). Copy it — the token is embedded in the URL, so treat the whole URL like a password. *** ## Step 2 (token mode): Pick a destination channel Findings are posted with the standard `chat.postMessage` REST endpoint, so the destination accepts the same formats Rocket.Chat does: | Destination | Format | Example | | ------------------------ | --------------- | ------------ | | Public/private channel | `#channel-name` | `#incidents` | | Direct message to a user | `@username` | `@marcos` | Make sure the token's account is a member of the channel (or has permission to post there). *** ## Step 3: Configure the integration ### Option A: Onboarding wizard (recommended) Interactive shell: ```text theme={null} /onboard ``` CLI: ```bash theme={null} opensre onboard ``` Choose **Rocket.Chat** from the integration list, then pick **token**, **webhook**, or **both**. The wizard prompts for: * Token mode: **Server URL** (`ROCKETCHAT_SERVER_URL`), **personal access token** (OS keyring via `sync_env_secret`, not plain `.env`), **user ID** (`ROCKETCHAT_USER_ID`), and **default channel** (`ROCKETCHAT_DEFAULT_CHANNEL`) * Webhook mode: **webhook URL** — saved to the integration store (`~/.opensre/integrations.json`) and/or `ROCKETCHAT_WEBHOOK_URL` in `.env`; never written to the keyring (`*_URL` is non-keyring config). The URL often embeds a token — treat it like a password for logging/masking. Credentials are also saved to `~/.opensre/integrations.json` via `upsert_integration("rocketchat", ...)`. ### Option B: Environment variables Set in `.env` (PAT can also live in the keyring after wizard setup): ```bash theme={null} # Token mode ROCKETCHAT_SERVER_URL=https://chat.example.com ROCKETCHAT_AUTH_TOKEN= ROCKETCHAT_USER_ID= ROCKETCHAT_DEFAULT_CHANNEL=#incidents # Webhook mode (either mode alone is enough; both may be set) ROCKETCHAT_WEBHOOK_URL=https://chat.example.com/hooks// ``` | Variable | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ROCKETCHAT_SERVER_URL` | Base URL of your Rocket.Chat server. Required for token mode. | | `ROCKETCHAT_AUTH_TOKEN` | Personal access token. Required for token mode. Resolved via env then keyring. | | `ROCKETCHAT_USER_ID` | User ID shown alongside the token. Required for token mode. | | `ROCKETCHAT_DEFAULT_CHANNEL` | Default delivery destination (`#channel` or `@user`). Required for token-mode delivery. | | `ROCKETCHAT_WEBHOOK_URL` | Incoming webhook URL. Store/env only — never keyring (still treat like a password when logging). Enables webhook mode on its own; preferred over token mode when both are set. | OpenSRE picks these up at startup and registers Rocket.Chat as an active integration. **Credential resolution.** Token mode: store → `resolve_env_credential("ROCKETCHAT_AUTH_TOKEN")` (env then keyring). Webhook mode: store → plain `ROCKETCHAT_WEBHOOK_URL` env — never keyring. Server URL, user id, and default channel stay plain env / store. *** ## Step 4: Verify Interactive shell: ```text theme={null} /integrations verify rocketchat ``` Or: ```text theme={null} /verify rocketchat ``` CLI: ```bash theme={null} opensre integrations verify rocketchat ``` With token credentials configured, this calls Rocket.Chat's [`/api/v1/me`](https://developer.rocket.chat/apidocs/get-user-information) endpoint and reports the authenticated `@username`. With webhook-only configuration, it runs a **non-posting reachability probe** against the webhook URL (a 404 means the URL or its embedded token is wrong; no message is delivered by the probe). You can also trigger a real delivery test against a bundled fixture: Interactive shell: ```text theme={null} /investigate tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` CLI: ```bash theme={null} opensre investigate --input tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` Findings should appear in the configured channel as a message with an attachment titled **Investigation Complete**. Long reports are truncated to 4,096 characters. *** ## Programmatic messaging: `rocketchat_send_message` tool When the Rocket.Chat integration is configured, OpenSRE exposes a `rocketchat_send_message` tool. The tool can send user-requested action messages, incident notifications, or follow-up updates to the configured default channel, or to an explicit `channel` when one is supplied. Ask in plain language from the interactive shell (for example: "send a Rocket.Chat message to the team that DB CPU is back below 70%"). ```json theme={null} { "name": "rocketchat_send_message", "arguments": { "channel": "#incidents", "message": "DB CPU is back below 70%; keeping the incident open for 10 minutes." } } ``` The tool resolves credentials internally from the integration store, then the same env/keyring rules as setup (PAT via `resolve_env_credential`; webhook URL via plain env) — the agent never sees them. In token mode an explicit `channel` (or the configured `default_channel`) is targeted via `chat.postMessage`; in webhook-only mode messages go to the webhook's fixed destination, and an explicit `channel` returns a configuration error instead of being silently ignored. Delivery is an external side effect and requires approval. The result includes a stable `status`, `sent`, `error_type`, `channel`, and `message_length` shape so follow-up tool calls can tell configuration failures from Rocket.Chat delivery failures. *** ## Background RCA completion notifications When the interactive shell runs investigations in background mode, Rocket.Chat can deliver the RCA summary as soon as a job finishes. Configure the integration first, then in the shell: ```text theme={null} /background on /background notify set rocketchat # or: email,rocketchat /investigate datadog ``` On completion the summary — root cause, top analysis, next steps, and a short stats block — is posted as plain text, capped at 4,096 characters. Token mode posts to the configured `default_channel`; webhook-only setups deliver to the webhook's fixed destination. Confirm delivery with `/background show `: the `notify` row reads `rocketchat:sent`, `rocketchat:failed: `, or `rocketchat:missing rocketchat integration: …` when neither token credentials (with a default channel) nor a webhook is configured. A notification problem never fails the investigation itself. See [Background investigations](/docs/background-investigations) for the full command set. *** ## Watchdog alarms The process watchdog (`/watch`, `opensre watchdog`) can send threshold alarms to Rocket.Chat instead of Telegram: ```text theme={null} /watch --max-cpu 80 --provider rocketchat --chat-id "#alerts" ``` CLI: ```bash theme={null} opensre watchdog --pid --max-cpu 80 --provider rocketchat --chat-id "#alerts" ``` `--chat-id` overrides the configured `default_channel` (token mode) or is optional in webhook-only mode (the webhook's destination is fixed). Unlike scheduled deliveries (cron, Sentry digest), watchdog alarms accept **either** token credentials with a resolvable channel **or** a configured incoming webhook — there is no fixed `--chat-id` requirement forcing token-only mode here. Cooldown suppresses repeat alarms for the same threshold (`--cooldown`, default 5 minutes). *** ## Hermes incident escalation `opensre hermes watch` (live-tail Hermes error logs, classify incidents, escalate) can deliver to Rocket.Chat instead of Telegram: ```bash theme={null} opensre hermes watch --provider rocketchat --chat-id "#incidents" ``` Same credential rule as watchdog alarms above (token mode with a channel, or a configured webhook), and the same per-incident-fingerprint cooldown (`--cooldown-seconds`). See [Hermes](/docs/hermes) for the full command reference. *** ## Scheduled deliveries (cron) With token credentials configured (Steps 1–4), Rocket.Chat can receive recurring reports: Interactive shell: ```text theme={null} /cron add --kind daily_summary --cron "0 9 * * 1-5" --tz America/Sao_Paulo --provider rocketchat --chat-id "#ops" ``` CLI: ```bash theme={null} opensre cron add --kind daily_summary --cron "0 9 * * 1-5" \ --tz America/Sao_Paulo --provider rocketchat --chat-id "#ops" ``` `--chat-id` is the Rocket.Chat destination (`#channel` or `@user`). Scheduled deliveries always target that explicit destination, so they require **token credentials** — an incoming webhook's destination is fixed at creation time and cannot honor `--chat-id`. See [Cron](/docs/cron) for task kinds and scheduler daemon setup. *** ## Troubleshooting `/integrations verify rocketchat` only calls `/api/v1/me`, so it surfaces credential errors but cannot detect channel-routing problems. Delivery-time errors only show up when an investigation actually posts; they appear in OpenSRE logs as `[rocketchat] post message failed: `. **`Rocket.Chat auth failed: auth_token or user_id is invalid or expired.`** The token was revoked or the user ID does not match the token's account. Regenerate the token under **My Account → Personal Access Tokens** and update both values. **`error-room-not-found` (delivery-time)** The channel in `ROCKETCHAT_DEFAULT_CHANNEL` does not exist or the token's account cannot see it. Check the spelling (including the `#` prefix) and make sure the account is a member of the channel. **`Rocket.Chat API check failed: `** `ROCKETCHAT_SERVER_URL` is wrong or unreachable from the machine running OpenSRE. Confirm the URL opens in a browser and includes the scheme (`https://`). **`Rocket.Chat webhook returned 404; the URL looks invalid.`** The webhook was deleted, disabled, or the URL was copied incompletely. Open **Administration → Integrations**, confirm the incoming webhook is enabled, and copy the full URL again. **Findings never arrive, but `verify` passes** `/api/v1/me` only confirms token credentials and the webhook probe only confirms reachability; neither tests a real delivery. In token mode, confirm `ROCKETCHAT_DEFAULT_CHANNEL` is set — without it (and without a webhook), delivery is silently skipped (logged as `rocketchat delivery: skipped`). In webhook mode, confirm the webhook is enabled and posts to the channel you expect. # Slack Source: https://opensre.com/docs/messaging/slack Deliver investigation findings via webhook, and chat with the agent via Socket Mode. The Slack integration covers two surfaces: 1. **Incoming webhook** — outbound delivery of investigation findings to a channel. 2. **Socket Mode bot** — two-way chat with the OpenSRE agent (mentions and DMs), managed the same way as the [Telegram gateway](/docs/messaging/telegram#two-way-chat-gateway-dm-text). *** ## Prerequisites * A Slack workspace where you can create or install apps (workspace admin or app-install permissions). * The channel you want findings posted to (webhook) and/or where the bot will be invited (Socket Mode). *** ## Step 1: Create a Slack incoming webhook 1. Visit [https://api.slack.com/apps](https://api.slack.com/apps) and click **Create New App → From scratch**. 2. Name the app (e.g. `OpenSRE`) and pick your workspace. 3. In the left sidebar, open **Incoming Webhooks** and toggle the feature **On**. 4. Click **Add New Webhook to Workspace**. 5. Pick the channel where findings should be posted and click **Allow**. 6. Copy the generated URL. It has three path segments — a workspace ID, a channel/app binding ID, and a per-webhook secret — for example: ```text theme={null} https://hooks.slack.com/services/// ``` Treat this URL like a password — anyone holding it can post to your channel. If your workspace already has a Slack app you want to reuse, you can add a new webhook to it instead of creating a fresh app; the URL format is the same. *** ## Step 2: Configure the integration in OpenSRE You have two equivalent paths: ```bash theme={null} opensre integrations setup slack ``` Choose **webhook**, **Socket Mode**, or **both**. Credentials are persisted to `~/.opensre/integrations.json` (and can be merged on re-run). Add to `.env`: ```bash theme={null} SLACK_WEBHOOK_URL=https://hooks.slack.com/services/// # Optional Socket Mode gateway: SLACK_BOT_TOKEN=xoxb-… SLACK_APP_TOKEN=xapp-… SLACK_ALLOWED_USERS=U0123ABCD ``` OpenSRE reads env as a fallback when no Slack entry exists in `~/.opensre/integrations.json`. Bot / app tokens use `resolve_env_credential` (process env, then OS keyring). `SLACK_WEBHOOK_URL` is store/env only — never keyring. ```bash theme={null} opensre onboard ``` Select **Slack** and choose webhook / Socket Mode / both. The wizard validates the webhook when configured, then persists credentials to the integration store. Socket Mode tokens are dual-written to the OS keyring; the webhook URL is not. **Credential resolution.** Store first when present. Then: `SLACK_BOT_TOKEN` / `SLACK_APP_TOKEN` → env then keyring; `SLACK_WEBHOOK_URL` → plain env only (never keyring). *** ## Step 3: Verify ```bash theme={null} opensre integrations verify slack ``` A successful run reports the integration as `passed`. Webhook-only configs confirm the URL is present; Socket Mode configs also run Slack `auth.test` on the bot token. To also confirm webhook delivery, add the `--send-slack-test` flag: ```bash theme={null} opensre integrations verify slack --send-slack-test ``` This posts a small test message to the configured channel. Expected failure modes are listed below. You can also trigger a real investigation against a bundled fixture: ```bash theme={null} opensre investigate --input tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` Findings should appear in the configured channel. *** ## Agent Slack tools Teammate tools (bot token) plus webhook blast. Credentials resolve inside the tools (never in tool-call traces). Shared client: `integrations/slack/web_client.py`. | Tool | What it does | Needs | Approval | | ------------------------- | ---------------------------------------------- | -------------------------------- | -------- | | `slack_send_message` | Post to the webhook's fixed channel | `SLACK_WEBHOOK_URL` | Yes | | `slack_reply_message` | Post to any channel/thread (`C…` or `#name`) | bot token, `chat:write` | Yes | | `slack_read_messages` | Read channel history or a thread (`thread_ts`) | bot token, history + list scopes | No | | `slack_search_messages` | Workspace message search | bot token, `search:read` | No | | `slack_list_team_members` | List workspace members | bot token, `users:read` | No | | `slack_join_channel` | Join a public channel | bot token, `channels:join` | Yes | | `slack_add_reaction` | React to a message ts | bot token, `reactions:write` | No | Prefer `slack_reply_message` / `slack_read_messages` when the bot token is configured and the user names a channel or thread. Use `slack_send_message` only for the fixed webhook channel. For "add task", "remind me", and "what should we focus on next" requests, use OpenSRE work management. Gateway turns can default reminders and recurring check-ins to the current Slack channel. Example prompts: * "Read the last 20 messages in #incidents and summarize." * "Search Slack for 'Windows install' this week." * "Join #ops then reply that mitigation is rolled out." * "Who is on the team, and what does Vaibhav do?" * "Add task: ping owners about the deploy window." Add the scopes in the table under **Create Socket Mode tokens**, then **Reinstall to Workspace**. Invite the bot (`/invite @OpenSRE`) to private channels it should read or post in. *** ## Production Engineer schedules (Slack) Wire morning digests and PR sweeps to a Slack channel (`C…` id from channel details). The gateway daemon must be running so the scheduler can deliver. **Sentry morning digest** (unresolved issues → Slack): ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * 1-5" \ --tz Europe/London \ --provider slack \ --chat-id C0123ABCD opensre sentry digest schedule run # dogfood once ``` **Daily ops summary** (investigation pipeline → Slack): ```bash theme={null} opensre cron add --kind daily_summary --cron "0 9 * * 1-5" \ --tz Europe/London --provider slack --chat-id C0123ABCD opensre cron run ``` **GitHub PR sweep** (mergeable / stale / conflicted → Slack; needs GitHub configured): ```bash theme={null} opensre integrations verify github opensre cron add --kind github_pr_sweep --cron "0 9 * * 1-5" \ --tz Europe/London --provider slack --chat-id C0123ABCD opensre cron run ``` See also [Scheduled deliveries](/docs/cron) and [Sentry morning digest](/docs/sentry#morning-digest-scheduled). *** ## Two-way chat gateway (Socket Mode) OpenSRE can also run a **Slack messaging gateway** so you can chat with the agent from Slack mentions or DMs. Each thread is its own conversation. No public inbound HTTPS URL is required — the gateway holds an outbound websocket. ### Create Socket Mode tokens 1. In your Slack app settings, enable **Socket Mode** and create an app-level token with the `connections:write` scope (`xapp-…`). 2. Under **OAuth & Permissions**, grant the bot scopes below, then install the app and copy the bot token (`xoxb-…`). 3. Under **Event Subscriptions**, subscribe to the bot events `app_mention` and `message.im`. **Bot Token Scopes (Socket Mode chat + teammate tools):** | Scope | Needed for | | --------------------- | ----------------------------------------------------------------------- | | `app_mentions:read` | Gateway `@mention` inbound | | `chat:write` | Gateway replies + `slack_reply_message` | | `im:history` | Gateway DMs + reading DM history | | `channels:history` | `slack_read_messages` + thread seeding in public channels | | `groups:history` | `slack_read_messages` in private channels | | `mpim:history` | `slack_read_messages` in multi-party DMs | | `groups:read` | Resolve private channel names | | `groups:write` | Manage / create private channels | | `files:read` | Inbound file-attachment downloads + discover Slack Lists (`files.list`) | | `files:write` | Upload files as the bot | | `lists:read` | `slack_read_list` — read Slack List rows (`slackLists.items.list`) | | `search:read.public` | `slack_search_messages` in public channels | | `search:read.private` | Search private channels | | `search:read.im` | Search DMs | | `search:read.mpim` | Search group DMs | | `search:read.files` | Search files | | `search:read.users` | Search users | | `usergroups:read` | View workspace user groups | | `users.profile:read` | Speaker profile details | | `users:read` | `slack_list_team_members` + user lookup | Add these only if you use the feature — they are not in the current app config: `reactions:write` (gateway ack 👀 / ✓ reactions + `slack_add_reaction`), `channels:read` / `im:read` / `mpim:read` (resolve `#channel-name` and DM names → ID), and `channels:join` (`slack_join_channel`). Optional env: `SLACK_TEAM_TASKS_LIST_ID` (`F…`) so `slack_read_list` can open the default team-tasks List without a name search. Reinstall the app after adding scopes so the bot token picks them up. ### Allow your Slack user Find your Slack **member ID** (`U…`) in the Slack app: 1. Open your profile (avatar / name). 2. Click **⋯** (More) next to **View as** / profile actions. 3. Choose **Copy member ID**. Then allow that id: Interactive shell: ```text theme={null} /messaging allow -p slack -u U0123ABCD ``` CLI: ```bash theme={null} opensre messaging allow -p slack -u U0123ABCD ``` or set `SLACK_ALLOWED_USERS=U0123ABCD` in `.env`. The integration store takes precedence when both are set. Use the Slack **member ID** (`U…`) from **Copy member ID**, not `@display-name` (e.g. not `@Yauhen`). Handles can be reassigned; inbound authorization only matches stable user IDs. For dogfood only, you may set `SLACK_ALLOW_OPEN_WORKSPACE=1` instead (any workspace member can talk to the bot). ### DM pairing (optional) Same policy as Telegram. Generate a code: ```bash theme={null} opensre messaging pair -p slack ``` Then DM the bot (or mention it) and send: ```text theme={null} /pair ``` Check / revoke: ```bash theme={null} opensre messaging status -p slack opensre messaging revoke -p slack -u U0123ABCD ``` ### Start the gateway daemon ```bash theme={null} opensre gateway start opensre gateway status # want: slack: connected via socket mode ``` | Command | What it does | | ------------------------- | -------------------------------------------------- | | `opensre gateway start` | Start the daemon (web, Telegram, Slack, scheduler) | | `opensre gateway status` | Show daemon and component state | | `opensre gateway logs -f` | Follow live gateway logs | | `opensre gateway stop` | Stop the daemon | Built-in chat commands: `/new` (fresh session), `/help`, `/pair `. While a turn runs, the gateway adds an 👀 (`eyes`) reaction on the inbound message, then swaps to ✅ (`white_check_mark`) when the reply is finalized. If Slack is not configured the daemon still runs the other components and `opensre gateway status` shows `slack: not configured`. ### Deploying the Slack gateway The Slack gateway backend is deployed and operated separately — not from this repo. Configure the integration here (tokens and scopes above); provisioning the hosted backend is out of scope for this repo. The EC2 gateway path (`make deploy-gateway`) is **Telegram-only** and ignore `SLACK_*` variables with a validation warning: Slack Socket Mode is single-consumer, so a second gateway holding the same tokens would split events. *** ## Environment variables | Variable | Description | | ---------------------------- | ------------------------------------------------------------------------------------------- | | `SLACK_WEBHOOK_URL` | Incoming webhook URL. Required for outbound delivery. | | `SLACK_BOT_TOKEN` | Bot token (`xoxb-…`) for the two-way Slack bot. Env or integration store. | | `SLACK_APP_TOKEN` | App-level token (`xapp-…`) for Socket Mode. Env or integration store. | | `SLACK_ALLOWED_USERS` | Comma-separated Slack user IDs allowed to talk to the bot (required unless open workspace). | | `SLACK_ALLOW_OPEN_WORKSPACE` | Set to `1` to allow any workspace member (dogfood escape hatch). | *** ## Troubleshooting **`error: webhook_url is required.` from `opensre integrations setup slack`** You chose webhook (or both) and left the URL empty. Re-run and paste the full URL including the `https://` prefix, or choose Socket Mode only. **`slack: not configured` from `opensre gateway status`** Missing `SLACK_BOT_TOKEN` / `SLACK_APP_TOKEN` (env or store) or empty allowlist without `SLACK_ALLOW_OPEN_WORKSPACE=1`. Run `opensre messaging allow -p slack -u ` then `gateway stop` / `start`. **Connected, but deny reply** Your `U…` is not in `SLACK_ALLOWED_USERS` / store allowlist. Pair with `/pair ` after `opensre messaging pair -p slack`, or add yourself with `messaging allow`. **`invalid_payload` or `channel_not_found` from Slack** The webhook URL was created against a channel that has since been archived or renamed in a way that broke the binding. Create a new webhook in the Slack app settings and replace `SLACK_WEBHOOK_URL`. **Findings posted to the wrong channel** A webhook is bound to the channel it was created against. To change channels, create a new webhook in Slack pointed at the new channel and update `SLACK_WEBHOOK_URL`. **Webhook returns `no_service`** The Slack app or webhook was deleted. Re-create it and update the URL. # Telegram Source: https://opensre.com/docs/messaging/telegram Deliver investigation findings to a Telegram chat or channel. OpenSRE's Telegram integration delivers investigation findings to any chat your bot has been added to — useful for mobile-first on-call rotations and personal alerting. Start the interactive shell with `opensre` (no subcommand). Slash commands below are run from that REPL. *** ## Prerequisites * A Telegram account. * The Telegram mobile or desktop app, signed in. * The chat (group, channel, or direct message) where you want to receive findings. *** ## Step 1: Create a bot with BotFather [BotFather](https://t.me/BotFather) is Telegram's official bot for creating other bots. 1. Open Telegram and search for `@BotFather`. Open the chat and tap **Start**. 2. Send `/newbot`. 3. When prompted, send a **display name** for your bot (e.g. `OpenSRE Alerts`). 4. Send a **username** that ends in `bot` (e.g. `opensre_alerts_bot`). It must be globally unique. 5. BotFather replies with an **HTTP API token** of the form `:`. Copy it — it is your **bot token**. Treat it like a password. Anyone holding it can send messages as your bot. You can change the bot name, picture, and description later by sending `/mybots` to BotFather and selecting your bot. *** ## Step 2: Add the bot to a chat The bot can deliver to three kinds of destinations. Pick the one that fits your team: 1. Open the group where you want findings to land. 2. Tap the group name → **Add members** → search for your bot username → **Add**. 3. By default, bots in groups only see messages addressed to them, which is fine for delivery-only. 1. Open your channel and tap its name. 2. Tap **Administrators → Add Administrator**, search for your bot, and add it. 3. Grant the **Post Messages** permission. No other admin permissions are required. Open a chat with your bot directly (search its username) and send `/start`. The bot will not reply, but Telegram registers the chat so the bot can message you. *** ## Step 3: Find your `chat_id` The **chat ID** identifies where the bot should post. It is required — without it the bot has nowhere to send anything. **Posting to a public channel?** Skip this step — setup accepts the channel's `@name` (for example `@acme_alerts`) directly. Private groups and DMs have no `@name`, so they need the steps below. 1. Send any message in the destination chat — for a channel, post anything; for a DM, send `/start` to your bot. 2. In a browser, open: ``` https://api.telegram.org/bot/getUpdates ``` (replace `` with the token from Step 1) 3. In the JSON response, look for a `chat.id` field. The value depends on the chat type: | Chat type | Format | Example | | -------------------------- | ------------------------------------- | ---------------- | | Direct message with a user | Positive integer | `123456789` | | Group | Negative integer | `-987654321` | | Large group or channel | Negative integer starting with `-100` | `-1001234567890` | Copy the entire value, **including the leading minus sign** for groups and channels. If `getUpdates` returns an empty array, post a fresh message in the chat and reload — Telegram only buffers recent updates. *** ## Step 4: Configure the integration ### Option A: Onboarding wizard (recommended) Interactive shell: ```text theme={null} /onboard ``` CLI: ```bash theme={null} opensre onboard ``` Choose **Telegram** from the integration list. The wizard prompts for: * **Bot token** (required) — stored in the system keyring (not plain `.env`) * **Default chat ID or `@channelname`** (required) — written to `.env` as `TELEGRAM_DEFAULT_CHAT_ID` Credentials are also saved to `~/.opensre/integrations.json` via `upsert_integration("telegram", ...)`. Both answers are checked before anything is saved, so a wrong token or a chat the bot was never added to fails here rather than silently at the first alert. Setup confirms the bot can *see* the chat, not that it may post there. For a channel the bot still needs the **Post Messages** permission from Step 2. Non-interactive setup: Interactive shell: ```text theme={null} /integrations setup telegram ``` CLI: ```bash theme={null} opensre integrations setup telegram ``` ### Option B: Environment variables Set in `.env` (bot token can also live in the keyring after wizard setup): ```bash theme={null} TELEGRAM_BOT_TOKEN=: TELEGRAM_DEFAULT_CHAT_ID= ``` | Variable | Description | | -------------------------- | ------------------------------------------------------------ | | `TELEGRAM_BOT_TOKEN` | Bot HTTP API token from BotFather. Required. | | `TELEGRAM_DEFAULT_CHAT_ID` | Default delivery destination. Required for delivery to work. | OpenSRE picks these up at startup and registers Telegram as an active integration. **Credential resolution.** Every Telegram delivery surface — investigations, **background RCA completion notifications**, the scheduler, `/watchdog`, `/hermes watch`, and `/watch` — resolves the bot token the same way: integration store first, then `resolve_env_credential("TELEGRAM_BOT_TOKEN")` (process env, then OS keyring). Chat id is non-secret: `--chat-id` → store `default_chat_id` → `TELEGRAM_DEFAULT_CHAT_ID` env (plain `os.getenv`, never keyring). Either setup option above works for all of them. *** ## Step 5: Verify Interactive shell: ```text theme={null} /integrations verify telegram ``` Or: ```text theme={null} /verify telegram ``` CLI: ```bash theme={null} opensre integrations verify telegram ``` This calls Telegram's [`getMe`](https://core.telegram.org/bots/api#getme) endpoint. On success it reports the bot `@username`. On failure it reports the Telegram API error message verbatim. **Verify only checks that your bot token is valid.** It does not start a listening process and does not test two-way communication. If you DM your bot at this point, it will appear online but not reply. To receive and respond to messages from Telegram, complete Step 6 below. You can also trigger a real delivery test against a bundled fixture: Interactive shell: ```text theme={null} /investigate tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` CLI: ```bash theme={null} opensre investigate --input tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` Findings should appear in the configured chat. Long reports are truncated to Telegram's 4,096-character message limit. *** ## Step 6: Enable two-way chat (DM the agent from Telegram) > **Skip this step if you only need outbound delivery** (alerts, cron reports, investigation findings posted to a chat). Steps 1–5 are sufficient for that. After verify succeeds, you have outbound delivery but the bot will not reply to your messages yet. Two extra steps are required: ### 6a — Allow your Telegram user Find your **numeric user id** with [@userinfobot](https://t.me/userinfobot), then: Interactive shell: ```text theme={null} /messaging allow -p telegram -u 123456789 ``` CLI: ```bash theme={null} opensre messaging allow -p telegram -u 123456789 ``` Or set in `.env`: ```bash theme={null} TELEGRAM_ALLOWED_USERS=123456789 # comma-separated for multiple users ``` Use the **numeric user id** (Telegram's `from.id`), not a `@username` and not the bot handle. Inbound authorization only ever matches the numeric id, so a `@handle` produces an allow-list entry that can never match a real sender — you will see `User is not in the allowed users list` for every message. The CLI now rejects non-numeric Telegram ids, but older entries may still be wrong; check with `/messaging status -p telegram` (or `opensre messaging status -p telegram`) and re-add with the numeric id. ### 6b — Start the gateway daemon ```bash theme={null} opensre gateway start ``` This runs the background daemon that listens for inbound Telegram DMs and routes them to the OpenSRE agent. Once it is running, DM your bot from Telegram — use `/new` for a fresh session or `/help` for built-in commands. Useful follow-up commands: | Command | What it does | | ------------------------- | ------------------------------- | | `opensre gateway status` | Show daemon and component state | | `opensre gateway logs -f` | Follow live gateway logs | | `opensre gateway stop` | Stop the daemon | The gateway uses long polling — no public HTTPS URL or port forwarding is required for local use. See the [Two-way chat gateway](#two-way-chat-gateway-dm-text) section below for deployment and remote-host setup. *** ## Programmatic messaging: `telegram_send_message` tool When the Telegram integration is configured, OpenSRE exposes a `telegram_send_message` tool. The tool can send user-requested action messages, incident notifications, or follow-up updates to the configured default chat, or to an explicit `chat_id` when one is supplied. Ask in plain language from the interactive shell (for example: "send a Telegram message to the team that DB CPU is back below 70%"). ```json theme={null} { "name": "telegram_send_message", "arguments": { "chat_id": "-1001234567890", "message": "DB CPU is back below 70%; keeping the incident open for 10 minutes." } } ``` The tool resolves the bot token from the same credential chain as the watchdog: integration store, then `resolve_env_credential("TELEGRAM_BOT_TOKEN")` (env then keyring). If `chat_id` is omitted, it sends through the configured `default_chat_id`, so a user can ask OpenSRE to send a message through the configured Telegram bot without knowing or exposing the bot token. Delivery is an external side effect. Its result includes a stable `status`, `sent`, `error_type`, `chat_id`, `reply_to_message_id`, and `message_length` shape so follow-up tool calls can tell configuration failures from Telegram delivery failures. *** ## Two-way chat gateway (DM text) OpenSRE can also run a **Telegram messaging gateway** so you can chat with the agent from your phone in a private DM. v1 supports **text-only direct messages** (no groups, voice, or attachments). ### Allow your Telegram user Find your **numeric user id** with [@userinfobot](https://t.me/userinfobot), then: Interactive shell: ```text theme={null} /messaging allow -p telegram -u 123456789 ``` CLI: ```bash theme={null} opensre messaging allow -p telegram -u 123456789 ``` or set `TELEGRAM_ALLOWED_USERS=123456789` in `.env`. Use the **numeric user id** (Telegram's `from.id`), not a `@username` and not the bot handle. Inbound authorization only ever matches the numeric id, so a `@handle` produces an allow-list entry that can never match a real sender — you will see `User is not in the allowed users list` for every message. The CLI now rejects non-numeric Telegram ids, but older entries may still be wrong; check with `/messaging status -p telegram` (or `opensre messaging status -p telegram`) and re-add with the numeric id. ### DM pairing (optional) Same policy as other messaging platforms. Generate a code: Interactive shell: ```text theme={null} /messaging pair -p telegram ``` CLI: ```bash theme={null} opensre messaging pair -p telegram ``` Then open Telegram, DM your bot, and send: ```text theme={null} /pair ``` Check pairing status: Interactive shell: ```text theme={null} /messaging status -p telegram ``` CLI: ```bash theme={null} opensre messaging status -p telegram ``` Revoke access for a user: Interactive shell: ```text theme={null} /messaging revoke -p telegram -u 123456789 ``` CLI: ```bash theme={null} opensre messaging revoke -p telegram -u 123456789 ``` ### Long polling (local and production) No public HTTPS URL required. The gateway is OpenSRE's **background agent daemon** — one process that runs the Telegram chat worker, the web health app, and the cron task scheduler. It runs in the background by default: | Command | What it does | | ---------------------------------- | ----------------------------------------------------- | | `opensre gateway start` | Start the daemon (web, Telegram chat, task scheduler) | | `opensre gateway start -f` | Run attached to the terminal instead | | `opensre gateway status` | Show the daemon and each component's state | | `opensre gateway logs [-n N] [-f]` | Print (or follow) the daemon logs | | `opensre gateway stop` | Stop the daemon | ```bash theme={null} export TELEGRAM_BOT_TOKEN= uv run opensre gateway start ``` Starting prints the log location; logs are stored in `~/.opensre/gateway/gateway.log`. If Telegram is not configured the daemon still runs the other components and `opensre gateway status` shows `telegram: not configured`. The same controls are available inside the interactive shell via `/gateway`: ```text theme={null} /gateway start # start the background daemon /gateway status # daemon + per-component state /gateway logs 50 # last 50 log lines /gateway stop # stop it ``` Send a DM to your bot. Use `/new` to start a fresh session; `/help` for built-in commands. The gateway uses the same headless agent harness as the interactive shell, with Telegram-specific wiring: * **Prompt grounding** — CLI reference, AGENTS.md, integration list, and investigation flow context * **Read-only evidence tools** — live integration queries (GitHub, logs, metrics, etc.) via the gather pass when integrations are configured * **Action tools** — `shell_run` and `investigation_start` Investigation delivery, background RCA completion notifications, watchdog alerts, and cron still use the outbound-only paths documented above — they do not require the gateway process. ### Deploying the gateway to a remote host Running the gateway on a server with `make deploy-gateway` (EC2) is different from local use in one important way: **the remote host cannot read your local machine's keychain**. Guided setup stores the bot token (and your LLM API key) in the system keyring, which is perfect locally but does not travel to the deployed instance. So `make deploy-gateway` validates that the required secrets are present as **plaintext env vars** in `.env`, and aborts if they are only in the keychain: ```bash theme={null} TELEGRAM_BOT_TOKEN=: TELEGRAM_ALLOWED_USERS=123456789 # numeric user id(s), comma-separated ANTHROPIC_API_KEY=... # or your provider's key env ``` If `make deploy-gateway` reports `MISSING: TELEGRAM_BOT_TOKEN — API key not set` or `MISSING: OPENAI_API_KEY — API key not set` (or your provider's key env) even though local setup succeeded, this is why — copy the values into `.env` (or `.env.deploy.example`) for the deploy. A "missing" here means "not in the deploy env", not "not configured". *** ## Scheduled, background, and watchdog delivery ### Background investigation completion (RCA) When the interactive shell runs investigations in background mode, Telegram can deliver the RCA summary as soon as a job finishes. Configure Telegram first (Steps 4–5 — the gateway in Step 6 is **not** required), then in the shell: ```text theme={null} /background on /background notify set telegram # or: email,telegram /investigate datadog ``` Keep using the shell while the RCA runs. On completion the summary — root cause, top analysis, next steps, and a short stats block — is posted as plain text to your default chat, truncated to Telegram's 4,096-character limit. Confirm delivery with `/background show `: the `notify` row reads `telegram:sent`, `telegram:failed: `, or `telegram:missing telegram integration: …` when the bot token or chat id is not configured. A notification problem never fails the investigation itself. See [Background investigations](/docs/background-investigations) for the full command set. ### Cron (recurring reports) Interactive shell: ```text theme={null} /cron add --kind daily_summary --cron "0 9 * * 1-5" --tz Asia/Kolkata --provider telegram --chat-id /cron list /cron run /cron logs ``` CLI: ```bash theme={null} opensre cron add --kind daily_summary --cron "0 9 * * 1-5" \ --tz Asia/Kolkata --provider telegram --chat-id opensre cron list opensre cron run opensre cron logs ``` See [Cron](/docs/cron) for task kinds and scheduler daemon setup. ### Watchdog (process threshold alarms) Configure Telegram first (Steps 4–5), then: Interactive shell: ```text theme={null} /watch [--max-cpu N] [--max-runtime D] [--max-rss S] [--cooldown D] [--interval N] [--once] /watches /unwatch ``` Or: ```text theme={null} /watchdog --pid [--max-cpu N] [--max-runtime D] [--max-rss S] ``` CLI: ```bash theme={null} opensre watchdog --pid [--max-cpu N] [--max-runtime D] [--max-rss S] ``` ### Hermes incident escalation Interactive shell: ```text theme={null} /hermes watch ``` CLI: ```bash theme={null} opensre hermes watch ``` See [Hermes](/docs/hermes) for log tailing and incident classification setup. *** ## Troubleshooting `/integrations verify telegram`, `/verify telegram`, and `opensre integrations verify telegram` only call Telegram's `getMe` endpoint, so they surface token-validity errors but cannot detect chat-routing problems. Delivery-time errors only show up when an investigation actually posts. ### Errors from verify **`Missing bot_token`** `TELEGRAM_BOT_TOKEN` is empty. Re-check `.env` and restart any long-running OpenSRE process so it re-reads the file. **`Telegram API check failed: 401 Client Error: Unauthorized for url: …`** The bot token is invalid or has been revoked. Generate a new one in BotFather (`/mybots → your bot → API Token → Revoke current token`) and update `.env`. The `for url:` portion of the message shows the token as `` — Telegram's API endpoint embeds the token in the path, so the verifier scrubs it before surfacing the error. ### Errors that only surface during delivery These are Telegram API responses that come back when OpenSRE actually tries to post a finding. `verify` only calls `getMe`, so it cannot catch them. They appear in OpenSRE logs as `[telegram] post message failed: ` with the Telegram description copied verbatim. **`description: chat not found`** The bot is not in the chat, or `TELEGRAM_DEFAULT_CHAT_ID` is wrong. Re-add the bot and re-fetch `chat_id` from `getUpdates`. **`description: bot was kicked from the large group …` (or similar)** Re-add the bot. For channels, the bot must be an administrator with **Post Messages** permission. **Findings never arrive, but `verify` passes** `getMe` only confirms the token is valid; it does not test delivery. Send a fresh message in the destination chat and re-fetch `chat_id` from `getUpdates` — your `chat_id` may have changed (for example, if a group was upgraded and now uses a `-100` prefix). **Gateway: `User is not in the allowed users list`** Run `/messaging status -p telegram` and confirm the sender's numeric user id is listed. Re-add with `/messaging allow -p telegram -u ` or complete `/messaging pair -p telegram` pairing. # Twilio SMS Source: https://opensre.com/docs/messaging/twilio-sms Deliver investigation findings via Twilio SMS as a standalone, independently configurable channel. OpenSRE's Twilio integration delivers investigation findings via **SMS** using the Twilio Messaging API. It is configured independently of the existing `whatsapp` integration, though both can share the same Twilio account credentials. WhatsApp delivery is owned entirely by the separate `whatsapp` integration (`opensre integrations setup whatsapp`) — see the [WhatsApp](/docs/messaging/whatsapp) page. The `twilio` integration documented here is SMS-only. For voice paging on critical incidents, route through PagerDuty or Opsgenie, which provide voice escalation with proper acknowledgement. *** ## Prerequisites * A Twilio account: [Sign up](https://www.twilio.com/try-twilio). * A Twilio-provisioned phone number **or** a Messaging Service SID. *** ## Step 1: Configure the integration ### Via CLI wizard (recommended) ```bash theme={null} opensre integrations setup twilio ``` You'll be prompted for: * **Twilio Account SID** (starts with `AC...`) * **Twilio Auth Token** * **Twilio SMS From number** (E.164, e.g. `+14155551234`) — or leave blank and provide a **Messaging Service SID** (starts with `MG...`) * optional **default recipient** ### Via environment variables Add to your `.env` file: ```bash theme={null} TWILIO_ACCOUNT_SID=AC... TWILIO_AUTH_TOKEN=your_auth_token # Set either TWILIO_SMS_FROM or TWILIO_SMS_MESSAGING_SERVICE_SID TWILIO_SMS_FROM=+14155551234 TWILIO_SMS_MESSAGING_SERVICE_SID= TWILIO_SMS_DEFAULT_TO=+1234567890 ``` The `twilio` env-bootstrap activates when the account + token are set and an SMS sender (`TWILIO_SMS_FROM` or `TWILIO_SMS_MESSAGING_SERVICE_SID`) is present. The legacy `whatsapp` record is bootstrapped independently from `TWILIO_WHATSAPP_FROM`. *** ## Step 2: Verify ```bash theme={null} opensre integrations verify twilio ``` A passing verification confirms: * The Twilio account credentials authenticate against the Twilio Account API. * The SMS channel is enabled and has a usable sender (`from_number` or `messaging_service_sid`). Sample passing output: ``` twilio: passed — Connected to Twilio account My Company; SMS channel ready. ``` If the account authenticates but the SMS channel has no sender, verification returns `failed` with a message telling you to set a `from_number` or `messaging_service_sid`. *** ## Step 3: Test with an investigation Trigger a real investigation against a bundled fixture: ```bash theme={null} opensre investigate --input tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` When the investigation finishes, the RCA summary is truncated to the SMS body limit (1600 chars) and posted via the Twilio Messaging API to the configured recipient. *** ## Programmatic notifications: `twilio_notify` tool When the Twilio integration is configured, the investigation planner exposes a `twilio_notify` tool. The tool sends a short notification body via SMS and returns the Twilio Message SID for traceability. ```json theme={null} { "name": "twilio_notify", "arguments": { "to": "+14155550000", "body": "DB CPU > 95% — paging on-call" } } ``` *** ## Troubleshooting ### `opensre integrations verify twilio` errors | Detail | Likely cause | | ------------------------------ | ---------------------------------------------------------------------------- | | `Missing account_sid` | `TWILIO_ACCOUNT_SID` is unset. | | `Missing auth_token` | `TWILIO_AUTH_TOKEN` is unset. | | `Twilio API check failed: 401` | The SID/token pair is invalid. | | `SMS channel is not ready` | No SMS sender — set `TWILIO_SMS_FROM` or `TWILIO_SMS_MESSAGING_SERVICE_SID`. | ### SMS never arrives but verify passes * Confirm `TWILIO_SMS_FROM` (or the Messaging Service SID) is provisioned for the destination country. * Confirm the recipient is in E.164 format (`+14155550000`). * For trial accounts: the recipient must be a verified caller ID. * Inspect `Messages` in the Twilio Console for an error code; common failures (`30007`, `21610`, etc.) are documented at [twilio.com/docs/api/errors](https://www.twilio.com/docs/api/errors). # WhatsApp Source: https://opensre.com/docs/messaging/whatsapp Deliver investigation findings to WhatsApp via Twilio. OpenSRE's WhatsApp integration delivers investigation findings to any WhatsApp number via Twilio's Messaging API. This integration uses Twilio as the WhatsApp provider. *** ## Prerequisites * A Twilio account: [Sign up](https://www.twilio.com/try-twilio) * Twilio WhatsApp Sandbox (for demos) or a production WhatsApp-enabled sender *** ## Step 1: Configure the integration ### Via CLI wizard (recommended) ```bash theme={null} opensre integrations setup whatsapp ``` You'll be prompted for: * **Twilio Account SID** (starts with `AC...`) * **Twilio Auth Token** * **Twilio WhatsApp From number** (for example `whatsapp:+14155238886`) * **Default recipient phone number** (optional) ### Via environment variables Add to your `.env` file: ```bash theme={null} TWILIO_ACCOUNT_SID=AC... TWILIO_AUTH_TOKEN=your_auth_token TWILIO_WHATSAPP_FROM=whatsapp:+14155238886 WHATSAPP_DEFAULT_TO=+1234567890 ``` *** ## Step 2: Verify ```bash theme={null} opensre integrations verify whatsapp ``` This calls the Twilio Account API to verify the credentials. *** ## Step 3: Test with an investigation Trigger a real investigation against a bundled fixture: ```bash theme={null} opensre investigate --input tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` Findings should arrive in WhatsApp as a plain-text message. Long reports are truncated to WhatsApp limits. *** ## Troubleshooting ### `opensre integrations verify whatsapp` errors **`Missing account_sid`** `TWILIO_ACCOUNT_SID` is empty. **`Missing auth_token`** `TWILIO_AUTH_TOKEN` is empty. **`Twilio API check failed: 401`** The SID/token pair is invalid. ### Messages never arrive but verify passes * Ensure `TWILIO_WHATSAPP_FROM` is a valid Twilio WhatsApp sender (sandbox or production). * Ensure recipient uses international format (for example `+1234567890`). * For sandbox usage, join the Twilio sandbox from the destination WhatsApp number. # MongoDB Source: https://opensre.com/docs/mongodb Connect MongoDB so OpenSRE can diagnose database issues during investigations OpenSRE uses MongoDB diagnostics to investigate database-related alerts — checking server health, finding slow queries, monitoring replica sets, and analyzing collection statistics. ## Prerequisites * MongoDB 4.0+ (4.4+ recommended) * Network access from the OpenSRE environment to your MongoDB instance * Valid credentials (if authentication is enabled) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **MongoDB** when prompted and provide your connection string and target database. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} MONGODB_CONNECTION_STRING=mongodb+srv://user:pass@cluster.example.net MONGODB_DATABASE=production MONGODB_AUTH_SOURCE=admin MONGODB_TLS=true ``` | Variable | Default | Description | | --------------------------- | --------- | ------------------------------------------------- | | `MONGODB_CONNECTION_STRING` | — | **Required.** MongoDB connection URI | | `MONGODB_DATABASE` | *(empty)* | Target database for profiler and collection stats | | `MONGODB_AUTH_SOURCE` | `admin` | Authentication database | | `MONGODB_TLS` | `true` | Use TLS for the connection | ### Option 3: Persistent store Integrations are automatically persisted to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "mongodb-prod", "service": "mongodb", "status": "active", "credentials": { "connection_string": "mongodb+srv://user:pass@cluster.example.net", "database": "production", "auth_source": "admin", "tls": true } } ] } ``` ## Connection string formats ```bash theme={null} # Localhost (no auth) mongodb://localhost:27017 # Username/password mongodb://user:password@host:27017/database?authSource=admin # Replica set mongodb://host1:27017,host2:27017,host3:27017/?replicaSet=rs0 # Atlas (SRV) mongodb+srv://user:password@cluster.example.net ``` URL-encode special characters in credentials: `@` → `%40`, `:` → `%3A`, `#` → `%23` ## TLS configuration TLS is enabled by default. For custom certificates: ```bash theme={null} MONGODB_CA_CERT=/path/to/ca.pem # Custom CA certificate MONGODB_TLS_INSECURE=true # Skip validation (dev only) ``` ## Investigation tools When OpenSRE investigates a MongoDB-related alert, five diagnostic tools are available: ### Server status Retrieves version, uptime, connection counts, operation counters, and memory usage. Useful for spotting high connection counts or unusual memory pressure. ### Current operations Lists operations running longer than a configurable threshold (default 1 s). Surfaces long-running queries and lock contention. ### Replica set status Reports member states, health, heartbeat intervals, and optime lag. Identifies unreachable members or sync delays. ### Profiler Reads the `system.profile` collection to surface slow queries, collection scans, and index usage. Requires `MONGODB_DATABASE` to be configured and profiling enabled on the target database. Enable profiling with `db.setProfilingLevel(1)` (slow queries only) or `db.setProfilingLevel(2)` (all queries). ### Collection statistics Returns document count, storage size, index count, and average object size for a given collection. ## Verify ```bash theme={null} opensre integrations verify mongodb ``` Expected output: ``` Service: mongodb Status: passed Detail: Connected to MongoDB 6.0.5; target database: production ``` ## Troubleshooting | Symptom | Fix | | --------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | **Connection refused** | Verify the host/port, check firewalls, and ensure MongoDB is running. For Atlas, whitelist the OpenSRE IP. | | **Authentication failed** | Confirm username/password and `authSource`. Test with `mongosh` first. | | **SSL: CERTIFICATE\_VERIFY\_FAILED** | Provide a CA cert via `MONGODB_CA_CERT` or set `MONGODB_TLS_INSECURE=true` for dev. | | **Profiling is disabled** | Run `db.setProfilingLevel(1)` on the target database. | | **Server is not part of a replica set** | Expected for standalone instances — replica set tools return empty results, other tools still work. | ## Security best practices * Use a **read-only** MongoDB user for monitoring — avoid admin credentials. * Always enable **TLS** in production. * Store connection strings in `.env`, never in code. * Rotate credentials periodically. # MongoDB Atlas Source: https://opensre.com/docs/mongodb-atlas Connect MongoDB Atlas to give OpenSRE visibility into cluster health, performance metrics, deployment status, and operational insights during investigations Databases in the cloud can be harder to troubleshoot. That's where OpenSRE comes in. Connect to your MongoDB Atlas cluster to get instant visibility into cluster health, performance metrics, slow queries, and replica set status when alerts fire. ## What you'll need * MongoDB Atlas API Public Key * MongoDB Atlas API Private Key * Atlas Project ID * Permissions to view cluster metrics and configuration ## Getting connected ### Quick setup: Interactive mode Let's walk through it step by step: ```bash theme={null} opensre integrations setup ``` Choose **MongoDB Atlas** and enter your Atlas API credentials when prompted. ### DIY mode: Environment variables Prefer manual config? Add these to your `.env`: ```bash theme={null} MONGODB_ATLAS_PUBLIC_KEY=your_public_key MONGODB_ATLAS_PRIVATE_KEY=your_private_key MONGODB_ATLAS_PROJECT_ID=your_project_id MONGODB_ATLAS_BASE_URL=https://cloud.mongodb.com/api/atlas/v2 ``` | Variable | Default | Description | | --------------------------- | ---------------------------------------- | -------------------------------------- | | `MONGODB_ATLAS_PUBLIC_KEY` | — | **Required.** Atlas API public key | | `MONGODB_ATLAS_PRIVATE_KEY` | — | **Required.** Atlas API private key | | `MONGODB_ATLAS_PROJECT_ID` | — | **Required.** Atlas project identifier | | `MONGODB_ATLAS_BASE_URL` | `https://cloud.mongodb.com/api/atlas/v2` | Atlas Admin API base URL | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "mongodb-atlas-prod", "service": "mongodb_atlas", "status": "active", "credentials": { "public_key": "your_public_key", "private_key": "your_private_key", "project_id": "your_project_id" } } ] } ``` ## Finding your Atlas API credentials 1. Log in to MongoDB Atlas 2. Navigate to Access Manager → API Keys 3. Create a Project API Key 4. Copy the Public Key 5. Copy the Private Key 6. Locate your Project ID from the Atlas project settings This integration uses the **Atlas Admin API**, not a MongoDB connection string. Create a Project API key with read access to cluster metrics and alerts. ## Investigation tools When OpenSRE investigates a MongoDB Atlas-related alert, these tools query the Atlas Admin API: ### Clusters Lists all clusters in the project — state, MongoDB version, instance size, and replication topology. ### Metrics Retrieves performance metrics for a cluster (CPU, disk I/O, connections, opcounters). ### Alerts Fetches open Atlas alerts including event type, affected cluster, and current metric values. ### Events Returns recent project events such as cluster scaling, backup completions, and maintenance windows. ### Performance advisor Surfaces Atlas Performance Advisor recommendations for slow queries and missing indexes. ## Verify it works Let's make sure everything's connected: ```bash theme={null} opensre integrations verify mongodb_atlas ``` Expected output: ``` Service: mongodb_atlas Status: passed Detail: Connected to MongoDB Atlas project ... ``` ## Troubleshooting | Symptom | Fix | | -------------------------------- | -------------------------------------------------------------------------------------------------- | | **401 Unauthorized** | Regenerate the API key pair. Confirm the public and private keys are copied correctly. | | **403 Forbidden** | Grant the API key **Project Read Only** (or higher) permissions for the target project. | | **Invalid project ID** | Copy the Project ID from Atlas project settings — not the cluster name or connection string. | | **Verify passes but tools fail** | Confirm `MONGODB_ATLAS_PROJECT_ID` matches the project that owns the clusters under investigation. | ## Security best practices * Create a **dedicated Atlas API key** for OpenSRE with the minimum project role required. * Rotate API keys periodically and revoke unused keys. * Never commit API keys to source control — use `.env` or the integration store. # Multi-instance integrations Source: https://opensre.com/docs/multi-instance-integrations Configure multiple accounts, regions, or clusters per provider ## Overview Real deployments have multiple clusters, regions, teams, and accounts for the same provider — a prod and staging Grafana, two AWS accounts, three Kubernetes clusters. OpenSRE's integration model now supports multiple **named instances** per provider with **tags** for filtering, while remaining fully backward-compatible with existing single-instance configurations. ## Configuring multiple instances There are two ways to configure multi-instance integrations. ### 1. Environment variable (JSON array) Set `_INSTANCES` to a JSON array. Each entry can use either a nested `credentials` object or a flat shape. ```bash theme={null} export GRAFANA_INSTANCES='[ {"name":"prod", "tags":{"env":"prod"}, "endpoint":"https://prod.grafana.net", "api_key":"..."}, {"name":"staging", "tags":{"env":"staging"}, "endpoint":"https://staging.grafana.net", "api_key":"..."} ]' ``` Supported env vars: `GRAFANA_INSTANCES`, `DD_INSTANCES`, `HONEYCOMB_INSTANCES`, `CORALOGIX_INSTANCES`, `AWS_INSTANCES`, `ARGOCD_INSTANCES`. When `_INSTANCES` is set, the legacy single-instance vars for that service (e.g. `GRAFANA_INSTANCE_URL`, `GRAFANA_READ_TOKEN`) are ignored. If the JSON is invalid the loader logs a warning and falls back to the legacy vars. ### 2. Store file (`~/.opensre/integrations.json`) The store uses a v2 schema with multiple instances per record: ```json theme={null} { "version": 2, "integrations": [ { "id": "grafana-prod-staging", "service": "grafana", "status": "active", "instances": [ {"name": "prod", "tags": {"env": "prod"}, "credentials": {"endpoint": "...", "api_key": "..."}}, {"name": "staging", "tags": {"env": "staging"}, "credentials": {"endpoint": "...", "api_key": "..."}} ] } ] } ``` v1 stores are migrated automatically on first load — no manual action needed. ## Selecting a specific instance during an investigation ### By alert hint (Grafana, shipping now) Alerts can carry a `grafana_instance` hint — either at the top level of the raw alert payload, or inside `annotations`: ```json theme={null} { "alert_source": "grafana", "grafana_instance": "staging", ... } ``` When set, OpenSRE selects the matching instance. If the hint is absent or unknown, the default (first) instance is used. ### Programmatic selectors ```python theme={null} from integrations.selectors import ( get_default_instance, get_instance_by_name, get_instances_by_tag, select_instance, ) # Flat default (backward-compat shape) default = get_default_instance(resolved_integrations, "grafana") # By name prod = get_instance_by_name(resolved_integrations, "grafana", "prod") # By tag prod_cluster = get_instances_by_tag(resolved_integrations, "grafana", "env", "prod") # Either picked = select_instance(resolved_integrations, "grafana", name="prod") picked = select_instance(resolved_integrations, "grafana", tags={"env": "staging"}) ``` ## Backward compatibility * v1 store files are migrated on load; version bumped from 1 to 2; structural fields (`id`, `service`, `status`) preserved at the top level * Legacy env vars (`GRAFANA_INSTANCE_URL`, `DD_API_KEY`, etc.) continue to work unchanged * `resolved_integrations[]` still returns the flat config dict of the default (first) instance — no existing consumer code changes * A sibling key `_all__instances` is published only when multiple instances exist (or an instance has a non-default name) * Existing single-instance tests continue to pass without modification ## Current end-to-end provider support | Provider | `_INSTANCES` env | Classifier multi-instance | `detect_sources` selection | | --------- | ------------------------- | ------------------------- | ------------------------------- | | Grafana | ✅ | ✅ | ✅ (via `grafana_instance` hint) | | Datadog | ✅ | ✅ | Default instance only | | AWS | ✅ | ✅ | Default instance only | | Honeycomb | ✅ | ✅ | Default instance only | | Coralogix | ✅ | ✅ | Default instance only | | Argo CD | ✅ | ✅ | Default instance only | | Others | — | Default instance only | Default instance only | Providers without end-to-end selection fall back to the default (first) instance — identical behavior to before this feature. ## Known limitations * Only Grafana honors an alert-provided `grafana_instance` hint in this release; extending per-provider selection is a follow-up. * Operators must configure multi-instance via env vars or direct JSON edit; the CLI wizard is not yet instance-aware. * `verify_integrations` currently validates only the default instance of a multi-instance record. * When both the store and env vars configure the same service, the store still wins (existing precedence). To use multi-instance env vars, either remove the store entry for that service or add instances via the store directly. # MySQL Source: https://opensre.com/docs/mysql Connect MySQL so OpenSRE can diagnose database issues during investigations OpenSRE uses MySQL diagnostics to investigate database-related alerts — checking server health, surfacing slow queries, monitoring replication status, and analyzing table statistics. ## Prerequisites * MySQL 5.7+ (8.0+ recommended for full `performance_schema` support) * Network access from the OpenSRE environment to your MySQL instance * A read-only user with access to `information_schema` and `performance_schema` ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **MySQL** when prompted and provide your host, database, and credentials. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} MYSQL_HOST=your-mysql-host MYSQL_PORT=3306 MYSQL_DATABASE=your-database MYSQL_USERNAME=opensre_readonly MYSQL_PASSWORD=your-password MYSQL_SSL_MODE=preferred # preferred, required, or disabled ``` | Variable | Default | Description | | ---------------- | ----------- | ------------------------------------------------ | | `MYSQL_HOST` | — | **Required.** MySQL hostname or IP | | `MYSQL_PORT` | `3306` | MySQL port | | `MYSQL_DATABASE` | — | **Required.** Target database | | `MYSQL_USERNAME` | `root` | Username | | `MYSQL_PASSWORD` | *(empty)* | Password | | `MYSQL_SSL_MODE` | `preferred` | SSL mode: `preferred`, `required`, or `disabled` | ### Option 3: Persistent store Integrations are automatically persisted to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "mysql-prod", "service": "mysql", "status": "active", "credentials": { "host": "prod-primary.mysql.example.com", "port": 3306, "database": "application_db", "username": "opensre_readonly", "password": "your-password", "ssl_mode": "preferred" } } ] } ``` ## Creating a read-only user ```sql theme={null} CREATE USER 'opensre_readonly'@'%' IDENTIFIED BY 'secure-password'; -- Required for server status and process list GRANT PROCESS ON *.* TO 'opensre_readonly'@'%'; -- Required for table statistics GRANT SELECT ON information_schema.* TO 'opensre_readonly'@'%'; -- Required for slow query analysis GRANT SELECT ON performance_schema.* TO 'opensre_readonly'@'%'; FLUSH PRIVILEGES; ``` `performance_schema` is enabled by default in MySQL 5.7+. Slow query data will not be available if it has been explicitly disabled in `my.cnf`. ## Investigation tools When OpenSRE investigates a MySQL-related alert, five diagnostic tools are available: ### Server status Retrieves version, uptime, connection counts (current, running, max used), query rates, and InnoDB buffer pool hit ratio and deadlock counts. Useful for spotting connection saturation, high deadlock rates, or poor buffer pool efficiency. ### Current processes Lists active queries running longer than a configurable threshold (default 1 s), excluding sleeping connections. Surfaces long-running queries and lock contention that may be blocking other operations. ### Replication status Reports replica IO and SQL thread health, seconds behind source, and last replication error. Uses `SHOW REPLICA STATUS` (MySQL 8.0.22+) with automatic fallback to `SHOW SLAVE STATUS` for older versions. Returns a note if the server is not configured as a replica. ### Slow queries Reads `performance_schema.events_statements_summary_by_digest` to surface queries with the highest average execution time. Results include call count, average/total/min/max execution time (in ms), rows examined, rows sent, and full-table scan indicators. Slow query data requires `performance_schema` to be enabled (`performance_schema=ON` in `my.cnf`). This is the default in MySQL 5.7+. ### Table statistics Returns row count estimates, data size, and index size for all base tables in the target database, sourced from `information_schema.TABLES`. Results are ordered by total size descending. ## Verify ```bash theme={null} opensre integrations verify mysql ``` Expected output: ``` Service: mysql Status: passed Detail: Connected to MySQL 8.0.32; target database: application_db ``` ## Troubleshooting | Symptom | Fix | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | **Connection refused** | Verify host, port, and firewall rules. Confirm MySQL is running and accepting remote connections (`bind-address` in `my.cnf`). | | **Authentication failed** | Check username and password. Ensure the user exists for the connecting host (`'user'@'%'` or specific IP). | | **SSL error** | Set `MYSQL_SSL_MODE=disabled` to test without SSL, or `required` to enforce it. | | **Access denied on performance\_schema** | Grant `SELECT ON performance_schema.*` to the OpenSRE user. | | **Slow query data empty** | Confirm `performance_schema=ON` in `my.cnf` and restart MySQL. | | **Replication shows empty** | Expected for primary servers — replication tools return a note, other tools still work. | ## Security best practices * Use a **dedicated read-only user** — avoid root credentials for monitoring. * Enable **SSL** (`MYSQL_SSL_MODE=required`) in production environments. * Restrict the user to specific hosts rather than `'%'` where possible. * Store credentials in `.env`, never in source code. * Rotate credentials periodically. # OpenClaw Source: https://opensre.com/docs/openclaw Connect OpenSRE to OpenClaw for native context lookup and RCA write-back OpenSRE integrates with OpenClaw in a native, one-way support loop: * OpenSRE can search recent OpenClaw conversations during an investigation. * OpenSRE can read full OpenClaw threads for extra engineering context. * OpenSRE writes completed RCA findings back into OpenClaw through the OpenClaw bridge. OpenSRE no longer ships a separate `opensre-mcp` command. You do not need to register OpenSRE itself as an MCP server inside OpenClaw. ## Setup ### 0. Preflight the local OpenClaw CLI Before you onboard OpenClaw in OpenSRE, make sure the OpenClaw CLI itself is healthy in the same shell: ```bash theme={null} node -v openclaw --help openclaw gateway status ``` Expected result: * `node -v` shows Node `v22.12+` or newer * `openclaw --help` exits cleanly * `openclaw gateway status` does not fail immediately If you use `nvm`, the safe sequence is: ```bash theme={null} nvm install 22 nvm use 22 node -v openclaw --help ``` `nvm alias default 22` only affects future shells. It does not switch the current shell that launches `uv run opensre`. ### 1. Install and configure OpenSRE locally ```bash theme={null} make install uv run opensre onboard ``` ### 2. Configure the OpenSRE → OpenClaw bridge Use the guided wizard: ```bash theme={null} uv run opensre onboard ``` Or use the direct integration setup command: ```bash theme={null} uv run opensre integrations setup openclaw uv run opensre integrations verify openclaw ``` Recommended bridge settings: ```bash theme={null} OPENCLAW_MCP_MODE=stdio OPENCLAW_MCP_COMMAND=openclaw OPENCLAW_MCP_ARGS="mcp serve" ``` ### 3. Verify the bridge From the repo root: ```bash theme={null} uv run opensre integrations verify openclaw uv run opensre health ``` ### 4. Run a smoke test Use the bundled OpenClaw fixture directly: ```bash theme={null} uv run opensre investigate -i tests/fixtures/openclaw_test_alert.json ``` If verification still fails, the next manual checks are: ```bash theme={null} openclaw gateway status openclaw gateway run uv run opensre integrations verify openclaw ``` ## Write-back behavior After every completed investigation, OpenSRE attempts to write the full RCA report back into OpenClaw. * If the investigation state contains an `openclaw_context.conversation_id`, OpenSRE first tries to append the result to that conversation. * Otherwise, OpenSRE creates a new OpenClaw conversation containing the RCA report, root cause, remediation steps, and confidence score. ## Investigation behavior When OpenClaw is configured, OpenSRE now prefers the high-signal conversation actions first: * `search_openclaw_conversations` to find recent related context * `get_openclaw_conversation` to read the full thread * `send_openclaw_message` to append a concrete follow-up The lower-level raw bridge tools remain available, but they are treated as fallback paths instead of the default investigation path. ## Accurate RCA OpenClaw improves RCA quality by supplying engineering context, but OpenClaw alone is not enough for a strong investigation. For accurate RCA, also configure at least one of these: * `Grafana` or `Datadog` for metrics/logs/traces * `AWS` for infra topology and runtime state * `GitHub` for deploy/code-change context A good local validation sequence is: ```bash theme={null} uv run opensre integrations verify openclaw uv run opensre integrations verify grafana uv run opensre integrations verify github uv run opensre investigate -i tests/fixtures/openclaw_test_alert.json ``` Contributor-facing regression coverage for this path lives in: * `tests/integrations/openclaw/test_integration.py` * `tests/tools/test_openclaw_mcp_tool.py` * `tests/utils/test_openclaw_delivery.py` * `tests/e2e/openclaw/` (e2e suite — see the "End-to-end tests" section below) ## Environment variables | Variable | Meaning | Default | | ------------------------- | --------------------------------------------------------------- | ----------------- | | `OPENCLAW_MCP_MODE` | OpenClaw bridge transport: `stdio`, `streamable-http`, or `sse` | `streamable-http` | | `OPENCLAW_MCP_COMMAND` | Executable used for `stdio` mode | `openclaw` | | `OPENCLAW_MCP_ARGS` | Space-separated arguments for `stdio` mode | `mcp serve` | | `OPENCLAW_MCP_URL` | Remote MCP endpoint for `streamable-http` or `sse` | *none* | | `OPENCLAW_MCP_AUTH_TOKEN` | Optional bearer token for remote MCP transports | *empty* | ## Troubleshooting * `OpenClaw bridge validation failed: Node.js ... is required`: switch the current shell to Node `22.12+` or newer. With `nvm`: `nvm install 22 && nvm use 22 && nvm alias default 22`. With Homebrew: `brew install node@22 && brew link --overwrite node@22`. Then re-run `openclaw --help` before retrying onboarding. * `uv run opensre integrations verify openclaw` fails against `http://127.0.0.1:18789/`: that URL is the OpenClaw Control UI/Gateway, not the MCP bridge. Use `stdio` mode with `openclaw mcp serve`. * `search_openclaw_conversations` or write-back fails with `Connection closed` or `ECONNREFUSED`: the OpenClaw Gateway is not running. Check `openclaw gateway status`, then run `openclaw gateway run`. * `Command not found: openclaw`: install the OpenClaw CLI or point `OPENCLAW_MCP_COMMAND` at the full executable path. ## End-to-end tests The OpenClaw e2e suite lives at [`tests/e2e/openclaw/`](https://github.com/Tracer-Cloud/opensre/tree/main/tests/e2e/openclaw). See its [README](https://github.com/Tracer-Cloud/opensre/tree/main/tests/e2e/openclaw/README.md) for prerequisites, how to run the suite, scenario layout, and CI behavior. # OpenObserve Source: https://opensre.com/docs/openobserve Connect OpenObserve so OpenSRE can pull structured log and trace evidence during investigations When something breaks, you want to know *why* — and logs tell that story. OpenSRE connects to OpenObserve to retrieve log and trace data that helps explain what was happening when an alert fired, making it easier to correlate errors, anomalies, and service interactions during investigations. ## What you need * An OpenObserve instance (self-hosted or cloud-hosted) * An OpenObserve access token * The URL of your OpenObserve deployment * Access to the organization you want OpenSRE to query ## Getting set up ### Guided setup Start here if you want step-by-step guidance: ```bash theme={null} opensre integrations setup ``` Select **OpenObserve** and enter your OpenObserve credentials when prompted. ### Manual setup with environment variables Or add these to your `.env` file: ```bash theme={null} OPENOBSERVE_URL=https://openobserve.example.com OPENOBSERVE_TOKEN=your_access_token OPENOBSERVE_ORG=default OPENOBSERVE_STREAM=logs OPENOBSERVE_MAX_RESULTS=100 ``` | Variable | Default | Description | | ------------------------- | --------- | ---------------------------------------------- | | `OPENOBSERVE_URL` | — | **Required.** URL of your OpenObserve instance | | `OPENOBSERVE_TOKEN` | — | **Required.** Authentication token | | `OPENOBSERVE_ORG` | `default` | Organization name | | `OPENOBSERVE_STREAM` | *(empty)* | Optional stream to query | | `OPENOBSERVE_MAX_RESULTS` | `100` | Maximum number of results returned | ### Alternative authentication If your OpenObserve deployment uses username/password authentication instead of tokens, you can configure: ```bash theme={null} OPENOBSERVE_USERNAME=your_username OPENOBSERVE_PASSWORD=your_password OPENOBSERVE_ORG=default ``` Use either token-based authentication or username/password authentication depending on your OpenObserve deployment. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "openobserve-prod", "service": "openobserve", "status": "active", "credentials": { "url": "https://openobserve.example.com", "token": "your_access_token", "org": "default" } } ] } ``` ## Finding your OpenObserve credentials 1. Log in to your OpenObserve instance 2. Navigate to your user or organization settings 3. Create or retrieve an access token 4. Copy your OpenObserve URL 5. Note the organization name you want OpenSRE to query 6. Add these values to your OpenSRE configuration ## What OpenSRE can query Once connected, OpenSRE can search across: * **Logs** — Search by timestamp, service name, log level, and message content * **Traces** — Investigate spans, follow distributed traces, and analyze latency * **Metrics** — Query observability metrics stored in OpenObserve Use a token with the minimum permissions required for investigation workflows whenever possible. ## Test your connection Make sure everything is configured correctly: ```bash theme={null} opensre integrations verify openobserve ``` Expected output: ``` Service: openobserve Status: passed Detail: Configured for OpenObserve at https://openobserve.example.com ``` ## Troubleshooting | Symptom | Fix | | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | | **401 Unauthorized** | Regenerate the access token or confirm username/password credentials. | | **404 on query** | Check `OPENOBSERVE_ORG` matches your organization slug. Verify the stream name if `OPENOBSERVE_STREAM` is set. | | **Connection refused** | Confirm `OPENOBSERVE_URL` includes the correct protocol and port. Ensure network access from OpenSRE to the instance. | | **Empty log results** | Widen the time range in the investigation query. Confirm logs are ingested into the target stream. | # OpenSearch / Elasticsearch Source: https://opensre.com/docs/opensearch Connect OpenSearch or Elasticsearch so OpenSRE can search application logs and analytics indices during investigations OpenSRE queries OpenSearch (or Elasticsearch) to retrieve application logs, error events, and analytics records — pulling concrete log lines into the investigation alongside metrics and traces from your other observability tools. ## Prerequisites * OpenSearch 1.x/2.x or Elasticsearch 7.x/8.x reachable from the machine running OpenSRE * The cluster URL (e.g. `https://my-cluster.us-east-1.es.amazonaws.com`) * Credentials for one of: * HTTP Basic Auth (username and password) — typical for self-hosted OpenSearch * API key — typical for Elastic Cloud * No auth — only when the cluster has the security plugin disabled OpenSearch authenticates clients via Basic Auth by default; the security plugin does not natively issue API keys ([opensearch-project/security#4009](https://github.com/opensearch-project/security/issues/4009)). Most self-hosted clusters use Basic Auth. ## Setup ### Option 1: Interactive onboarding wizard ```bash theme={null} opensre onboard ``` Pick **OpenSearch / Elasticsearch** from the integration menu. The wizard asks for: 1. **OpenSearch URL** — the base URL of your cluster 2. **Authentication method** — choose one of: * **Username + Password** (default) — for self-hosted OpenSearch with the security plugin enabled * **API key** — for Elastic Cloud or clusters with API-key auth configured * **None (security disabled)** — for clusters running with the security plugin disabled The wizard validates the configuration by calling `GET /_cluster/health` before saving, so you'll see immediate feedback if the URL or credentials are wrong. ### Option 2: Legacy CLI ```bash theme={null} opensre integrations setup opensearch ``` Same prompts as the wizard, in a smaller standalone form. ### Option 3: Environment variables Add to your `.env`: ```bash theme={null} OPENSEARCH_URL=https://my-cluster.example.com # Basic Auth (typical for self-hosted OpenSearch) OPENSEARCH_USERNAME=admin OPENSEARCH_PASSWORD=secret # API key (typical for Elastic Cloud — use instead of username/password) OPENSEARCH_API_KEY=your-api-key ``` | Variable | Default | Description | | --------------------- | ------- | ----------------------------------------------------------------- | | `OPENSEARCH_URL` | — | **Required.** Base URL of your OpenSearch / Elasticsearch cluster | | `OPENSEARCH_USERNAME` | — | Basic auth username | | `OPENSEARCH_PASSWORD` | — | Basic auth password | | `OPENSEARCH_API_KEY` | — | API key (used in `Authorization: ApiKey ...` header) | When both `OPENSEARCH_API_KEY` and Basic Auth credentials are set, the API key takes precedence. If only one of `OPENSEARCH_USERNAME` / `OPENSEARCH_PASSWORD` is set, no `Authorization` header is emitted (the cluster will reject the request, surfacing the misconfiguration). ### Option 4: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "opensearch-prod", "service": "opensearch", "status": "active", "credentials": { "url": "https://my-cluster.example.com", "username": "admin", "password": "secret", "api_key": "", "index_pattern": "logs-*" } } ] } ``` ## Verify ```bash theme={null} opensre integrations verify opensearch ``` Expected output on success: ``` Service: opensearch Status: passed Detail: Connected to OpenSearch cluster 'my-cluster' (green, 3 node(s)). ``` ## How it works in investigations When an OpenSearch integration is configured, OpenSRE automatically includes it as an evidence source during every investigation. Two tools become available to the investigation agent: ### `query_elasticsearch_logs` Searches log indices for messages matching a Lucene/KQL query within a bounded time window. The agent uses this to: * Pull error and exception messages around the alert timestamp * Filter logs by service, container, or correlation ID extracted from alert annotations * Surface stack traces and panic messages that explain a metric anomaly * Cross-reference logs against alerts firing in Datadog, Grafana, or Alertmanager The tool returns both the raw matching logs and a separate `error_logs` slice pre-filtered for keywords like `error`, `exception`, `traceback`, and `panic`, so the agent can prioritize signal over noise. ### `query_opensearch_analytics` Runs bounded analytics queries against any index pattern (default `*`). The agent uses this for non-log data stored in OpenSearch — APM events, audit trails, business metrics, or any custom analytics index your team maintains. Both tools share the same configured credentials, so configuring the integration once enables both query paths. ## Troubleshooting | Symptom | Fix | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Status: missing** | Set `OPENSEARCH_URL` or run `opensre onboard` and pick OpenSearch | | **Connection refused** | Verify the URL is reachable from this host; check firewall and VPC rules | | **401 Unauthorized** | Verify credentials are correct. For self-hosted OpenSearch, use Basic Auth (`OPENSEARCH_USERNAME` + `OPENSEARCH_PASSWORD`), not API key — most installs do not have API keys enabled | | **SSL certificate errors** | For self-signed certificates, ensure the CA cert is in the system trust store | | **Empty results from a known-good query** | Confirm the index pattern matches your data; the default `*` matches every index but specific patterns like `logs-*` are faster and more reliable | | **`security_exception: no permissions`** | The user needs at least `read` and `view_index_metadata` permissions on the queried indices | ## Security best practices * Create a **read-only** OpenSearch user for OpenSRE — the agent only reads logs and analytics indices and never writes to the cluster during investigations. * Limit the role to the indices OpenSRE needs (typically logs and APM indices), not cluster-wide. * Store credentials in `.env` or the integration store, not in source code. * For Elastic Cloud, generate a scoped API key and rotate it on a schedule rather than using the master deployment credentials. * For self-hosted clusters behind a reverse proxy, prefer Basic Auth over disabling security entirely — even on internal networks. # OpsGenie Source: https://opensre.com/docs/opsgenie Connect OpsGenie so OpenSRE can read active alerts and incident context during investigations OpenSRE queries OpsGenie to retrieve active alerts and their details — correlating on-call incidents with infrastructure events and investigation findings. ## Prerequisites * OpsGenie account (Atlassian or standalone) * API key with read access ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **OpsGenie** when prompted and provide your API key. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} OPSGENIE_API_KEY=your-api-key OPSGENIE_REGION=us # or "eu" for EU accounts ``` | Variable | Default | Description | | ------------------ | ------- | ------------------------------ | | `OPSGENIE_API_KEY` | — | **Required.** OpsGenie API key | | `OPSGENIE_REGION` | `us` | Region: `us` or `eu` | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "opsgenie-prod", "service": "opsgenie", "status": "active", "credentials": { "api_key": "your-api-key", "region": "us" } } ] } ``` ## Creating an API key 1. In OpsGenie, go to **Settings** → **API key management** 2. Click **Add new API key** 3. Name it `opensre` and enable **Read** access 4. Copy the key EU accounts use a different endpoint. Set `OPSGENIE_REGION=eu` if your OpsGenie URL is `app.eu.opsgenie.com`. ## Verify ```bash theme={null} opensre integrations verify opsgenie ``` Expected output: ``` Service: opsgenie Status: passed Detail: Connected to OpsGenie (US region); API key accepted ``` ## Troubleshooting | Symptom | Fix | | -------------------- | ---------------------------------------------------------------- | | **401 Unauthorized** | Check the API key — ensure it has Read access | | **404 Not Found** | Try setting `OPSGENIE_REGION=eu` if you're on the EU instance | | **Rate limited** | OpsGenie enforces per-minute rate limits; reduce query frequency | ## Security best practices * Use a **read-only API key** — OpsGenie supports granular permission scopes. * Store the API key in `.env`, not in source code. # PagerDuty Source: https://opensre.com/docs/pagerduty Connect PagerDuty so OpenSRE can read incidents, on-call schedules, and service topology during investigations OpenSRE queries PagerDuty to retrieve active incidents, their timelines, on-call responders, and service/escalation-policy configuration — correlating incident management data with infrastructure events during root cause analysis. ## Prerequisites * PagerDuty account * REST API key with read access ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **PagerDuty** when prompted and provide your API key. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} PAGERDUTY_API_KEY=your-api-key ``` | Variable | Default | Description | | -------------------- | --------------------------- | ------------------------------------------------------- | | `PAGERDUTY_API_KEY` | — | **Required.** PagerDuty REST API v2 key | | `PAGERDUTY_BASE_URL` | `https://api.pagerduty.com` | API base URL (override only for EU or custom instances) | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "pagerduty-prod", "service": "pagerduty", "status": "active", "credentials": { "api_key": "your-api-key" } } ] } ``` ## Creating an API key 1. In PagerDuty, go to **Integrations** → **Developer Tools** → **API Access Keys** 2. Click **Create New API Key** 3. Add a description (e.g. `opensre read-only`) 4. Copy the key immediately — it won't be shown again PagerDuty API keys are account-level. Use a read-only key to follow the principle of least privilege. ## Verify ```bash theme={null} opensre integrations verify pagerduty ``` Expected output: ``` Service: pagerduty Status: passed Detail: Connected to PagerDuty; API key accepted ``` ## Available tools | Tool | Description | | --------------------------- | ----------------------------------------------------------------------------- | | `pagerduty_incidents` | List and search incidents (filter by status, urgency, service, time range) | | `pagerduty_incident_detail` | Fetch full incident details and activity timeline (log entries) | | `pagerduty_oncall` | Fetch current on-call responders by escalation policy | | `pagerduty_services` | List services with escalation policies, integrations, and alert routing rules | ## Example investigation context During an investigation, OpenSRE may use PagerDuty tools to: * Find which incidents were triggered around the alert time window * Identify who was on-call and how quickly incidents were acknowledged * Map which services and escalation policies were involved * Correlate PagerDuty incident timelines with Grafana/Datadog metrics ## Troubleshooting | Symptom | Fix | | ------------------------- | ------------------------------------------------------------------------- | | **401 Unauthorized** | Check the API key — ensure it's a valid REST API v2 key | | **403 Forbidden** | The key may lack required permissions; recreate with full read access | | **429 Rate Limited** | PagerDuty enforces rate limits (960 requests/min); reduce query frequency | | **No incidents returned** | Check time range filters — default returns recent incidents only | ## Security best practices * Use a **read-only API key** — OpenSRE never needs to modify PagerDuty resources. * Store the API key in `.env`, not in source code. * Rotate API keys periodically via the PagerDuty Developer Tools page. # Pi coding tasks Source: https://opensre.com/docs/pi-coding Let OpenSRE hand a coding task to the Pi agent, which edits the working tree and returns a diff. The **Pi coding tool** lets OpenSRE submit a coding task to the [Pi](https://pi.dev) agent. Pi edits files in a workspace to implement the change and returns a summary plus the git diff. It **does not commit, push, or open a pull request** — it only edits the working tree so you can review the diff. This is different from using Pi as your LLM provider (`LLM_PROVIDER=pi`, see [LLM providers](/docs/llm-providers)). There Pi is the reasoning engine; here Pi is the thing that actually changes code. This is a **mutating** tool — it changes files on disk. It is **disabled by default**: it only becomes available to the agent when `PI_CODING_ENABLED=1`. Enable it only when you want OpenSRE to make code changes. It never commits, pushes, or opens a PR, so you always review the diff. ## Quick reference | Env var | What it does | | --------------------------- | ------------------------------------------------------------------------------- | | `PI_CODING_ENABLED` | Opt-in switch. Set to `1` to make the tool available. Off by default. | | `PI_CODING_MODEL` | Optional Pi model in `provider/model` form (e.g. `anthropic/claude-haiku-4-5`). | | `PI_CODING_WORKSPACE` | Default repository path Pi edits. Defaults to the current directory. | | `PI_CODING_TIMEOUT_SECONDS` | Per-task timeout (default 600, clamped 60–1800). | | `PI_BIN` | Optional explicit path to the `pi` binary. | ## Enable it 1. Install and authenticate the Pi CLI (same binary/credentials as the Pi provider): ```bash theme={null} npm i -g @earendil-works/pi-coding-agent # then authenticate: run `pi` and use /login, or export a provider key such as GEMINI_API_KEY ``` 2. Turn the tool on: ```bash theme={null} export PI_CODING_ENABLED=1 export PI_CODING_MODEL=anthropic/claude-haiku-4-5 # optional ``` 3. Confirm Pi is ready: ```bash theme={null} uv run opensre doctor ``` ## How it works When the tool runs, it: 1. runs `pi` in headless mode inside the workspace with your task plus a fixed set of rules (follow `AGENTS.md`, do not commit or push, do not run destructive git commands, preserve unrelated changes, summarize what changed), 2. lets Pi edit the files, 3. returns `success`, a `summary`, the list of `changed_files`, and the `diff` (truncated if large). You review the diff and decide whether to keep, commit, or revert the changes. ## Example Once enabled, the tool is available to the agent (investigation surface). Describe a scoped coding task, for example: ``` add input validation to parse_config() in config/loader.py ``` If the agent calls the tool, Pi edits `PI_CODING_WORKSPACE`, and you get back a summary and the diff to review. Nothing is committed. (The agent decides whether to call the tool; for a deterministic run, invoke `pi_coding_task` directly — see the tests.) ## Notes * Nothing is committed or pushed. Opening a PR is out of scope for now. * It is **disabled by default**. Only when `PI_CODING_ENABLED=1` does the agent see it and get to choose to call it — enable it deliberately, since it edits files. * If `PI_CODING_ENABLED` is unset, or the Pi CLI is missing or unauthenticated, the tool stays unavailable and returns a clear message instead of running. # PostgreSQL Source: https://opensre.com/docs/postgresql Connect PostgreSQL so OpenSRE can diagnose database issues during investigations OpenSRE uses PostgreSQL diagnostics to investigate database-related alerts — checking server health, surfacing slow queries, monitoring replication status, and analyzing table statistics. ## Prerequisites * PostgreSQL 10+ (12+ recommended for full `pg_stat_statements` support) * Network access from the OpenSRE environment to your PostgreSQL instance * A read-only user with access to system views ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **PostgreSQL** when prompted and provide your host, database, and credentials. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} POSTGRESQL_HOST=your-postgresql-host POSTGRESQL_PORT=5432 POSTGRESQL_DATABASE=your-database POSTGRESQL_USERNAME=opensre_readonly POSTGRESQL_PASSWORD=your-password POSTGRESQL_SSL_MODE=prefer # prefer, require, or disable ``` | Variable | Default | Description | | --------------------- | ---------- | ------------------------------------------- | | `POSTGRESQL_HOST` | — | **Required.** PostgreSQL hostname or IP | | `POSTGRESQL_PORT` | `5432` | PostgreSQL port | | `POSTGRESQL_DATABASE` | — | **Required.** Target database | | `POSTGRESQL_USERNAME` | `postgres` | Username | | `POSTGRESQL_PASSWORD` | *(empty)* | Password | | `POSTGRESQL_SSL_MODE` | `prefer` | SSL mode: `prefer`, `require`, or `disable` | ### Option 3: Persistent store Integrations are automatically persisted to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "postgresql-prod", "service": "postgresql", "status": "active", "credentials": { "host": "prod-primary.postgres.example.com", "port": 5432, "database": "application_db", "username": "opensre_readonly", "password": "your-password", "ssl_mode": "prefer" } } ] } ``` ## Creating a read-only user ```sql theme={null} -- Create the user CREATE USER opensre_readonly WITH PASSWORD 'secure-password'; -- Grant access to system views GRANT pg_monitor TO opensre_readonly; -- Grant access to the target database GRANT CONNECT ON DATABASE application_db TO opensre_readonly; \c application_db GRANT USAGE ON SCHEMA public TO opensre_readonly; GRANT SELECT ON ALL TABLES IN SCHEMA public TO opensre_readonly; ``` `pg_monitor` (available in PostgreSQL 10+) grants read access to all monitoring views including `pg_stat_activity`, `pg_stat_replication`, and `pg_stat_statements` without superuser privileges. ## Enabling slow query tracking Slow query analysis requires the `pg_stat_statements` extension. Add to `postgresql.conf`: ```ini theme={null} shared_preload_libraries = 'pg_stat_statements' pg_stat_statements.track = all ``` Then restart PostgreSQL and run: ```sql theme={null} CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` ## Investigation tools When OpenSRE investigates a PostgreSQL-related alert, five diagnostic tools are available: ### Server status Retrieves version, uptime, connection counts (total, active, idle, max), transaction commit/rollback rates, and buffer cache hit ratio per database. Useful for spotting connection saturation or cache efficiency drops. ### Current queries Lists active queries running longer than a configurable threshold (default 1 s), excluding the monitoring connection itself. Includes PID, user, client address, duration, wait event, and a truncated query string. ### Replication status Uses `pg_is_in_recovery()` to reliably detect replica vs primary. On a primary, reports WAL position and per-replica lag (write, flush, replay). Returns a note if the server is a replica or if no replicas are connected. ### Slow queries Reads `pg_stat_statements` to surface queries with the highest mean execution time. Results include call count, total/mean/min/max execution time (in ms with sub-millisecond precision), rows returned, and buffer cache hit percentage. Slow query data requires the `pg_stat_statements` extension to be installed and loaded via `shared_preload_libraries`. OpenSRE returns an informative message if the extension is not available. ### Table statistics Reads `pg_stat_user_tables` and `pg_class` for a given schema (default `public`). Returns insert/update/delete/live/dead tuple counts, sequential vs index scan ratios, last vacuum/analyze timestamps, and table sizes in bytes and MB. ## Verify ```bash theme={null} opensre integrations verify postgresql ``` Expected output: ``` Service: postgresql Status: passed Detail: Connected to PostgreSQL 16.1; target database: application_db ``` ## Troubleshooting | Symptom | Fix | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | **Connection refused** | Verify host, port, and firewall rules. Check `listen_addresses` in `postgresql.conf` and `pg_hba.conf` for the connecting host. | | **Authentication failed** | Confirm username and password. Check `pg_hba.conf` for the correct auth method (`md5`, `scram-sha-256`). | | **SSL error** | Set `POSTGRESQL_SSL_MODE=disable` to test without SSL, or `require` to enforce it. | | **Permission denied on pg\_stat\_activity** | Grant `pg_monitor` role to the OpenSRE user. | | **pg\_stat\_statements not found** | Add `pg_stat_statements` to `shared_preload_libraries`, restart PostgreSQL, then run `CREATE EXTENSION pg_stat_statements;`. | | **Replication shows replica, not primary** | Expected — the tool correctly identifies the server as a replica and returns a note. Connect to the primary for replication lag details. | ## Security best practices * Use a **dedicated read-only user** with `pg_monitor` — avoid superuser credentials for monitoring. * Enable **SSL** (`POSTGRESQL_SSL_MODE=require`) in production environments. * Use `scram-sha-256` authentication in `pg_hba.conf` rather than `md5`. * Store credentials in `.env`, never in source code. * Rotate credentials periodically. # PostHog Source: https://opensre.com/docs/posthog Connect PostHog REST credentials so OpenSRE can verify access to your project OpenSRE stores **your** PostHog project credentials for REST API access (project metadata and correlation during investigations). Configure with `opensre integrations setup posthog` or `POSTHOG_*` environment variables. Looking for agent tools during investigations (analytics, feature flags, HogQL)? Use [PostHog (MCP)](/docs/posthog-mcp) with `opensre integrations setup posthog_mcp`. ## PostHog in OpenSRE — quick guide | What you want | Use this | How to set it up | | ----------------------------------------------- | ----------------------------- | --------------------------------------------------------------- | | Store PostHog REST credentials for your project | **PostHog** — this page | `opensre integrations setup posthog` or env vars (below) | | Let the agent explore PostHog during incidents | [PostHog (MCP)](/docs/posthog-mcp) | `opensre integrations setup posthog_mcp` or onboarding wizard | | OpenSRE's own usage telemetry | Built-in product analytics | Controlled by `OPENSRE_NO_TELEMETRY` — not your PostHog project | ## Prerequisites * A PostHog account (US, EU, or self-hosted) * Your PostHog **project ID** * A **personal API key** with read access to that project — see [PostHog's personal API keys docs](https://posthog.com/docs/api/personal-api-keys) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup posthog ``` Or run the onboarding wizard and select **PostHog** under Observability. You will be prompted for: * **Project ID** — from PostHog project settings * **Personal API key** (`phx_...`) — [Personal API keys](https://posthog.com/docs/api/personal-api-keys) with read access * **API base URL** — defaults to US SaaS (`https://us.i.posthog.com`); use `https://eu.i.posthog.com` for EU On success, credentials are saved to your integration store and synced to `.env`. ### Option 2: Environment variables Add credentials directly to your `.env` file: ```bash theme={null} POSTHOG_PROJECT_ID=your_project_id POSTHOG_PERSONAL_API_KEY=phx_your_personal_api_key POSTHOG_BASE_URL=https://us.i.posthog.com # optional — US SaaS default POSTHOG_TIMEOUT_SECONDS=15.0 # optional ``` | Variable | Default | Description | | -------------------------- | -------------------------- | ---------------------------------------------------------------------------- | | `POSTHOG_PROJECT_ID` | — | **Required.** Your PostHog project ID | | `POSTHOG_PERSONAL_API_KEY` | — | **Required.** Personal API key for the project | | `POSTHOG_BASE_URL` | `https://us.i.posthog.com` | API base URL (`https://eu.i.posthog.com` for EU, or your self-hosted origin) | | `POSTHOG_TIMEOUT_SECONDS` | `15.0` | HTTP timeout in seconds | ## Verify After saving your `.env`, confirm OpenSRE can reach your PostHog project: ```bash theme={null} opensre integrations verify posthog ``` A successful check prints `PostHog validated.` For the MCP integration, use `opensre integrations verify posthog_mcp` instead. ## Troubleshooting | Symptom | Fix | | ------------------------------- | ----------------------------------------------------------------------- | | **Authentication failed (401)** | Check `POSTHOG_PERSONAL_API_KEY` is valid and has read access | | **Project not found (404)** | Confirm `POSTHOG_PROJECT_ID` matches the instance in `POSTHOG_BASE_URL` | | **Verify says not configured** | Both `POSTHOG_PROJECT_ID` and `POSTHOG_PERSONAL_API_KEY` must be set | ## Security * Use a dedicated personal API key with the minimum read permissions needed * Store credentials in environment variables or a secret manager — not in source code * Rotate keys periodically # PostHog (MCP) Source: https://opensre.com/docs/posthog-mcp Connect PostHog's hosted MCP server so OpenSRE can query analytics, feature flags, error tracking, and HogQL during investigations OpenSRE connects to PostHog's hosted [Model Context Protocol (MCP)](https://posthog.com/docs/model-context-protocol) server, exposing PostHog's products — product analytics, feature flags, error tracking, experiments, surveys, and HogQL queries — as tools the agent can call while investigating an incident. This is distinct from the [PostHog REST integration](/docs/posthog), which only stores project credentials for the REST API. Use MCP when you want the agent to explore PostHog data directly during investigations. The onboarding wizard lists **PostHog** (REST) and **PostHog (MCP)** as separate choices — configure REST credentials with `opensre integrations setup posthog`. ## Tools | Tool | What it does | | -------------------- | -------------------------------------------------------------------------------------------- | | `list_posthog_tools` | List the tools the connected PostHog MCP server exposes (compact, filterable) | | `call_posthog_tool` | Call a named PostHog MCP tool (e.g. run a HogQL query, list feature flags, inspect an error) | The agent typically calls `list_posthog_tools` first to discover what is available for your project, then `call_posthog_tool` with the chosen tool name and arguments. The hosted PostHog MCP server exposes 240+ tools, each with a full input schema. Returning all of them at once is far larger than any model's context window, so `list_posthog_tools` returns a **compact, bounded listing** — tool names plus short descriptions, without schemas. To work with it efficiently: * Pass `name_filter` (space- or comma-separated terms, e.g. `"events query sql"`) to narrow the list to relevant tools. * Pass `include_schema=true` on a narrowed list to fetch the full input schema for the specific tool you intend to call. To query events for a person or across a project, call `call_posthog_tool` with `tool_name="execute-sql"` and a HogQL query (e.g. `SELECT event, count() FROM events WHERE ... GROUP BY event`). There is no `search_events` tool. ## Prerequisites * A PostHog account (US or EU — the hosted server routes you automatically) * A PostHog **personal API key** created with the **MCP Server** preset. See [PostHog's personal API keys docs](https://posthog.com/docs/api/personal-api-keys). OpenSRE defaults to **read-only** access (`x-posthog-read-only: true`) so investigations cannot mutate your PostHog project. Set `POSTHOG_MCP_READ_ONLY=false` only if you explicitly want the agent to perform writes. ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **PostHog (MCP)** when prompted, then paste your personal API key. The setup uses the hosted Streamable HTTP transport; keep the default URL unless you have a reason to change it. To run a local server instead, set `POSTHOG_MCP_MODE=stdio` via environment variables (see below). To skip the menu, name the service directly: ```bash theme={null} opensre integrations setup posthog_mcp opensre integrations verify posthog_mcp ``` ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} POSTHOG_MCP_MODE=streamable-http POSTHOG_MCP_URL=https://mcp.posthog.com/mcp POSTHOG_MCP_AUTH_TOKEN=phx_your_personal_api_key POSTHOG_MCP_PROJECT_ID=12345 # optional, scope to one project POSTHOG_MCP_ORGANIZATION_ID= # optional, scope to one organization POSTHOG_MCP_FEATURES= # optional, comma-separated feature filter POSTHOG_MCP_READ_ONLY=true # optional, default true ``` | Variable | Default | Description | | ----------------------------- | ----------------------------- | -------------------------------------------------------------------- | | `POSTHOG_MCP_AUTH_TOKEN` | — | **Required** (hosted). Personal API key with the `MCP Server` preset | | `POSTHOG_MCP_URL` | `https://mcp.posthog.com/mcp` | MCP server URL (use `https://mcp-eu.posthog.com/mcp` to pin EU) | | `POSTHOG_MCP_MODE` | `streamable-http` | Transport: `streamable-http`, `sse`, or `stdio` | | `POSTHOG_MCP_PROJECT_ID` | — | Scope tools to a specific PostHog project | | `POSTHOG_MCP_ORGANIZATION_ID` | — | Scope tools to a specific organization | | `POSTHOG_MCP_FEATURES` | — | Comma-separated feature filter (e.g. `flags,error-tracking`) | | `POSTHOG_MCP_READ_ONLY` | `true` | Send the read-only header so the agent cannot mutate PostHog | | `POSTHOG_MCP_COMMAND` | — | Command to launch a local MCP server (`stdio` mode only) | | `POSTHOG_MCP_ARGS` | — | Arguments for the local MCP command (`stdio` mode only) | To run a local PostHog MCP server instead of the hosted endpoint, use `stdio` mode: ```bash theme={null} POSTHOG_MCP_MODE=stdio POSTHOG_MCP_COMMAND=npx POSTHOG_MCP_ARGS=-y @posthog/mcp-server@latest POSTHOG_MCP_AUTH_TOKEN=phx_your_personal_api_key ``` ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "posthog-mcp-prod", "service": "posthog_mcp", "status": "active", "credentials": { "url": "https://mcp.posthog.com/mcp", "mode": "streamable-http", "auth_token": "phx_your_personal_api_key", "project_id": "12345", "read_only": true } } ] } ``` ## Verify ```bash theme={null} opensre integrations verify posthog_mcp ``` A successful check connects to the MCP server and reports how many tools it discovered. If it fails, the most common cause is a missing or invalid personal API key — confirm the key was created with the `MCP Server` preset and that outbound HTTPS to `mcp.posthog.com` is allowed. ## Metric report (scheduled) Deliver a per-metric PostHog analytics pulse to Telegram, Slack, or Rocket.Chat, on demand or on a schedule — coworker-style digests (what moved and why it matters), not a raw dashboard dump. Slack delivery needs a bot token (a webhook alone cannot honor `--chat-id`). This uses the headless **posthog-summary** skill path (schema discovery, then one bounded HogQL query per metric), not the investigation pipeline or generic `opensre cron` kinds. It requires the PostHog **MCP** integration above — the REST `posthog` integration alone cannot serve it. | Command | What it does | | ---------------------------------------------------------------------------- | ---------------------------------------------------- | | `opensre posthog report run` | Run once and print the per-metric report to stdout | | `opensre posthog report run --period 30d --metrics "active users,pageviews"` | Run once for a custom window and metric focus | | `opensre posthog report schedule add` | Schedule recurring delivery (cron + provider + chat) | | `opensre posthog report schedule list` | List PostHog report schedules | | `opensre posthog report schedule run TASK_ID` | Run a scheduled report immediately | | `opensre posthog report schedule remove TASK_ID` | Remove a schedule | Example — every Monday at 08:00 London time to Telegram: ```bash theme={null} opensre posthog report schedule add \ --cron "0 8 * * 1" \ --tz Europe/London \ --provider telegram \ --chat-id "-1001234567890" \ --period 7d ``` Example — same schedule to a Slack channel (`C…` member/channel id): ```bash theme={null} opensre posthog report schedule add \ --cron "0 8 * * 1" \ --tz Europe/London \ --provider slack \ --chat-id C0123ABCD \ --period 7d ``` Slack delivery needs `SLACK_BOT_TOKEN`. A `SLACK_WEBHOOK_URL` alone will not work here: a webhook always posts to the one channel it was created for, so it cannot honour `--chat-id`. `--period` accepts a relative window (`24h`, `7d`, `30d`; default `7d`). Optional `--metrics` narrows the report to a comma-separated set instead of the default metric set. The gateway daemon picks up scheduled reports automatically when it is running. If the LLM is unavailable, the run fails with an error (no deterministic fallback). The report compares the current window against the previous comparable window per metric. Real zeros are reported as zeros; failed queries are called out as failures — never silently widened or fabricated, and never presented as zero when the query did not run. # Pull request review flow Source: https://opensre.com/docs/pr-review-flow How Greptile, human review, CI, and automerge work together on OpenSRE PRs. This page is the contributor map for what happens after you open a pull request: Greptile review, required CI, human maintainer judgment, and the optional `automerge` label. Greptile and automerge help maintainers move the queue. They do **not** replace maintainer approval. ## The short path 1. Open a focused PR with the [PR template](https://github.com/Tracer-Cloud/opensre/blob/main/.github/PULL_REQUEST_TEMPLATE.md) filled in. 2. Leave **Allow edits from maintainers** enabled on fork PRs. 3. Wait for CI (the required check is **CI Gate**). 4. Trigger Greptile and work to **Confidence Score: 5/5** with **zero unresolved** threads. 5. A maintainer reviews and either merges, asks for changes, or adds the `automerge` label when the PR is ready to land on its own. Full contribution rules: [CONTRIBUTING.md](https://github.com/Tracer-Cloud/opensre/blob/main/CONTRIBUTING.md). Local checks before you push: [CI.md](https://github.com/Tracer-Cloud/opensre/blob/main/CI.md). ## Greptile (AI review) OpenSRE uses [Greptile](https://greptile.com) for automated review. Opening or reopening a PR posts a reminder comment. Greptile is a **human gate**: aim for **5/5** before you ask a maintainer to merge or add `automerge`. **Trigger a review** with a PR comment: ```text theme={null} @greptile review ``` Give it about **5–10 minutes** (sometimes longer). Fix the feedback, resolve threads you have addressed, then comment `@greptile review` again until you reach **5/5** with no open threads. Optional: the [greploop skill](https://skills.sh/greptileai/skills/greploop) can run that loop for you. Treat Greptile comments like normal review. Reply and leave a thread open only when you need a human decision. ## CI Required checks must be green before merge. The branch-protection aggregator is **CI Gate** (quality + unit tests). Some paths also run Interactive Shell Live checks — those must pass when they are required for your change. If CI is red, fix the branch before asking for another Greptile or maintainer pass. Do **not** open a test-only or CI-only PR just to chase failures that are already red on `main` unless a maintainer asked for that work — see [CONTRIBUTING.md](https://github.com/Tracer-Cloud/opensre/blob/main/CONTRIBUTING.md). ## Human review A maintainer decides when the PR is ready. Greptile 5/5 and green CI are necessary signals, not a merge button. Expect maintainers to look at: * Problem statement and proof in the PR body (demo, screenshot, or log) * AI-usage disclosure when you used AI assistance * Scope (one concern per PR; no refactor-only drive-bys) * Tests for behavioral changes Keep the **PR description** current when the problem, solution, or evidence changes. Comments are for discussion; the body is what reviewers re-read. Need help coordinating? Ask in Discord [#contribute](https://discord.gg/opensre). ## Automerge (maintainer opt-in) The `automerge` label is **maintainer-applied** after Greptile and human review are done. Contributors should not add it themselves unless a maintainer asks. When a PR has `automerge` and is otherwise ready, automation squash-merges it into `main` and deletes the branch. ### What automerge waits for * PR targets `main`, is open, and is not a draft * Required CI (and other non-skipped checks) are green * The branch is mergeable Greptile’s own check status is **not** what automerge waits on — maintainers add the label only after Greptile and human review are finished. ### When the branch is behind `main` Branch protection requires PRs to be up to date with `main`. Automerge only merges when GitHub reports the PR as mergeable. A behind branch usually stays stalled until someone clicks **Update branch** (or rebases onto `main`) and CI goes green again — including for PRs that already have the `automerge` label. Automerge does **not** refresh the branch for you today. ### Paths that never auto-merge PRs that touch these paths always need a **human** to press merge, even with `automerge`: | Prefix | Why | | -------------------- | ----------------------- | | `core/` | Agent runtime | | `platform/` | Multi-tenant platform | | `gateway/` | Chat / gateway surfaces | | `.github/workflows/` | CI and merge machinery | | `.github/scripts/` | Merge machinery | Automerge lists every changed file via the GitHub API (including rename origins). If fewer file entries come back than the PR’s changed-file count, it refuses and asks for a human — so a protected-path change cannot hide behind a partial list. Docs, tests, integrations, and most surface changes can still use automerge when a maintainer labels them. ## Improve a PR during review 1. Read Greptile and maintainer comments as the action list. 2. Push fixes; update the PR description when the story or evidence changed. 3. Resolve threads you have addressed. 4. Re-run `@greptile review` only after the branch and description are current. 5. Keep discussion on the PR when possible; use Discord when you need maintainer coordination or think automation is stuck. ## When things stay quiet * Greptile can take several minutes; do not spam `@greptile review`. * Automerge only runs for the `automerge` label and green checks. * If CI is pending, wait for the next green run. * If the PR is behind `main`, update the branch (or ask a maintainer), then wait for CI again. * If a maintainer is actively editing the branch, let them finish before pushing competing commits. Still blocked after CI is green and Greptile is 5/5? Ping in [#contribute](https://discord.gg/opensre) with the PR link and what you are waiting on. # Prefect Source: https://opensre.com/docs/prefect Connect Prefect so OpenSRE can inspect flow runs, workers, and deployments during investigations OpenSRE queries Prefect to retrieve recent flow runs, their logs, worker health, and deployment status — helping diagnose pipeline failures and identify stalled or crashed orchestration jobs. ## Prerequisites * Prefect Cloud account or self-hosted Prefect Server * API key (Prefect Cloud) or accessible API URL (self-hosted) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Prefect** when prompted and provide your API URL and API key. ### Option 2: Persistent store Add to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "prefect-prod", "service": "prefect", "status": "active", "credentials": { "api_url": "https://api.prefect.cloud/api", "api_key": "your-prefect-api-key", "account_id": "your-account-id", "workspace_id": "your-workspace-id" } } ] } ``` | Field | Default | Description | | -------------- | ------------------------------- | ------------------------------------------ | | `api_url` | `https://api.prefect.cloud/api` | Prefect API URL (override for self-hosted) | | `api_key` | — | Prefect Cloud API key | | `account_id` | — | Prefect Cloud account ID | | `workspace_id` | — | Prefect Cloud workspace ID | ## Prefect Cloud setup 1. In Prefect Cloud, go to your **profile icon** → **API Keys** 2. Click **Create API Key** 3. Copy the key, account ID, and workspace ID from the URL: `https://app.prefect.cloud/account//workspace//` ## Self-hosted Prefect Server For self-hosted Prefect Server, set `api_url` to your server's API endpoint (no API key required if unauthenticated): ```json theme={null} { "api_url": "http://prefect-server:4200/api", "api_key": "" } ``` ## Verify ```bash theme={null} opensre integrations verify prefect ``` Expected output on success (self-hosted, `api_url` set): ``` Service: prefect Status: passed Detail: Configured for Prefect at http://prefect-server:4200/api. ``` For Prefect Cloud (only `api_key` set, no `api_url`): ``` Service: prefect Status: passed Detail: Configured for Prefect at cloud. ``` This is a configuration-presence check (`api_url` or `api_key` is set) rather than a live API call. ## Investigation tools When OpenSRE investigates a Prefect-related alert, three diagnostic tools are available: * **Flow runs** — lists recent flow runs filtered by state (FAILED, CRASHED, etc.) * **Flow run logs** — retrieves log output for a specific flow run * **Workers** — checks worker health and heartbeat status across work pools ## Troubleshooting | Symptom | Fix | | ------------------------- | -------------------------------------------------------------------- | | **401 Unauthorized** | Check your API key and account/workspace IDs | | **Connection refused** | Verify `api_url` is reachable — check firewall rules for self-hosted | | **No flow runs returned** | Flow runs may have been purged — check retention settings | ## Security best practices * Use a **service account API key** rather than a personal key. * For self-hosted, restrict network access to the Prefect API to trusted IPs. * Store credentials in `~/.opensre/integrations.json`, not in source code. # Principal-scoped storage (Slack team installs) Source: https://opensre.com/docs/principal-scoped-storage How OpenSRE gives each Slack user laptop-like session context while sharing org integrations on a multi-tenant gateway. # Principal-scoped storage (Slack team installs) For a **Slack user in a team workspace**, OpenSRE resolves one **principal** (the organization this deployment serves) and one **actor** (the Slack user). Team integrations and billing follow the principal. Conversation sessions follow the actor — the same private context a laptop CLI user keeps under `~/.opensre`. Local CLI and Telegram stay on the flat host home when no org scope is bound. ## Who owns the turn | Context | Principal | Actor | | -------------- | ----------------------------------- | ------------- | | Slack | Organization from `ORGANIZATION_ID` | Slack user id | | Other surfaces | Unchanged (not part of this change) | — | With no organization configured, or a team outside `OPENSRE_SILO_TEAM_IDS`, the Slack turn is **refused** rather than billed to the wrong owner. ## On-disk layout A laptop user keeps everything in `~/.opensre`. A Slack user gets the same private conversation context, filed under their organization. Deployed, the organization root is the mounted S3 Files volume named by `OPENSRE_CONTEXT_ROOT` (`/workspace/memories`). The infrastructure chroots that mount to one organization through a per-org access point, so OpenSRE adds no org segment of its own: ```text theme={null} /workspace/memories/ # = organizations// integrations.json # shared by every member of the org users// # that user's own context sessions/.jsonl memory/ ``` Without the mount, a bound org principal nests under the host home so several organizations can be exercised on one machine: ```text theme={null} ~/.opensre/ gateway/… # host: pidfile, logs integrations.json # laptop / unbound: unchanged sessions/ memory/ orgs// integrations.json users// sessions/… memory/ ``` Integrations sit at the org root because credentials belong to the team: whoever connected Datadog, every member investigates with it. Sessions and memory sit under the user because conversation history is personal, the same way it is on a laptop. With no scope bound — a terminal run — every path resolves to the flat `~/.opensre` layout, unchanged. ## What changes for Slack 1. Resolve `StorageScope` (`principal` + `actor`) at the start of each Slack turn. 2. Bind the scope for the turn (`bound_storage_scope`). 3. Resolve paths: org home for integrations, member home for sessions. 4. Look up / create session bindings with `(platform, chat_id, principal_id, actor_id)`. 5. Consume credits against `principal.id` (the org). Alice and Bob in the same Slack thread get different session files and different binding rows. They share the org's integrations store. ## Other surfaces `principal` and `actor` are optional on the binding store and resolver. Callers that omit them — Telegram, and anything else on main's path — key bindings by empty principal/actor ids, which is exactly the behavior they had before. ## Which organization a deployment serves There is no install catalog. The team → organization mapping is control-plane data the webapp already owns (`organizations.workspace_provider` and the Slack team id on the `organizations` row), so OpenSRE does not keep a second copy that could drift. A deployment is told which organization it serves: ```bash theme={null} export ORGANIZATION_ID=org_... export OPENSRE_SILO_TEAM_IDS=T0123 # recommended in production ``` With `OPENSRE_SILO_TEAM_IDS` set, only those teams are served and every other workspace is refused. Unset, any workspace that installed the app is served from the configured organization — convenient for dogfood, and logged as a warning because it means an uninvited workspace would inherit that organization's credentials. **Consequence:** one process serves one organization. A gateway fronting several workspaces would need the team → organization lookup back, reading the webapp rather than a local catalog. ## Upgrading an existing deployment Session bindings moved out of the host SQLite database into a per-organization `bindings.json` beside that organization's context. SQLite is gone from the live path: the context root is an NFS-backed mount, where its advisory locking is unreliable, while a write-temp-then-rename is atomic. Opening the JSON file for the first time adopts rows from any old SQLite index — including one written before scoping existed, which has neither `principal_id` nor `actor_id`; those adopt as unscoped rows and keep working for Telegram and the CLI. A deployment whose context moved onto a mounted volume also adopts from the host database. Slack members still get a **new session on their first turn after upgrade**: an adopted row carries an empty actor id, so it no longer matches a member-scoped lookup. The transcript survives and remains readable. Session transcripts and `integrations.json` are not copied automatically. To keep continuity on a silo, copy them once: ```bash theme={null} mkdir -p ~/.opensre/orgs/$ORGANIZATION_ID/users cp ~/.opensre/integrations.json ~/.opensre/orgs/$ORGANIZATION_ID/ ``` Telegram and other surfaces keep matching their migrated empty-actor rows. ## Related env vars | Env | Purpose | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ORGANIZATION_ID` | The organization this deployment serves. Required for Slack turns, and required to use `OPENSRE_CONTEXT_ROOT` | | `OPENSRE_CONTEXT_ROOT` | Mounted, org-scoped volume holding this organization's context (`/workspace/memories` in the Slack service). Requires `ORGANIZATION_ID` to say who owns it | | `OPENSRE_SILO_TEAM_IDS` | Comma-separated Slack team ids this deployment serves. Unset, any workspace is served (with a warning); set, every other workspace is refused | ## Non-goals (this change) * CLI emulation of a Slack member * Per-user integration credentials or LLM auth * Nesting `opensre.json`, investigations, or REPL history under member homes * Changing local CLI individual home layout * A database of any kind: bindings are a JSON file * A team → organization catalog, so one process serves one organization # Python API Source: https://opensre.com/docs/python-api Drive the OpenSRE agent in-process from your own Python code Use this when your Python code runs on the same machine as OpenSRE and you want agent answers without shelling out to the CLI. To call OpenSRE over the network instead, use the [HTTP API](/docs/api). ## Prerequisites The standalone CLI installer does not expose an importable package, so embed from a source checkout: ```bash theme={null} git clone https://github.com/Tracer-Cloud/opensre.git cd opensre && make install ``` Configure a provider once (`opensre onboard`) — the session API reuses the same config and credentials as the CLI. Run your script inside the checkout's environment with `uv run python your_script.py`. Register adapters before the first turn (tools and investigation): ```python theme={null} from bootstrap.process import EMBEDDED_PROFILE, configure_process configure_process(EMBEDDED_PROFILE) ``` ## One API — chat and investigate Every surface uses the same two verbs: ```python theme={null} from core.agent_harness import AgentSession session = AgentSession.start() result = session.chat("why is checkout-api slow?") if result.answered: print(result.primary_response_text) report = session.investigate({"alert_name": "HighLatency", "severity": "warning"}) print(report.report) ``` `AgentSession.start()` resolves the environment, opens a session, and attaches an agent with the standard ports — the same tools and prompts the interactive shell uses. `investigate` does not require an attached chat agent; it uses the payload runner installed by `configure_process`. Always check `result.answered` before trusting chat text: when a turn fails (for example the LLM provider is unreachable), the error message itself lands in `result.primary_response_text`. ### Internal seams (not for hosts) Chat hosts terminate at `dispatch_chat_turn` → `run_turn`. Investigation terminates at the installed payload runner (`run_investigation_payload`). Do not invent parallel public entrypoints. ## A conversation Each `chat` call is one turn in the same session, so follow-ups see earlier context: ```python theme={null} session.chat("list unresolved Sentry issues from the last 24 hours") result = session.chat("which of those affect checkout?") ``` To resume a previous session instead of starting a new one, pass its ID: ```python theme={null} from core.agent_harness import AgentSession, SessionConfig session = AgentSession.start(SessionConfig(session_id="abc123")) ``` ## Your own grounding context The agent builds its prompts from the session by default. To ground it your own way — a different system prompt, your own retrieved context — pass a provider: ```python theme={null} from core.agent_harness import AgentSession, SessionConfig session = AgentSession.start(SessionConfig(prompts=my_provider)) ``` Omit it and the built-in provider applies (`core.agent_harness.DefaultPromptContextProvider`). A custom provider must satisfy `core.agent_harness.ports.PromptContextProvider`. The same `prompts=` argument is accepted by `build_default_headless_agent` on the custom-ports path below. ## Custom output and ports `start()` buffers all output. To capture tool progress yourself — stream to a websocket, collect for a report — build the agent explicitly and pass your own sink: ```python theme={null} from core.agent_harness import ( AgentSession, BufferOutputSink, SessionConfig, build_default_headless_agent, ) session = AgentSession(SessionConfig()) startup = session.startup() sink = BufferOutputSink() agent = build_default_headless_agent( session=startup.session, output=sink, prompts=my_provider, # optional — same port as SessionConfig.prompts ) session.attach_agent(agent) session.chat("summarize open incidents") print(sink.lines) # rendered output lines print(sink.streamed) # streamed answer chunks ``` Any object implementing the `OutputSink` protocol (`core.agent_harness.ports.OutputSink`: `print`, `render_response_header`, `render_error`, `stream`) works in place of `BufferOutputSink`. `build_default_headless_agent` also accepts a custom logger, prompt surface, and tool observers — see its docstring for the full port list. ## Your own tools Register a package before the first tool lookup: ```python theme={null} from tools.registry import register_external_tool_package import my_agent_tools register_external_tool_package(my_agent_tools) ``` Two gotchas: 1. Declare tools with `surfaces=("action",)` (or include `"action"`). The `@tool` default is `("investigation",)` only — the chat/Python-API action loop will not see investigation-only tools. 2. Tools may live in the package's `__init__.py` or in submodules; both are scanned after registration. # Quickstart Source: https://opensre.com/docs/quickstart Choose one install method. The one-line installer tracks the latest stable build from `main`. ```bash theme={null} brew tap tracer-cloud/tap brew install tracer-cloud/tap/opensre ``` ```bash theme={null} curl -fsSL https://install.opensre.com | bash ``` Installing OpenSRE with the one-line curl installer ```powershell theme={null} irm https://install.opensre.com | iex ``` Installing OpenSRE on Windows with the PowerShell installer Run the onboarding wizard to configure your LLM provider and connect your integrations (Grafana, Datadog, Honeycomb, Coralogix, Slack, AWS, GitHub MCP, Sentry): ```bash theme={null} opensre onboard ``` After you pick a provider and a model, the wizard sends one tiny request to that provider with the credential you just entered — so a typo, an expired key, or an Ollama model you have not pulled yet is caught here, not on your first investigation. Onboarding OpenSRE with the setup wizard to configure your LLM provider and integrations Onboarding OpenSRE on Windows with the setup wizard Choose either path — both use the same local `opensre` binary: **Interactive prompt shell** — start OpenSRE with no arguments (TTY) to type incidents in plain language and use slash commands (`/help`, `/status`, `/effort`, `/exit`, …): ```bash theme={null} opensre ``` **Direct investigation** — one-shot run from your shell with an alert file: ```bash theme={null} opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` For file-based runs, OpenSRE fetches alert context, reasons across connected systems, and generates a structured root-cause report. Running an OpenSRE investigation on a sample alert through to a structured root-cause report Running an OpenSRE investigation on Windows through to a structured root-cause report ```bash theme={null} opensre update ``` ## Uninstall To remove opensre and all its local data from your machine: ```bash theme={null} opensre uninstall ``` Pass `--yes` to skip the confirmation prompt: ```bash theme={null} opensre uninstall --yes ``` ## Troubleshooting * **Docker is not running**: Start Docker Desktop, OrbStack, or Colima before running setup. * **`make` is missing**: Install it via your package manager (`brew install make` on macOS, `choco install make` on Windows). * **No local LLM provider is configured**: Run `opensre onboard` to select and configure your LLM credentials. * **Onboarding says your key could not be verified, but you know it is valid**: The check needs to reach the provider, so a firewall, a proxy, or being offline will fail it. Choose **Save anyway without validating** to keep the key and move on — you can rerun `opensre onboard` to re-enter it later. * **Onboarding could not save your key**: Follow the setup steps it prints for your system keychain, then choose **Retry saving to the system keychain**. To get going right now, choose **Continue without saving (this session only)** — the key works for this run and you re-enter it next time. # Quickstart Source: https://opensre.com/docs/quickstart-monitoring Start monitoring your first pipelines in minutes ## 2 minutes walkthrough Tracer runs anywhere from the OS, just follow the steps below: Use the Tracer CLI and dashboard below to start tracking your pipelines in minutes. ```bash theme={null} curl -sSL https://install.tracer.cloud | sh -s ``` ```bash theme={null} sudo tracer login ``` This will open up a browser window to log in to your Tracer account. ```bash theme={null} sudo tracer init ``` You will be prompted to configure the pipeline name. Filling this out ensures that each pipeline is uniquely identifiable, customizable, and easy to search later on.
Tracer is now tracking your pipeline.
Every run you launch for this pipeline will be automatically monitored.
Note: You will only need to run tracer init again for a new pipeline, not per pipeline run.
```bash theme={null} sudo tracer demo ``` Alternatively: Run your usual command (Nextflow, Snakemake, WDL, etc.)
Tracer works automatically once the agent is active. Watch your pipeline run in the [Tracer dashboard](https://app.tracer.cloud/). See all our environment specific guides below: | [Linux (Cloud)](/docs/environments/linux-cloud) | [macOS](/docs/environments/macos) | [AWS Batch](/docs/environments/aws-batch) | [Linux (Local)](/docs/environments/linux-local) | [Docker](/docs/environments/docker) | [Windows](/docs/environments/windows-local) | | :----------------------------------------: | :--------------------------: | :----------------------------------: | :----------------------------------------: | :----------------------------: | :------------------------------------: | Let's start monitoring! # RabbitMQ Source: https://opensre.com/docs/rabbitmq Connect RabbitMQ so OpenSRE can diagnose queue backlogs, consumer issues, and broker health during investigations OpenSRE uses the RabbitMQ Management HTTP API to investigate message-bus incidents — checking queue backlogs, consumer health, broker-wide resource usage, cluster partition state, and connection patterns. ## Prerequisites * RabbitMQ 3.12+ (3.13 recommended) * The **`rabbitmq_management`** plugin must be enabled on the broker: ```bash theme={null} rabbitmq-plugins enable rabbitmq_management ``` * Network access from the OpenSRE environment to the Management API port (default **15672**) * A user with at least the **`monitoring`** tag (read-only access to all management endpoints) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup rabbitmq ``` You will be prompted for host, management port, username, password, vhost, and whether to enable SSL. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} RABBITMQ_HOST=rmq.example.com RABBITMQ_MANAGEMENT_PORT=15672 RABBITMQ_USERNAME=opensre_ro RABBITMQ_PASSWORD=... RABBITMQ_VHOST=/ RABBITMQ_SSL=false RABBITMQ_VERIFY_SSL=true ``` | Variable | Default | Description | | -------------------------- | --------- | --------------------------------------------------------------------------------------------- | | `RABBITMQ_HOST` | — | **Required.** RabbitMQ server hostname or IP | | `RABBITMQ_MANAGEMENT_PORT` | `15672` | Management API HTTP port (use `15671` for HTTPS) | | `RABBITMQ_USERNAME` | — | **Required.** Management API user | | `RABBITMQ_PASSWORD` | *(empty)* | Management API password | | `RABBITMQ_VHOST` | `/` | Target vhost — diagnostic queries are scoped to this vhost | | `RABBITMQ_SSL` | `false` | Use HTTPS instead of HTTP for the Management API | | `RABBITMQ_VERIFY_SSL` | `true` | Verify the server TLS certificate; set `false` only for self-signed certs in trusted networks | ### Option 3: Persistent store Credentials are automatically persisted to `~/.opensre/integrations.json` with `0o600` permissions: ```json theme={null} { "version": 1, "integrations": [ { "id": "rabbitmq-prod", "service": "rabbitmq", "status": "active", "credentials": { "host": "rmq.example.com", "management_port": 15672, "username": "opensre_ro", "password": "...", "vhost": "/", "ssl": false, "verify_ssl": true } } ] } ``` ## Recommended user setup Create a dedicated monitoring user so OpenSRE has read-only access: ```bash theme={null} # Create user rabbitmqctl add_user opensre_ro strong-password # Grant the monitoring tag (read-only management API access) rabbitmqctl set_user_tags opensre_ro monitoring # Grant read-only permissions on the target vhost rabbitmqctl set_permissions -p / opensre_ro "^$" "^$" ".*" ``` The `monitoring` tag grants read access to all management endpoints without the ability to publish, consume, create, or delete any resources. The permissions line grants no configure or write access (`^$` matches nothing), and read access to all resources (`.*`). ## TLS configuration SSL is disabled by default because most RabbitMQ Management API deployments use HTTP internally. For production environments exposed over the network, enable HTTPS: ```bash theme={null} RABBITMQ_SSL=true RABBITMQ_MANAGEMENT_PORT=15671 ``` Set `RABBITMQ_VERIFY_SSL=false` only when connecting to brokers with self-signed certificates in trusted networks. ## Investigation tools When OpenSRE investigates a RabbitMQ-related alert, five diagnostic tools are available: ### Queue backlog Lists queues ranked by pending message count (ready + unacknowledged). Returns queue name, vhost, state, message counts, consumer count, consumer utilisation, memory usage, and publish/deliver/ack rates. Results are scoped to the configured vhost. ### Consumer health Lists active consumers with per-queue diagnostics: consumer tag, ack mode, prefetch count, active state, and the channel/connection each consumer is bound to. Helps identify stalled or missing consumers behind a growing backlog. ### Broker overview Returns a cluster-wide summary: RabbitMQ version, cluster name, total message counts, publish/deliver rates, queue/consumer/connection/channel totals, plus alarm health-check status (memory, disk, and file-descriptor alarms). ### Node health Returns per-node resource utilisation: memory used vs. limit (with alarm flag), disk free vs. limit (with alarm flag), file descriptors, sockets, Erlang process usage, and cluster partition state. Essential for diagnosing backpressure, partitions, or node-level resource exhaustion. ### Connection stats Lists active connections sorted by receive byte rate. Reports user, vhost, protocol, channel count, peer host/port, TLS status, and recv/send byte rates. Helps spot connection exhaustion, slow consumers, or noisy publishers. Results are filtered to the configured vhost. ## Verify ```bash theme={null} opensre integrations verify rabbitmq ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL rabbitmq local env passed Connected to RabbitMQ 3.13.0 (cluster: rabbit@prod-01, vhost: /). ``` ## Troubleshooting | Symptom | Fix | | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **Connection refused on port 15672** | Verify the management plugin is enabled (`rabbitmq-plugins enable rabbitmq_management`) and that the port is reachable from the OpenSRE host. | | **Management API not found (404)** | The `rabbitmq_management` plugin is not enabled. Run `rabbitmq-plugins enable rabbitmq_management` and restart the broker if needed. | | **Authentication failed (401)** | Confirm the username/password and that the user exists (`rabbitmqctl list_users`). | | **Forbidden (403)** | The user lacks sufficient tags. Grant at least `monitoring`: `rabbitmqctl set_user_tags opensre_ro monitoring`. | | **SSL: CERTIFICATE\_VERIFY\_FAILED** | The server certificate is not trusted by the system CA bundle. Install the correct CA or set `RABBITMQ_VERIFY_SSL=false` in trusted networks. | | **Queues/consumers from other vhosts appear** | Check that `RABBITMQ_VHOST` is set correctly. Queue and consumer queries are scoped to this vhost. Connection stats are filtered client-side. | | **Empty consumer list** | Confirm consumers are connected to queues on the configured vhost. Check with `rabbitmqctl list_consumers -p /your-vhost`. | ## Security best practices * Use a **dedicated `monitoring` user** — never the default `guest` account or an `administrator`-tagged user. * Always enable **TLS** when the Management API is exposed over the network. * Keep passwords out of source control — use `.env` or the persistent store. * Rotate credentials periodically. * The Management API is **read-only** from OpenSRE's perspective — no messages are published, consumed, or deleted. # Railway Source: https://opensre.com/docs/railway Configure Railway deployment inspection and confirmed service redeploys ## Overview The Railway integration inspects the latest successful deployment of any configured Railway service and can request a redeploy after explicit confirmation. ## Setup Install the Railway CLI, then either authenticate it interactively or configure a Railway token: ```bash theme={null} npm install -g @railway/cli railway login opensre integrations setup railway ``` The setup flow accepts an optional token and default project, service, and environment. The token is used when present; an authenticated Railway CLI can be used without one. Tool calls can override any configured default scope. Environment configuration is also supported: ```bash theme={null} RAILWAY_TOKEN= RAILWAY_PROJECT=project-id-or-name RAILWAY_SERVICE=service-id-or-name RAILWAY_ENVIRONMENT=production RAILWAY_PATH=railway ``` Verify the local integration with: ```bash theme={null} opensre integrations verify railway ``` ## Tools | Tool | What it does | Confirmation | | ---------------------------- | ------------------------------------------------------------------------------------------------- | -------------------- | | `inspect_railway_deployment` | Shows the latest successful deployment and available source commit metadata for a Railway service | No | | `redeploy_railway_service` | Requests a Railway service redeploy | Yes (`confirm=true`) | ### Inspect a deployment When a default scope is configured, no parameters are required. Otherwise provide the service scope: ```json theme={null} { "project": "project-id-or-name", "service": "service-id-or-name", "environment": "production" } ``` The result includes deployment ID, status, and source commit hash/message when Railway provides them. ### Redeploy a service ```json theme={null} { "project": "project-id-or-name", "service": "service-id-or-name", "environment": "production", "confirm": true } ``` `confirm` must be explicitly `true`. Railway reuses the latest deployment and returns its new deployment ID. Commands always use explicit project, service, and environment flags and never write Railway link state into the workspace. ## Errors and security * `deployment_unavailable` means Railway returned no successful deployment in the inspected history. * `confirmation_required` means redeploy was called without `confirm=true`. * Missing configuration, CLI, or authentication failures return structured errors. * Railway tokens are never returned in tool output; Railway commands are non-interactive and errors are redacted. # AWS RDS Source: https://opensre.com/docs/rds Investigate AWS RDS instance health and recent events during incidents OpenSRE uses AWS RDS to investigate database instance health and surface recent operational events — failovers, maintenance windows, parameter changes, and backup activity — when an alert fires against a managed RDS database. All RDS API calls are read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot mutate your RDS resources. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/aws) (role ARN recommended) * An RDS DB instance you want OpenSRE to investigate * IAM permissions for the two RDS describe actions listed below ## Setup ### Environment variables ```bash theme={null} RDS_DB_INSTANCE_IDENTIFIER=prod-orders-db AWS_REGION=us-east-1 ``` | Variable | Default | Description | | ---------------------------- | ----------- | ---------------------------------------------------------------------------------------------------- | | `RDS_DB_INSTANCE_IDENTIFIER` | — | Required. The DB instance identifier OpenSRE should investigate. | | `AWS_REGION` | `us-east-1` | AWS region the instance lives in. Used by both the integration config and per-tool param extraction. | | `RDS_REGION` | `us-east-1` | Fallback used only when `AWS_REGION` is not set. | Region resolution order (highest priority first): 1. `region` field on the source dict (when configured via the integrations store) 2. `AWS_REGION` environment variable 3. `RDS_REGION` environment variable 4. `us-east-1` (default) ## IAM permissions The integration only needs two read-only RDS actions: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "rds:DescribeDBInstances", "rds:DescribeEvents" ], "Resource": "*" } ] } ``` Attach this policy to the same IAM role or user already configured for the [AWS integration](/docs/aws). If you are already using the AWS managed `ReadOnlyAccess` policy, both actions are already covered. ## Tools | Tool | AWS API call | What it returns | | ----------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `describe_rds_instance` | `rds:DescribeDBInstances` | Instance status, engine + version, instance class, Multi-AZ flag, endpoint address/port, storage type and size, availability zone, and backup window. | | `describe_rds_events` | `rds:DescribeEvents` | Recent events for the DB instance — failovers, maintenance, parameter group changes, and backup activity. Defaults to the last 60 minutes; bounded to 20160 minutes (14 days, the AWS limit). | Both tools become available to the planner whenever `rds.db_instance_identifier` is present in the resolved sources. ### Use cases * Verifying RDS instance status (`available`, `modifying`, `failed`) when an alert fires * Detecting Multi-AZ failover events around an incident timestamp * Tracing recent maintenance, parameter group changes, or backup activity that may correlate with the incident ## Troubleshooting | Symptom | Fix | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | **AccessDenied on `rds:DescribeDBInstances`** | Add the IAM policy above to the role or user used by the AWS integration. | | **DBInstanceNotFound** | Confirm `RDS_DB_INSTANCE_IDENTIFIER` matches an instance in `AWS_REGION`. | | **Tool reports the wrong region** | Either `AWS_REGION` is set to a different region, or the source dict has a stale `region` field. Check the resolution order above. | ## Upstream correlation validation OpenSRE also includes a deterministic upstream-correlation smoke validation path for no-trace-ID RDS CPU spike investigations. This allows validating correlation output locally without requiring live Datadog credentials or full LLM investigation setup. ### Local smoke validation Run: ```bash theme={null} opensre tests upstream-correlation-smoke ``` Expected output includes separate sections for: * correlated signals * most likely causal driver(s) Example: ``` Upstream Correlation Smoke Validation Correlated signals: - upstream-correlation (source=runtime, score=0.9) Most likely causal driver(s): - system.cpu.user{service:orders-web} (confidence=0.9) rationale=time_window=1.0, topology=1.0, periodicity=1.0, operator_hint=0.0 ``` For machine-readable output: ```bash theme={null} opensre tests upstream-correlation-smoke --json ``` ### Live investigation validation For live validation, configure Datadog and trigger an investigation against an RDS CPU spike alert. The upstream correlation runtime automatically scopes RDS metrics to the alerting DB instance using the `dbinstanceidentifier` tag to avoid cross-instance aggregation in multi-RDS environments. Recommended alert fields: ```json theme={null} { "service": "orders", "resource": "orders-rds-prod", "upstream_services": ["orders-web"] } ``` Then run a live investigation: ```bash theme={null} opensre investigate ``` When runtime evidence is available, the final report includes: * correlated signals * most likely causal driver(s) This validation flow is intended as a lightweight smoke/integration path and does not require synthetic benchmark execution. # Redis Source: https://opensre.com/docs/redis Connect Redis so OpenSRE can diagnose cache, queue, and session issues during investigations OpenSRE uses Redis diagnostics to investigate cache and key-value store alerts — checking memory pressure and eviction rates, surfacing slow commands, monitoring replication lag, and inspecting key counts and TTLs. Redis is one of the most common components in SRE stacks (caching, queues, session storage, rate limiting), and these tools give an investigation visibility into all of them. ## Prerequisites * Redis 5.0+ (or a compatible server such as Valkey) * Network access from the OpenSRE environment to your Redis instance * Credentials, if authentication (`requirepass` or ACLs) is enabled ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Redis** when prompted and provide your host and port. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} REDIS_HOST=localhost REDIS_PORT=6379 REDIS_USERNAME= REDIS_PASSWORD= REDIS_DATABASE=0 REDIS_SSL=false ``` | Variable | Default | Description | | ---------------- | --------- | ----------------------------------------------------------- | | `REDIS_HOST` | — | **Required.** Redis hostname or IP | | `REDIS_PORT` | `6379` | Redis port | | `REDIS_USERNAME` | *(empty)* | ACL username (Redis 6+); leave blank for password-only auth | | `REDIS_PASSWORD` | *(empty)* | Password (`requirepass` or ACL) | | `REDIS_DATABASE` | `0` | Database number to inspect | | `REDIS_SSL` | `false` | Connect using TLS | ### Option 3: Persistent store Integrations are automatically persisted to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "redis-prod", "service": "redis", "status": "active", "credentials": { "host": "cache.example.net", "port": 6379, "username": "", "password": "s3cret", "db": 0, "ssl": true } } ] } ``` ## Authentication * **Password only** (`requirepass`): set `REDIS_PASSWORD` and leave `REDIS_USERNAME` blank. * **ACL user** (Redis 6+): set both `REDIS_USERNAME` and `REDIS_PASSWORD`. * **No auth**: leave both blank (development only). ## TLS configuration Set `REDIS_SSL=true` to connect over TLS. Confirm the server has TLS enabled (e.g. `tls-port 6379`). ## Investigation tools When OpenSRE investigates a Redis-related alert, seven read-only diagnostic tools are available: ### Server info Retrieves version, uptime, memory usage (used, RSS, peak, `maxmemory`, fragmentation ratio, eviction policy), connected/blocked clients, throughput and hit/miss counters, eviction and expiry counts, and per-database keyspace statistics. Useful for spotting memory pressure, high eviction rates, or connection saturation. ### Slow log Returns recent `SLOWLOG` entries — the command, execution duration (microseconds), and originating client. Surfaces expensive commands such as large `KEYS`, `SMEMBERS`, or `SORT` operations. ### Replication Reports the node role, master link health (for replicas), connected replicas, and per-replica offset lag in bytes (for masters). Identifies broken replication or replicas falling behind. ### Key scan Counts keys matching a glob pattern and samples their TTL and type. Key discovery uses the non-blocking `SCAN` cursor — never `KEYS` — so it is safe to run against large production keyspaces. Total iteration is capped (10,000 keys) and TTL/type sampling is bounded, so a wide pattern can never run unbounded. ### Client list Summarizes connected clients via `CLIENT LIST`: total connections, how many are blocked (waiting on `BLPOP`/`BRPOP`/`XREAD`), how many are in pub/sub mode, the longest idle time, and breakdowns by source address and last command. Surfaces connection-pool exhaustion and stuck or blocked clients. Aggregate counts cover every client; the per-client sample is bounded. ### List/queue depth Reports a list key's depth via `LLEN`, with an optional bounded head/tail sample via `LRANGE`. Useful for job-queue backlogs and stuck workers (Sidekiq, Celery, Bull, Resque-style queues). The key's `TYPE` is checked first, so a missing key reports `exists: false` and a non-list key returns a clear message rather than a `WRONGTYPE` error. Each sampled element is length-capped. ### Latency doctor Runs `LATENCY DOCTOR` for a human-readable diagnosis of recent latency spikes (fork/RDB save, AOF rewrite, blocking commands, slow disk) and lists the latest monitored events via `LATENCY LATEST`; an optional `event` argument adds `LATENCY HISTORY` for that event. Latency monitoring must be enabled for events to be recorded — set `latency-monitor-threshold` to a value greater than `0` (in milliseconds). The tool reads this threshold via `CONFIG GET`, so `monitoring_active` reflects whether monitoring is *enabled* (a healthy, enabled-but-quiet server reports `monitoring_active: true` with no events). When the threshold is `0`, `monitoring_active` is `false` and `monitoring_threshold_ms` is `0`; the `report` field still carries Redis's raw `LATENCY DOCTOR` output (Redis itself does not emit a special "disabled" message). ## Verify ```bash theme={null} opensre integrations verify redis ``` Expected output: ``` Service: redis Status: passed Detail: Connected to Redis 7.2.4 at localhost:6379; database 0. ``` ## Troubleshooting | Symptom | Fix | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Connection refused** | Verify the host/port, check firewalls, and ensure Redis is running and bound to a reachable interface (`protected-mode`). | | **Authentication failed (NOAUTH/WRONGPASS)** | Set `REDIS_PASSWORD`. For ACL users, also set `REDIS_USERNAME`. | | **No permissions (NOPERM)** | Grant the user read access to the diagnostic commands it needs: `INFO`, `CLIENT`, `SLOWLOG`, `LATENCY`, `TYPE`, `LLEN`/`LRANGE`, `SCAN`/`TTL` (and `CONFIG GET` for the latency-monitoring threshold). | | **TLS handshake failed** | Set `REDIS_SSL=true`; confirm the server has TLS enabled. | | **Empty replication / no replicas** | Expected for a standalone instance — the role is reported as `master` with no replicas. | ## Security best practices * Use a **read-only** Redis ACL user for monitoring — the tools never write. * Always enable **TLS** (`REDIS_SSL=true`) for connections over untrusted networks. * Store the host and password in `.env`, never in code. * Rotate credentials periodically. # Remote Runtime Investigation Source: https://opensre.com/docs/remote-runtime-investigation Start an RCA investigation for a deployed service by name ## Overview `opensre investigate --service ` kicks off a runtime investigation for a deployed service. Instead of passing an alert payload, OpenSRE gathers live signals from the service (deployment status, recent logs, health probe) and feeds them into the existing investigation pipeline as evidence. ## Prerequisites You must have deployed an OpenSRE service (see [Deployment](/docs/deployment)), registered a named remote, and configured a remote ops provider. 1. **Deploy and register a named remote:** ```bash theme={null} # Deploy via your hosting provider, then register the agent URL: opensre remote --url https://my-service.up.railway.app health # or use the interactive wizard: opensre remote ``` 2. **Configure the remote ops provider (once):** ```bash theme={null} opensre remote ops status # prompts for provider/project/service on first run ``` ## Usage ```bash theme={null} opensre investigate --service ``` For example: ```bash theme={null} opensre investigate --service api-backend opensre investigate --service api-backend --output ./rca.json ``` The command will: 1. Resolve `` against your named-remote registry 2. Fetch deployment status via the configured ops provider (e.g. Railway) 3. Fetch the most recent \~100 log lines 4. Probe the service's `/health` or `/ok` endpoint 5. Package all of this into an alert payload 6. Run the standard RCA pipeline against it The output is the same structured RCA report you'd get from running `opensre investigate -i `. ## Incorporating Slack thread context Pass `--slack-thread CHANNEL/TS` to also pull the messages from a specific Slack thread as investigation context. This is useful when an incident originated in a Slack conversation. ```bash theme={null} export SLACK_BOT_TOKEN=xoxb-... opensre investigate --service api-backend --slack-thread C01234/1712345.000001 ``` Requirements: * `SLACK_BOT_TOKEN` must be set in the environment. The bot must have the `channels:history` and `groups:history` OAuth scopes for the channel you're reading. * The `CHANNEL/TS` reference can be obtained from Slack's "Copy link to message" option — it's the last two path segments of the link. The thread's messages, users, timestamps, and reactions are fetched via Slack's `conversations.replies` API and included under the `slack_thread` key in the alert payload. If fetching fails (bad token, wrong channel, network error), the investigation still proceeds with the error recorded in the payload. ## Mutual exclusion `--service` cannot be combined with `--input`, `--input-json`, `--interactive`, or `--print-template`. Use `--service` on its own. ## Extending to other providers The `RemoteOpsProvider` abstract class (in `infra/deployment/remote/ops.py`) defines the provider interface. To add support for another provider (EC2, ECS, Vercel, etc.), implement a new subclass with `status()`, `logs()`, `fetch_logs()`, and `restart()` methods, then register it in `resolve_remote_ops_provider()`. ## Known limitations * **Currently supports only Railway** — other providers have `status`/`logs` hooks but no `fetch_logs` implementation yet. * **Slack context is thread-scoped** — this initial version pulls a specific thread via `--slack-thread`. It does not search Slack history or resolve linked runbooks. * **`alert_source` is re-inferred by the LLM** — the LLM in the extract-alert step may infer an `alert_source` from the log text (e.g. "datadog" if the logs mention Datadog), which routes to provider-specific tools. This is the intended behavior. # AWS S3 Source: https://opensre.com/docs/s3 Inspect S3 objects, metadata, and pipeline markers during investigations OpenSRE uses AWS S3 to read object metadata and content when an alert points at data in a bucket — inspecting the files a pipeline produced, sampling an object to spot a schema change, pulling an audit payload, or confirming a run finished by checking for a completion marker. All S3 access is read-only and routed through the shared AWS client, so the integration cannot mutate or delete your objects. ## Prerequisites * AWS credentials with S3 read access, configured per the [AWS integration](/docs/aws) * An S3 bucket you want OpenSRE to read from * IAM permissions for the two S3 actions listed below ## Setup S3 has no credentials of its own — it uses the same AWS credentials as the rest of the [AWS integration](/docs/aws). The S3 tools build their client from the ambient AWS credential chain: environment keys, a shared credentials profile, or an attached instance/task role — plus the region. ```bash theme={null} AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY AWS_REGION=us-east-1 ``` `AWS_ROLE_ARN` is assumed only by `opensre integrations verify aws` — the S3 tools do **not** assume it. The credentials above must have S3 read access themselves. The bucket, key, and prefix OpenSRE reads are **not** environment variables — they come from the alert's resolved sources. Each S3 tool turns on for the planner only when the sources carry the fields it needs: | Source key | Fields | Turns on | | -------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `s3` | `bucket`, `key`, `prefix` | `list_s3_objects` (needs `bucket`), `inspect_s3_object` / `get_s3_object` (need `bucket` + `key`), `check_s3_marker` (needs `bucket` + `prefix`) | | `s3_audit` | `bucket`, `key` | `get_s3_object` — fetch an audit payload referenced by an alert | | `s3_processed` | `bucket` (`prefix` optional) | `check_s3_marker` — check a pipeline's output location | ## IAM permissions The integration only needs two read-only S3 actions: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject" ], "Resource": "*" } ] } ``` Attach this to the same IAM role or user configured for the [AWS integration](/docs/aws). If you already use the AWS managed `ReadOnlyAccess` policy, both actions are covered. ## Tools | Tool | IAM action | What it returns | | ------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------- | | `list_s3_objects` | `s3:ListBucket` | Object keys under an optional prefix, each with size, last-modified, ETag, and storage class (up to 100 keys). | | `inspect_s3_object` | `s3:GetObject` | Object metadata — size, content type, ETag, version ID, user metadata — plus a decoded text sample of the first few KB. | | `get_s3_object` | `s3:GetObject` | The full object body (up to 1 MB) as text, for audit payloads, config, or manifest files. | | `check_s3_marker` | `s3:ListBucket` | Whether a completion marker is present under a prefix, plus the file list — a signal that a pipeline run finished. | ### Use cases * Listing what a pipeline wrote to a bucket, or confirming an expected file is present * Sampling an object to detect a schema or format change in upstream input data * Reading an audit payload or manifest to trace an external vendor interaction * Confirming a batch job completed by checking for its `_SUCCESS` marker ## Verify S3 shares the AWS integration's verification. Verify credentials and region with: ```bash theme={null} opensre integrations verify aws ``` Expected output: ``` Service: aws Status: passed Detail: Connected to AWS STS via static-creds in us-east-1; caller identity account=123456789012 arn=arn:aws:iam::123456789012:user/opensre. ``` This confirms the credentials S3 rides on. It does not test bucket-level access — that is exercised the first time a tool runs against a bucket. ## Troubleshooting | Symptom | Fix | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **S3 tools never run** | The alert's sources have no `s3` entry, or it lacks the required field — object-level tools need both `bucket` **and** `key`, not just `bucket`. | | **AccessDenied on `GetObject` / `ListBucket`** | Attach the IAM policy above to the credentials the S3 tools actually use — the ambient credential chain (env keys or instance/task role), not only a role that the verifier assumes. | | **`check_s3_marker` reports no marker when the run finished** | It only counts files whose key contains `_SUCCESS` or `marker` (case-insensitive). A completion file named anything else won't be detected. | | **A binary object returns metadata but no readable text** | The object isn't UTF-8 (e.g. Parquet or gzip); you still get size, type, and ETag, but there is no text sample to read. | # Sentry Source: https://opensre.com/docs/sentry Connect Sentry so OpenSRE can surface error trends and issue details during investigations OpenSRE queries Sentry to retrieve recent issues, error events, and stack traces — correlating application errors with infrastructure alerts to identify root causes faster. ## Prerequisites * Sentry account with at least one organization * Auth token with `event:read` scope (Issues lookup) * For uptime watching: also `alerts:read` (or `org:read`) so `opensre sentry uptime` can list monitors ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Sentry** when prompted and provide your organization slug and auth token. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} SENTRY_ORG_SLUG=your-organization-slug SENTRY_AUTH_TOKEN=sntrys_your_token SENTRY_URL=https://sentry.io # optional, for self-hosted Sentry SENTRY_PROJECT_SLUG=my-project # optional, to scope to one project SENTRY_STATS_PERIOD=24h # optional, issue search time window ``` | Variable | Default | Description | | --------------------- | ------------------- | -------------------------------------------------------------------------------------- | | `SENTRY_ORG_SLUG` | — | **Required.** Your Sentry organization slug | | `SENTRY_AUTH_TOKEN` | — | **Required.** Sentry auth token with `event:read` (add `alerts:read` for uptime watch) | | `SENTRY_URL` | `https://sentry.io` | Override for self-hosted Sentry | | `SENTRY_PROJECT_SLUG` | — | Scope queries to a specific project | | `SENTRY_STATS_PERIOD` | `24h` | Time window for issue searches (e.g. `24h`, `14d`, `90d`) | A search returns up to 100 issues per query (Sentry's maximum page size) within the `SENTRY_STATS_PERIOD` window. Widen the window (e.g. `SENTRY_STATS_PERIOD=14d`) to surface older issues. If you expect more issues than appear, the cause is almost always the time window or a `SENTRY_PROJECT_SLUG` scope — not a cap of one. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "sentry-prod", "service": "sentry", "status": "active", "credentials": { "base_url": "https://sentry.io", "organization_slug": "your-org", "auth_token": "sntrys_your_token", "project_slug": "my-project" } } ] } ``` ## Creating an auth token **Recommended: Organization Token** 1. In Sentry, go to **Settings** → **Developer Settings** → **Organization Tokens** 2. Click **Create New Token** 3. Enable the `event:read` scope (and `alerts:read` if you use uptime watch) 4. Copy the token **Alternative: Internal Integration** For broader access, create an Internal Integration under **Settings** → **Developer Settings** → **Internal Integrations**. The organization slug appears in your Sentry URL: `https://sentry.io/organizations//` ## Verify ```bash theme={null} opensre integrations verify sentry ``` Expected output: ``` Service: sentry Status: passed Detail: Sentry validated for org your-org; 30 issue(s) in the last 7 days ``` ## Morning digest (scheduled) Deliver a daily unresolved-issues summary to Slack, Telegram, or Rocket.Chat. This uses the headless **sentry-summary** skill path (tools + LLM answer via the gateway), not the investigation pipeline or generic `opensre cron` kinds. When an uptime watch schedule is running, the skill also calls `get_sentry_uptime_digest` to include downtime/recovery context in the digest. | Command | What it does | | ----------------------------------------------- | ------------------------------------------------ | | `/sentry digest schedule add …` | Same as CLI — schedule from the REPL | | `opensre sentry digest run` | Run once and print the digest to stdout | | `opensre sentry digest schedule add` | Schedule daily delivery (cron + provider + chat) | | `opensre sentry digest schedule list` | List Sentry digest schedules | | `opensre sentry digest schedule run TASK_ID` | Run a scheduled digest immediately | | `opensre sentry digest schedule remove TASK_ID` | Remove a schedule | Example — weekdays at 08:00 London time to Telegram: ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * 1-5" \ --tz Europe/London \ --provider telegram \ --chat-id "-1001234567890" ``` Example — same schedule to a Slack channel (`C…` member/channel id): ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * 1-5" \ --tz Europe/London \ --provider slack \ --chat-id C0123ABCD ``` Example — same schedule to a Rocket.Chat channel (requires token credentials — see [Rocket.Chat](/docs/messaging/rocketchat), an incoming webhook alone cannot target an explicit `--chat-id`): ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * 1-5" \ --tz Europe/London \ --provider rocketchat \ --chat-id "#alerts" ``` Optional `--project my-service` scopes the digest to one Sentry project. When an **uptime watch** schedule is running, the `sentry-summary` skill automatically includes an **Uptime / downtime (last 24h)** section with monitors that are still down or recovered in the window. If no uptime watch history exists, the digest stays Issues-only. The gateway daemon picks up scheduled digests automatically when it is running. If the LLM is unavailable, the run fails with an error. The count reflects issues seen in the last 7 days (capped at 100, shown as `100+` when it saturates). Use a search (e.g. `search_sentry_issues` during an investigation) to enumerate the full issue set over a custom window. ## Uptime watch (downtime notifications) Poll Sentry **uptime monitors** (Alerts / Uptime — not the Issues digest) and ping Slack, Telegram, or Rocket.Chat only when a monitor transitions to **down** or **recovered**. Quiet polls deliver nothing. This is separate from the morning Issues digest (`sentry-summary` / Feature #10). | Command | What it does | | -------------------------------------------- | ------------------------------------------ | | `opensre sentry uptime check` | List current uptime monitor health once | | `opensre sentry uptime watch add` | Schedule polling (default every 5 minutes) | | `opensre sentry uptime watch list` | List uptime watch schedules | | `opensre sentry uptime watch run TASK_ID` | Run a watch tick now (may notify) | | `opensre sentry uptime watch remove TASK_ID` | Remove a watch schedule | Example — poll every 5 minutes and ping Slack on transitions: ```bash theme={null} opensre sentry uptime watch add \ --cron "*/5 * * * *" \ --tz UTC \ --provider slack \ --chat-id C0123ABCD ``` Requires `alerts:read` (or equivalent) on the Sentry token. The agent tool `list_sentry_uptime_alerts` exposes the same monitor list in chat/investigation. Schedule an uptime watch alongside the morning digest so the `sentry-summary` skill can include yesterday's downtime summary via `get_sentry_uptime_digest`; the watch records DOWN/RECOVERED transitions locally for that rollup. Follow-ups (not in v1): inbound Sentry webhooks and auto-remediation attempts. ## Telemetry knobs | Variable | Default | Description | | --------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `OPENSRE_SENTRY_DSN` | — | Override the bundled Sentry DSN | | `OPENSRE_SENTRY_DISABLED` | `0` | Set to `1` to disable Sentry entirely | | `OPENSRE_SENTRY_LOGGING_DISABLED` | `0` | Set to `1` to disable automatic forwarding of `logger.error` and `logger.exception` calls to Sentry as events, without affecting `capture_exception` | ## Verify Error Reporting Send one test event to confirm OpenSRE can report runtime errors: ```bash theme={null} opensre debug sentry ``` For a custom or self-hosted project, set the DSN first: ```bash theme={null} OPENSRE_SENTRY_DSN=https://public-key@example.ingest.sentry.io/123 opensre debug sentry ``` ```text theme={null} Sentry DSN host: example.ingest.sentry.io Sentry event ID: Sentry flush sent: yes ``` The event is synthetic and tagged `debug=true`. If telemetry is disabled, the command exits non-zero without sending anything. ## Troubleshooting | Symptom | Fix | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **403 Forbidden** | Ensure the token has `event:read`; for uptime watch also `alerts:read` | | **Organization not found** | Verify `SENTRY_ORG_SLUG` matches the slug in your Sentry URL | | **Connection refused** | Check `SENTRY_URL` for self-hosted instances | | **No issues returned** | Normal if no issues exist in the time window — check `SENTRY_PROJECT_SLUG` | | **Fewer issues than the Sentry UI shows** | Widen `SENTRY_STATS_PERIOD` (e.g. `14d`) and confirm the search isn't scoped to the wrong `SENTRY_PROJECT_SLUG` | ## Security best practices * Use an **Organization Token** with only `event:read` — do not use admin tokens. * Store the token in `.env`, not in source code. * Rotate tokens periodically. # Sentry (MCP) Source: https://opensre.com/docs/sentry-mcp Connect Sentry's hosted MCP server so OpenSRE can query issues, events, traces, releases, and Seer root-cause analysis during investigations OpenSRE connects to Sentry's hosted [Model Context Protocol (MCP)](https://mcp.sentry.dev) server, exposing Sentry's products — issues, events, traces, replays, releases, monitors, and Seer AI root-cause analysis — as tools the agent can call while investigating an incident. This is distinct from the [Sentry issue integration](/docs/sentry), which is a narrow REST client used for issue and event lookup. Use the MCP integration when you want the agent to explore Sentry data directly, including Seer-assisted debugging. ## Tools | Tool | What it does | | ------------------- | ------------------------------------------------------------------------------------------------- | | `list_sentry_tools` | List the tools the connected Sentry MCP server exposes | | `call_sentry_tool` | Call a named Sentry MCP tool (e.g. look up an issue, fetch a trace, run Seer root-cause analysis) | The agent typically calls `list_sentry_tools` first to discover what is available for your organization, then `call_sentry_tool` with the chosen tool name and arguments. ## Prerequisites * A Sentry account (hosted SaaS, or self-hosted via the `SENTRY_MCP_HOST` override) * A Sentry **user auth token** created in your account settings. The required scopes depend on the skills you use: * `org:read` — read-only inspection of issues, events, traces, releases, monitors (default) * `event:write`, `team:write`, `project:write` — for triage and project-management skills ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Sentry (MCP)** when prompted, then paste your user auth token. The setup uses the hosted Streamable HTTP transport; keep the default URL unless you run self-hosted Sentry. To run a local server instead, set `SENTRY_MCP_MODE=stdio` via environment variables (see below). ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} SENTRY_MCP_MODE=streamable-http SENTRY_MCP_URL=https://mcp.sentry.dev/mcp SENTRY_MCP_AUTH_TOKEN=your_sentry_user_auth_token SENTRY_MCP_HOST= # optional, self-hosted Sentry host SENTRY_MCP_ORGANIZATION_SLUG= # optional, scope to one organization SENTRY_MCP_PROJECT_SLUG= # optional, scope to one project SENTRY_MCP_SKILLS= # optional, comma-separated skill filter ``` | Variable | Default | Description | | ------------------------------ | ---------------------------- | -------------------------------------------------------------------------------- | | `SENTRY_MCP_AUTH_TOKEN` | — | **Required** (hosted). Sentry user auth token | | `SENTRY_MCP_URL` | `https://mcp.sentry.dev/mcp` | MCP server URL | | `SENTRY_MCP_MODE` | `streamable-http` | Transport: `streamable-http`, `sse`, or `stdio` | | `SENTRY_MCP_HOST` | — | Self-hosted Sentry hostname (e.g. `sentry.example.com`) | | `SENTRY_MCP_ORGANIZATION_SLUG` | — | Scope tools to a specific organization | | `SENTRY_MCP_PROJECT_SLUG` | — | Scope tools to a specific project | | `SENTRY_MCP_SKILLS` | — | Comma-separated skill filter (`inspect`, `seer`, `triage`, `project-management`) | | `SENTRY_MCP_COMMAND` | — | Command to launch a local MCP server (`stdio` mode only) | | `SENTRY_MCP_ARGS` | — | Arguments for the local MCP command (`stdio` mode only) | To run a local Sentry MCP server instead of the hosted endpoint, use `stdio` mode: ```bash theme={null} SENTRY_MCP_MODE=stdio SENTRY_MCP_COMMAND=npx SENTRY_MCP_ARGS=@sentry/mcp-server@latest SENTRY_MCP_AUTH_TOKEN=your_sentry_user_auth_token ``` The `search_events` and `search_issues` tools translate natural-language queries and require an `OPENAI_API_KEY` in the MCP server's environment. The rest of the MCP server works without it, so you can skip this if you do not need those tools. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "sentry-mcp-prod", "service": "sentry_mcp", "status": "active", "credentials": { "url": "https://mcp.sentry.dev/mcp", "mode": "streamable-http", "auth_token": "your_sentry_user_auth_token", "organization_slug": "my-org" } } ] } ``` ## Verify ```bash theme={null} opensre integrations verify sentry_mcp ``` A successful check connects to the MCP server and reports how many tools it discovered. If it fails, the most common cause is a missing or invalid user auth token — confirm the token has at least `org:read` scope and that outbound HTTPS to `mcp.sentry.dev` is allowed. # ServiceNow Source: https://opensre.com/docs/servicenow Connect a ServiceNow instance so OpenSRE can check its configuration and credentials OpenSRE connects to ServiceNow so the CLI and the interactive shell can answer configuration questions about your instance — `opensre integrations verify servicenow` and shell questions like *"Is ServiceNow configured?"* return a real result instead of suggesting commands to run. ## Prerequisites * A ServiceNow instance URL (a free [developer instance](https://developer.servicenow.com) works) * A user with read access to the `sys_user` table (used for the onboarding connectivity check) ## Setup ### Option 1: Onboarding wizard ```bash theme={null} opensre onboard ``` Select **ServiceNow** under *Incident & Comms* and provide your instance URL, username, and password. The wizard validates the credentials with a one-row authenticated read of the `sys_user` table before saving. ### Option 2: Legacy CLI ```bash theme={null} opensre integrations setup servicenow ``` ### Option 3: Environment variables ```bash theme={null} export SERVICENOW_INSTANCE_URL=https://dev12345.service-now.com export SERVICENOW_USERNAME=admin export SERVICENOW_PASSWORD=your-password ``` `SERVICENOW_PASSWORD` is also resolved from the OS keyring when the environment variable is not set (the wizard stores it there). ### Option 4: Persistent store Add to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "servicenow-prod", "service": "servicenow", "status": "active", "credentials": { "instance_url": "https://dev12345.service-now.com", "username": "admin", "password": "your-password" } } ] } ``` | Field | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `instance_url` | **Required.** Your ServiceNow instance URL (e.g., `https://dev12345.service-now.com`). Must use `https://` — plain `http://` is accepted only for localhost/loopback targets. | | `username` | **Required.** ServiceNow username for HTTP Basic authentication | | `password` | **Required.** Password for the user | ## Verify ```bash theme={null} opensre integrations verify servicenow ``` Expected output on success: ``` SERVICE │ SOURCE │ STATUS │ DETAIL ━━━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ servicenow │ local env │ ✓ passed │ Configured for ServiceNow at │ │ │ https://dev12345.service-now.com. ``` This is a configuration-presence check (`instance_url`, `username`, and `password` are all set) rather than a live API call. The live connectivity check runs during onboarding wizard validation. You can also ask the interactive shell directly: ``` > Is ServiceNow configured? ``` The shell runs the same verifier and answers from its result. ## Troubleshooting | Symptom | Fix | | -------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Status `missing`** | Set `instance_url`, `username`, and `password` via any setup option above | | **401 Unauthorized during onboarding** | Check the username and password combination | | **403 Forbidden during onboarding** | The user authenticated but cannot read the `sys_user` table — grant a role with table read access (e.g. `itil`) | | **404 Not Found during onboarding** | Verify the instance URL (include `https://`, no path) | | **Developer instance unreachable** | Free developer instances hibernate after inactivity — wake it from the developer portal and retry | ## Security best practices * Use a **dedicated ServiceNow user** for OpenSRE with the minimum roles it needs (read access only). * Prefer the onboarding wizard: it keeps the password in the OS keyring instead of `.env`. * Rotate the password periodically and remove the integration when no longer needed. # Session History Source: https://opensre.com/docs/sessions Pick up any past conversation exactly where you left off — browse recent sessions, restore context, and start fresh. ## Commands at a glance | Command | What it does | | ---------------------- | ----------------------------------------------------------------------------- | | `/sessions` | List your recent REPL sessions with name, duration, and turn counts | | `/resume ` | Restore a past session's conversation so you can keep asking questions | | `/resume :` | Restore a specific branch point by entry ID prefix | | `/compact` | Summarize older context into a replayable compaction entry | | `/new` | Start a fresh session while carrying forward the current conversation context | *** ## Browsing past sessions — `/sessions` Run `/sessions` inside the interactive shell to see your recent history: ``` /sessions ``` ``` Recent sessions # Session ID Name Started Duration Turns ───────────────────────────────────────────────────────────────────────────────── 1 3f8a1c2d (current) 2024-01-15 10:00 42m 8 2 9b2e4f7a why is CPU spiking on prod-api 2024-01-14 14:30 1h 5m 15 3 c1d3e8f2 investigate OOM killer 2024-01-13 09:10 12m 3 ``` Up to 20 sessions are shown, newest first. The current session updates its elapsed time live. *** ## Resuming a past session — `/resume` When you want to continue a past investigation or conversation, use `/resume` with the first few characters of the session ID shown in `/sessions`: ``` /resume 9b2e4f7a ``` From outside the REPL, pass the same ID or name substring to the CLI launcher (for example `opensre --resume 9b2e4f7a` or `o --resume OOM` when `o` is your shell alias): ```bash theme={null} opensre --resume ``` In a TTY, bare `/resume` opens an interactive picker of recent sessions. New turns after a resume append to that same session file — `/resume` does not create a duplicate entry in `/sessions`. To resume a specific branch point, append an entry ID prefix after the session ID: ``` /resume 9b2e4f7a:abc123 ``` When the branch contains a compaction entry, replay inserts the saved summary before the kept messages so the assistant has the compacted context. OpenSRE restores the conversation so the assistant remembers what you were working on. It displays the past exchanges so you can see where you left off, then lets you continue typing: ``` resumed session 9b2e4f7a · why is CPU spiking on prod-api (14 messages) ─── conversation history ──────────────────────────────────── ❯ why is CPU spiking on prod-api? ● assistant The root cause is a connection pool leak in the orders service... ───────────────────────────────────────────────────────────── ``` **What gets restored:** | | Restored by `/resume` | | ------------------------------------------------------------------- | ---------------------------------------- | | Full conversation context (what you asked, what the assistant said) | ✅ Yes | | Infra context (service name, cluster, region learned mid-session) | ✅ Yes | | Trust mode, reasoning effort | ❌ No — these are per-session preferences | You can also search by name instead of ID: ``` /resume redis # resumes the session whose name contains "redis" /resume cpu spike # matches by name substring ``` If the prefix is ambiguous (matches more than one session), OpenSRE will ask you to be more specific. *** ## When you exit When you leave the REPL, OpenSRE prints your session ID so you can resume later: | Exit path | Behavior | | --------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `/exit` or `/quit` | Prints `Resume this session with:`, `/resume `, and `opensre --resume `, then goodbye | | **Ctrl+C** twice within 2 seconds | Same resume hint, then exits | | **Ctrl+D** (EOF) | Same resume hint when no dispatch is running | Example: ``` Resume this session with: /resume 3f8a1c2d-… opensre --resume 3f8a1c2d-… Goodbye! ``` Copy the `/resume …` or `opensre --resume …` line from the terminal, or find the session later with `/sessions`. *** ## Starting fresh with context — `/new` `/new` closes the current session and opens a new one, but carries the conversation forward so you don't lose context: ``` /new ``` ``` new session started — conversation context carried forward. 14 messages in context · type to continue ``` Use `/new` after a long session to keep things tidy in `/sessions` without losing your place in the conversation. *** ## Compacting long sessions — `/compact` `/compact` summarizes older conversation context, keeps the recent turns, and writes a durable compaction entry into the session file: ``` /compact ``` OpenSRE also compacts automatically before a shell turn when the replayed branch context grows past the runtime threshold. The compaction entry stores the summary, the first kept entry ID, and before/after size estimates so future `/resume` calls can replay the compacted branch. **`/new` vs `/clear`:** | Command | What it does | | -------- | --------------------------------------------------------------- | | `/clear` | Clears the terminal screen only — session and context unchanged | | `/new` | Opens a new session file; conversation context carried forward | *** ## What OpenSRE saves Each session records: * When it started and ended, and how long it ran * Every message you sent and every response from the assistant * Tool calls, tool updates, tool results, model/tool changes, labels, custom messages, and compaction summaries * Infra context discovered during investigations (service names, cluster names, regions) * Turn counts for investigations and chats Sessions are stored as files under `~/.opensre/sessions/`. Each file is plain text and human-readable. *** ## Privacy and what is NOT saved * **Secrets or credentials** — known token shapes are redacted from the prompt log by default (set `OPENSRE_PROMPT_LOG_REDACT=0` to store raw text instead) * **Full investigation results** — stored separately under `~/.opensre/investigations/` * **Token usage** — tracked in `~/.opensre/prompt_log.jsonl` when prompt logging is enabled See [Interactive Shell Privacy](/docs/interactive-shell-privacy) for redaction controls. *** ## Related * [Interactive Shell Privacy](/docs/interactive-shell-privacy) — command history, prompt/response logging, and redaction settings * [Prompt Logging](/docs/configuration/environment-variables) — full prompt/response logging configuration # Showcase Source: https://opensre.com/docs/showcase Real-world OpenSRE workflows, sample outputs, and community examples Concrete examples of what OpenSRE looks like in practice — from install through root-cause analysis. ## Quickstart walkthrough Install, onboard, and run your first investigation in under ten minutes. Step-by-step install, onboarding, and sample alert investigation. GIF walkthroughs for macOS, Linux, and Windows. Running an OpenSRE investigation on a sample alert through to a structured root-cause report Typical first-run commands: ```bash theme={null} curl -fsSL https://install.opensre.com | bash opensre onboard opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json ``` ## Sample investigation output A completed local run writes structured RCA artifacts you can open in any editor: | File | What it contains | | ------------------------ | ---------------------------------------------------------------- | | `problem.md` | Incident framing, impacted components, initial hypothesis | | `theory/hypothesis_*.md` | Each hypothesis tested during the run, with supporting evidence | | `report.md` | Final root-cause summary, confidence, and recommended next steps | Export a single JSON file for automation: ```bash theme={null} opensre investigate -i --output ./rca.json ``` See [Investigations overview](/docs/investigation-overview) for the full workflow, slash commands, and Slack delivery. Slack incident summary after an OpenSRE investigation ## Benchmark: CloudOpsBench Run OpenSRE against the 452-scenario CloudOpsBench Kubernetes RCA corpus and compare to published LLM-alone baselines. ```bash theme={null} uv run python -m tests.benchmarks._framework.cli run \ --config tests/benchmarks/cloudopsbench/config.yaml \ --models claude-sonnet-4-20250514 ``` See [CloudOpsBench benchmark](/docs/cloudopsbench) for configuration, integrity guards, and cost tracking. ## Share your work Have a demo, runbook, or incident post-mortem powered by OpenSRE? 1. Open a [GitHub Discussion](https://github.com/Tracer-Cloud/opensre/discussions) with the tag **Showcase** — include commands, integrations used, and (if possible) redacted output. 2. Join the [bi-weekly giveaway](/docs/bi-weekly-giveaway) for a chance to win OpenSRE swag when you share what you built. # Signoz Source: https://opensre.com/docs/signoz # SigNoz Integration Query logs, metrics, and traces from [SigNoz](https://signoz.io) via the Query Range API. OpenSRE uses `POST /api/v5/query_range` for `query_signoz_logs`, `query_signoz_metrics`, and `query_signoz_traces`. Configure **`SIGNOZ_URL`** and **`SIGNOZ_API_KEY`** (service account key). ## Quick Start If you already run SigNoz, skip to env setup below. If you want a local stack, use the [official SigNoz Docker setup](https://signoz.io/docs/install/docker/). ```bash theme={null} docker compose -f signoz/docker/docker-compose.yaml up -d # ... later docker compose -f signoz/docker/docker-compose.yaml down ``` Local Docker serves the UI and API on **[http://localhost:8080](http://localhost:8080)** (not 3301). ### 1. Create an API key In SigNoz: **Settings → Service Accounts** → create a service account → **Keys** → **Add Key**. Copy the key (shown once). ### 2. Configure environment variables ```bash theme={null} export SIGNOZ_URL="http://localhost:8080" export SIGNOZ_API_KEY="" ``` ### 3. Verify connectivity ```bash theme={null} uv run opensre integrations verify signoz ``` Expected output mentions the SigNoz Query API (`/api/v2/metrics`, `/api/v5/query_range`). ### 4. Use the tools When `alert_source` is `signoz`, the agent auto-seeds three tools before the ReAct loop: * **`query_signoz_logs`** — Search logs by service, severity, and time window. * **`query_signoz_metrics`** — Query CPU, memory, and request-rate signals. * **`query_signoz_traces`** — Query error spans, latency percentiles, and dependencies. ## Webhook Configuration SigNoz emits Prometheus-style webhook payloads. To trigger OpenSRE investigations automatically: 1. In SigNoz, go to **Settings → Notification Channels**. 2. Create a **Webhook** channel pointing at your OpenSRE instance: ``` POST https://your-opensre-instance/investigate ``` 3. The agent will detect `alert_source: signoz` from the payload and auto-query logs, metrics, and traces. ## CLI Setup ```bash theme={null} opensre integrations setup signoz ``` You will be prompted for SigNoz URL and API key only. ## Supported Metrics (V1) | Alias | Actual SigNoz Metric | | -------------- | --------------------- | | `cpu_usage` | `system_cpu_usage` | | `memory_usage` | `system_memory_usage` | | `request_rate` | `signoz_calls_total` | You can also pass any raw metric name known to SigNoz. For latency percentiles (p95/p99), prefer `query_signoz_traces`. ## API reference * Endpoint: `POST {SIGNOZ_URL}/api/v5/query_range` * Auth header: `SigNoz-Api-Key: ` * Validation probe: `GET {SIGNOZ_URL}/api/v2/metrics` ## Example Investigation ```bash theme={null} uv run opensre investigate --input-json '{ "alert_source": "signoz", "alert_name": "HighErrorRate", "pipeline_name": "payment-service", "severity": "critical", "commonLabels": { "service_name": "payment-service", "severity": "critical" }, "commonAnnotations": { "summary": "Error rate exceeded 5%" } }' ``` ## Troubleshooting | Symptom | Fix | | ------------------------------------ | ------------------------------------------------------------------------------------------------------ | | `SigNoz configuration is incomplete` | Set both `SIGNOZ_URL` and `SIGNOZ_API_KEY`. | | `HTTP 401` on verify | Regenerate the service account key; check URL (local Docker: port **8080**). | | `No logs returned` | Confirm telemetry exists in SigNoz and filter fields match (`service.name` for logs). | | `No metrics returned` | Verify the metric name in SigNoz Metrics Explorer; empty results return a warning for unknown metrics. | # SMTP Source: https://opensre.com/docs/smtp Configure SMTP email delivery for background RCA completion notifications. OpenSRE can send **background RCA completion emails** through any existing SMTP relay you already use. You do **not** need to run your own mail server. Typical providers: * Google Workspace / Gmail SMTP * Microsoft 365 SMTP * AWS SES SMTP * SendGrid / Postmark / Mailgun SMTP * local dev SMTP sinks like Mailpit or MailHog *** ## What this powers In v1, the `smtp` integration is used for: * `/background` RCA completion notifications Email is one of two completion channels — [Telegram](/docs/messaging/telegram) is the other, and you can enable both at once. See [Background investigations](/docs/background-investigations). The email summary includes: * **Root cause** * **Top analysis** * **What to do next** * a small internal stats block *** ## Step 1: Configure the integration ### Via CLI wizard ```bash theme={null} opensre integrations setup smtp ``` You'll be prompted for: * SMTP host * SMTP port * security mode: `starttls`, `ssl`, or `none` * optional username/password * sender address * optional default recipient ### Via environment variables ```bash theme={null} SMTP_HOST=smtp.example.com SMTP_PORT=587 SMTP_SECURITY=starttls SMTP_USERNAME=mailer SMTP_PASSWORD=secret SMTP_FROM_ADDRESS=opensre@example.com SMTP_DEFAULT_TO=team@example.com ``` If authentication is required, set **both** `SMTP_USERNAME` and `SMTP_PASSWORD`. *** ## Step 2: Verify ```bash theme={null} opensre integrations verify smtp ``` A passing verification confirms OpenSRE can: * connect to the SMTP server * negotiate TLS when configured * authenticate when credentials are provided Sample success output: ```text theme={null} smtp: passed — Connected to SMTP server successfully. ``` *** ## Step 3: Use it with background investigations Enable background mode in the interactive shell **and select the `email` channel**. Notification channels are empty by default, so `/background on` on its own sends nothing: ```text theme={null} /background on /background notify set email /background notify list # -> email ``` Then start an investigation as usual. When it completes, OpenSRE sends the RCA summary email to the configured default recipient. To send the same summary to Telegram as well: ```text theme={null} /background notify set email,telegram ``` You can inspect the completed job in the shell with: ```text theme={null} /background list /background show ``` *** ## Troubleshooting | Detail | Likely cause | | ---------------------------------------------- | -------------------------------------------------------------- | | `Missing recipient email address` | No `SMTP_DEFAULT_TO` is configured yet. | | `username and password must both be set` | Only one auth field was provided. | | `from_address must look like an email address` | Sender address is malformed. | | `SMTP connection failed` | Host/port/TLS settings are wrong, or the relay is unreachable. | For local testing, Mailpit is a good fake SMTP target: ```bash theme={null} docker run -p 1025:1025 -p 8025:8025 axllent/mailpit ``` Then configure: ```bash theme={null} SMTP_HOST=localhost SMTP_PORT=1025 SMTP_SECURITY=none SMTP_FROM_ADDRESS=opensre@example.com SMTP_DEFAULT_TO=team@example.com ``` # Snowflake Source: https://opensre.com/docs/snowflake Connect Snowflake so OpenSRE can query data warehouse and investigate analytics issues Analytics pipelines can fail in subtle ways. OpenSRE connects to your Snowflake warehouse to help investigate issues by querying warehouse metadata, reviewing query history, checking warehouse health, and identifying performance bottlenecks. ## Before you start You'll need: * A Snowflake account (trial or production) * A configured warehouse * Snowflake credentials with appropriate permissions * A database and schema to query * Network access from your OpenSRE environment to Snowflake ## Connecting to Snowflake ### The guided way For a step-by-step walkthrough: ```bash theme={null} opensre integrations setup ``` Choose **Snowflake** and follow the prompts. ### The direct way: Environment variables Or add these to your `.env`: ```bash theme={null} SNOWFLAKE_ACCOUNT_IDENTIFIER=xy12345.us-east-1 SNOWFLAKE_USER=opensre_user SNOWFLAKE_TOKEN=your_programmatic_access_token SNOWFLAKE_PASSWORD=your_password SNOWFLAKE_WAREHOUSE=COMPUTE_WH SNOWFLAKE_DATABASE=analytics SNOWFLAKE_SCHEMA=public SNOWFLAKE_ROLE=viewer ``` | Variable | Default | Description | | ------------------------------ | ------- | -------------------------------------------------------------- | | `SNOWFLAKE_ACCOUNT_IDENTIFIER` | — | **Required.** Snowflake account identifier | | `SNOWFLAKE_ACCOUNT` | — | Alternative Snowflake account variable | | `SNOWFLAKE_USER` | — | Snowflake username | | `SNOWFLAKE_TOKEN` | — | **Required.** Programmatic access token for API authentication | | `SNOWFLAKE_PASSWORD` | — | Optional password (used alongside token where configured) | | `SNOWFLAKE_WAREHOUSE` | — | Warehouse name (for example, `COMPUTE_WH`) | | `SNOWFLAKE_DATABASE` | — | Default database | | `SNOWFLAKE_SCHEMA` | — | Default schema | | `SNOWFLAKE_ROLE` | — | Role with appropriate permissions | OpenSRE requires `SNOWFLAKE_ACCOUNT_IDENTIFIER` and `SNOWFLAKE_TOKEN` to activate the integration. Generate a programmatic access token in Snowflake under your user's security settings. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "snowflake-prod", "service": "snowflake", "status": "active", "credentials": { "account": "xy12345.us-east-1", "user": "opensre_user", "token": "your_programmatic_access_token", "password": "your_password", "warehouse": "COMPUTE_WH", "database": "analytics", "schema": "public", "role": "viewer" } } ] } ``` ## Finding your account identifier Snowflake account identifiers can vary depending on your region and organization setup. 1. Open the Snowflake web interface 2. Open your account or profile settings 3. Locate your account identifier 4. Use the full value for `SNOWFLAKE_ACCOUNT_IDENTIFIER` Examples: ```text theme={null} xy12345 xy12345.us-east-1 myorg-myaccount myorg-myaccount.eu-west-1 ``` Use the full account identifier shown by Snowflake. Do not remove region or organization information unless your Snowflake deployment documentation explicitly instructs you to do so. ## Best practice: Create a dedicated role for OpenSRE Instead of using an administrative account, create a role with only the permissions OpenSRE requires. ```sql theme={null} CREATE ROLE opensre_viewer; GRANT SELECT ON ALL TABLES IN DATABASE analytics TO ROLE opensre_viewer; GRANT MONITOR ON WAREHOUSE COMPUTE_WH TO ROLE opensre_viewer; CREATE USER opensre_user PASSWORD = 'strong_secure_password'; GRANT ROLE opensre_viewer TO USER opensre_user; ``` Then use those credentials in your OpenSRE configuration. ## Investigation tools When OpenSRE investigates a Snowflake-related alert, this tool is available: ### Query history Runs bounded, read-only queries against Snowflake query history to surface recent failed queries, long-running statements, and warehouse usage patterns during an incident. ## Security recommendations * Use a dedicated Snowflake user for OpenSRE * Grant only the permissions required for investigations * Rotate credentials regularly * Monitor access through Snowflake audit logs ## Test the connection Let's verify everything is working: ```bash theme={null} opensre integrations verify snowflake ``` Expected output: ``` Service: snowflake Status: passed Detail: Configured for Snowflake account xy12345.us-east-1 ``` ## Troubleshooting | Symptom | Fix | | ------------------------------ | ------------------------------------------------------------------------------------------ | | **Missing token credentials** | Set `SNOWFLAKE_TOKEN`. Password alone does not activate the integration. | | **Invalid account identifier** | Use the full identifier from Snowflake (including region/org suffix if shown). | | **Warehouse suspended** | Resume the warehouse in Snowflake or grant `OPERATE` on the warehouse to the OpenSRE role. | | **Insufficient privileges** | Grant `MONITOR` on the warehouse and `SELECT` on `ACCOUNT_USAGE.QUERY_HISTORY`. | # Splunk Source: https://opensre.com/docs/splunk Connect Splunk so OpenSRE can search logs using SPL during investigations OpenSRE queries Splunk using the REST API to surface relevant log evidence during alert investigations — searching indexes with SPL, correlating error patterns with incidents, and identifying root causes. ## Prerequisites * Splunk Enterprise or Splunk Cloud instance (version 8.x or later) * REST API access on port 8089 * A bearer token with search capability (see [Generating a Bearer Token](#generating-a-bearer-token)) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Splunk** when prompted and provide your REST API base URL and bearer token. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} SPLUNK_URL=https://splunk.corp.com:8089 # REST API base URL (port 8089 default) SPLUNK_TOKEN=your-bearer-token # API bearer token (NOT an HEC token) SPLUNK_INDEX=main # Default index to search (optional) SPLUNK_VERIFY_SSL=true # Set false to skip SSL verification (optional) SPLUNK_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem # Path to custom CA bundle (optional) ``` | Variable | Default | Description | | ------------------- | ------- | --------------------------------------------------------------- | | `SPLUNK_URL` | — | **Required.** REST API base URL including port | | `SPLUNK_TOKEN` | — | **Required.** Bearer token with search capability | | `SPLUNK_INDEX` | `main` | Default index searched when no index is specified in the alert | | `SPLUNK_VERIFY_SSL` | `true` | Set to `false` to disable SSL verification (dev/local only) | | `SPLUNK_CA_BUNDLE` | — | Path to a PEM CA bundle for enterprise self-signed certificates | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "splunk-prod", "service": "splunk", "status": "active", "credentials": { "base_url": "https://splunk.corp.com:8089", "token": "your-bearer-token", "index": "main", "verify_ssl": true, "ca_bundle": "/etc/ssl/certs/corp-ca.pem" } } ] } ``` ### Multi-instance setup To connect multiple Splunk instances (e.g. separate prod and staging clusters): ```bash theme={null} SPLUNK_INSTANCES='[ {"name":"prod","tags":{"env":"prod"},"credentials":{"base_url":"https://splunk-prod:8089","token":"prod-token","index":"prod"}}, {"name":"staging","tags":{"env":"staging"},"credentials":{"base_url":"https://splunk-staging:8089","token":"staging-token","index":"staging"}} ]' ``` When `SPLUNK_INSTANCES` is set it overrides the single-instance `SPLUNK_URL` / `SPLUNK_TOKEN` variables. ## Generating a bearer token OpenSRE uses bearer tokens — not basic auth and not HEC tokens. To generate one: **Via the Splunk UI:** 1. Go to **Settings** → **Tokens** 2. Click **New Token** 3. Set a name (e.g. `opensre`) and an expiry date 4. Copy the generated token **Via the REST API** (replace `` with your admin password): ```bash theme={null} curl -sk -u admin: \ https://splunk.corp.com:8089/services/authorization/tokens \ -X POST \ --data-urlencode "name=opensre" \ --data-urlencode "expires_on=+90d" \ --data-urlencode "output_mode=json" \ | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['entry'][0]['content']['token'])" ``` The token needs the `search` capability. The `admin` role includes this by default. For a dedicated service account, ensure the role includes: * `search` * `read_splunkd_private_settings` (needed for the verify call against `/services/server/info`) ## Verify ```bash theme={null} opensre integrations verify splunk ``` Expected output: ``` Service: splunk Status: passed Detail: Connected to Splunk 9.x.x ``` ## How queries are generated OpenSRE builds SPL queries deterministically from the alert payload — the LLM selects which tool to call but never writes the query itself. This keeps investigations reproducible and auditable. Query construction priority: | Priority | Source | Example | | -------- | ---------------------------------------------------------- | ----------------------------------------------------- | | 1 | `annotations.splunk_query` — verbatim SPL from your alert | `index=prod "PaymentTimeout" \| head 50` | | 2 | `annotations.query` or `annotations.log_query` | Any pre-populated query field | | 3 | `error_message` field — keyword search built automatically | `search index=main "NullPointerException" \| head 50` | | 4 | `alert_name` — last-resort keyword search | `search index=main "payments-error-spike" \| head 50` | | 5 | Fallback — index scan | `search index=main \| head 50` | To pass a specific SPL query through an alert, set `commonAnnotations.splunk_query`: ```json theme={null} { "alert_name": "Payment service errors", "commonAnnotations": { "splunk_query": "index=prod sourcetype=app_logs \"NullPointerException\" | head 50" } } ``` ## Test with the built-in alert template ```bash theme={null} opensre investigate --template splunk ``` This runs a synthetic investigation using a pre-built alert fixture — no live alert infrastructure needed. ## Troubleshooting | Symptom | Fix | | ----------------------------------- | ----------------------------------------------------------------------------------------------- | | `SSL: CERTIFICATE_VERIFY_FAILED` | Set `SPLUNK_CA_BUNDLE=/path/to/corp-ca.pem` (preferred) or `SPLUNK_VERIFY_SSL=false` (dev only) | | `HTTP 401 Unauthorized` | Token expired or was generated with the wrong account — regenerate | | `HTTP 403 Forbidden` | Token lacks `search` capability — check the role assigned to the token | | Empty search results | Data may not have been ingested yet, or the index name is wrong | | `Connection refused` on port 8089 | Splunk management port may be firewalled; confirm network access | | `opensre integrations verify` fails | Check `SPLUNK_URL` includes the protocol and port (`https://host:8089`) | ## Security best practices * Use a **read-only bearer token** — never use an admin token in production. * Store `SPLUNK_TOKEN` in `.env` or the credential store, not in source code or CI logs. * Prefer a dedicated `opensre` service account with only the `search` capability. * For enterprise self-signed certificates, set `SPLUNK_CA_BUNDLE` to the CA bundle path rather than disabling verification entirely. * Set `SPLUNK_VERIFY_SSL=false` only in local or dev environments when you cannot supply a CA bundle. * Rotate tokens on a schedule and revoke them when no longer needed. # AWS SQS Source: https://opensre.com/docs/sqs_queues Inspect SQS queue state — backlog depth, stuck consumers, and dead-letter wiring — during incidents OpenSRE uses AWS SQS queue attributes to answer the first question of every queue-backed incident: **"what state is this queue actually in?"** When a queue alert fires, the planner can read message depth, in-flight count, visibility timeout, and dead-letter queue wiring — the signals that distinguish a normal backlog from consumers that are stuck. Queue inspection is read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot send, consume, or delete messages. ## Why this exists Queue state lives in queue *attributes*, not in logs or metrics. A consumer that hangs without raising an exception writes no error line anywhere — so log search comes back clean while the queue quietly stops draining. The specific failure this surfaces: a message that causes a consumer to hang past its `VisibilityTimeout` becomes visible again and is redelivered to the next consumer, which also hangs. Every consumer ends up holding the same message. Nothing errors, so nothing reaches the logs, and without a `RedrivePolicy` there is no receive-count ceiling to break the cycle. Reading two attributes — `in_flight_count` pinned at the consumer count and `has_dlq: false` — identifies it immediately. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/aws) (role ARN recommended) — SQS reuses the same account credentials and region, so no extra setup is needed * IAM permissions for the two read-only SQS actions listed below ## How it works SQS inspection is account-wide, so the tool becomes available to the planner whenever the [AWS integration](/docs/aws) is configured — there is nothing queue-specific to set up. The region comes from the AWS integration (or `AWS_REGION`, defaulting to `us-east-1`). The tool calls `ListQueues` to discover queues, then `GetQueueAttributes` for each one, so a prefix filter and a queue cap keep the fan-out bounded. ## Tools | Tool | AWS API calls | What it returns | | -------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------ | | `get_sqs_queue_attributes` | `sqs:ListQueues`, `sqs:GetQueueAttributes` | Per-queue state — visible depth, in-flight count, visibility timeout, DLQ wiring, and FIFO flag. | ### Parameters | Parameter | Default | Description | | ------------------- | ----------- | ------------------------------------------------------------------------------------------------------- | | `queue_name_prefix` | — | Only inspect queues whose name starts with this prefix (e.g. `payments-`). Omit to inspect every queue. | | `max_queues` | `20` | Maximum queues to inspect (1–100). Bounds the per-queue attribute fan-out. | | `region` | `us-east-1` | AWS region to query. | ### Output fields | Field | Meaning | | ---------------------------- | --------------------------------------------------------------------------------- | | `visible_count` | Messages available for delivery (`ApproximateNumberOfMessages`). | | `in_flight_count` | Messages delivered but not yet deleted (`ApproximateNumberOfMessagesNotVisible`). | | `visibility_timeout_seconds` | How long a message stays hidden after delivery before redelivery. | | `has_dlq` / `redrive_policy` | Whether a dead-letter queue is configured, and its target plus `maxReceiveCount`. | | `is_fifo` | Whether the queue is FIFO. | ### Reading the output | Signal | Normal backlog | Stuck consumers | | ----------------- | ------------------------------------ | --------------------------------------------------- | | `visible_count` | High — producers outpacing consumers | Near zero | | `in_flight_count` | Low | Pinned at the consumer/pod count | | `has_dlq` | Usually `true` | Often `false` — nothing breaks the redelivery cycle | A **missing** numeric attribute is reported as `null`, not `0`. An absent measurement and a genuinely empty queue are different findings, and collapsing them would let a metrics gap read as "the queue is drained". Attributes prefixed `Approximate` are eventually consistent and can lag by around a minute. The field names keep the `approximate` semantics in mind — use them for shape and direction, not for exact reconciliation. If one queue's attributes cannot be read (for example a per-queue policy denying `GetQueueAttributes`), that queue is returned with an `attributes_error` field and the remaining queues are still inspected — a single unreadable queue never sinks the whole investigation. ## IAM permissions ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:ListQueues", "sqs:GetQueueAttributes" ], "Resource": "*" } ] } ``` Attach this policy to the same IAM role or user already configured for the [AWS integration](/docs/aws). If you are already using the AWS managed `ReadOnlyAccess` policy, both actions are already covered. **Execution identity:** the AWS integration's `role_arn` / credentials gate *availability* and supply the region, but the calls themselves run through boto3's standard credential chain (environment variables, shared config, or the host's instance role) — the configured role is **not** assumed for the call. Ensure the identity the OpenSRE process runs as can perform these actions. This matches the other AWS tools (RDS/EKS/CloudTrail). ## Troubleshooting | Symptom | Fix | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | | **AccessDenied on `sqs:ListQueues`** | Add the IAM policy above to the role or user used by the AWS integration. | | **No queues returned** | Check the prefix filter and confirm you are querying the region the queues live in. `ListQueues` is region-scoped. | | **A queue shows `attributes_error`** | That queue's resource policy likely denies `sqs:GetQueueAttributes`. Other queues are unaffected. | | **Fewer queues than expected** | The `max_queues` cap was hit; the response sets `truncated: true`. Raise `max_queues` (up to 100) or narrow with `queue_name_prefix`. | | **Tool reports the wrong region** | Set `AWS_REGION`, or check the `region` field on the configured AWS integration. | # Supabase Source: https://opensre.com/docs/supabase Connect Supabase so OpenSRE can diagnose database, API, and authentication issues during investigations Supabase powers many modern applications. When something goes wrong with your database, APIs, or authentication flows, OpenSRE can use your Supabase project configuration to investigate issues, inspect resources, and gather operational insights. ## What you'll need * A Supabase project * Your Supabase Project URL * A Supabase Service Role Key * Permissions to access project resources ## Connecting Supabase ### Guided walkthrough Let's set this up together: ```bash theme={null} opensre integrations setup ``` Pick **Supabase** and enter your project credentials when prompted. ### DIY: Using environment variables Or add these directly to your `.env`: ```bash theme={null} SUPABASE_URL=https://your-project.supabase.co SUPABASE_SERVICE_KEY=your_service_role_key ``` | Variable | Description | | ---------------------- | ------------------------- | | `SUPABASE_URL` | Supabase project URL | | `SUPABASE_SERVICE_KEY` | Supabase service role key | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "supabase-prod", "service": "supabase", "status": "active", "credentials": { "url": "https://your-project.supabase.co", "service_key": "your_service_role_key" } } ] } ``` ## Finding your Supabase credentials 1. Log in to Supabase 2. Open your project 3. Navigate to **Settings** → **API** 4. Copy the **Project URL** 5. Copy the **Service Role Key** 6. Add these values to your OpenSRE configuration The Service Role Key has elevated privileges. Store it securely and never expose it in client-side applications or public repositories. ## Investigation tools When OpenSRE investigates a Supabase-related alert, these tools are available: ### Service health Checks the health of PostgREST, Auth, Storage, and other Supabase services for the project. Useful when triaging 503 or 401 errors from a Supabase-backed application. ### Storage buckets Lists all Storage buckets and their configuration metadata — public vs private access, file size limits, and allowed MIME types. ## Verify it works Let's make sure everything is connected: ```bash theme={null} opensre integrations verify supabase ``` Expected output: ``` Service: supabase Status: passed Detail: Connected to Supabase project ... ``` ## Troubleshooting | Symptom | Fix | | --------------------------------------- | -------------------------------------------------------------------------------------- | | **401 Unauthorized** | Confirm the **Service Role Key** (not the anon key) is set in `SUPABASE_SERVICE_KEY`. | | **Invalid project URL** | Use the Project URL from Settings → API (`https://your-project.supabase.co`). | | **Health check shows degraded service** | Check the Supabase status page and project dashboard for the affected service. | | **Storage bucket not found** | Verify the bucket name in the investigation matches an existing bucket in the project. | ## Security best practices * Use the **service role key** only on the server side — never in client apps or public repos. * Rotate the service role key if it is ever exposed. * Prefer a dedicated Supabase project or restricted key for investigation workloads when possible. # Capabilities and compatibility Source: https://opensre.com/docs/technology/capabilities-compatibility What Tracer supports and guarantees Tracer is an execution-level observability platform designed for scientific and compute-intensive workloads. This page describes the capabilities, compatibility, and guarantees that apply across all Tracer products. For details on how execution data is collected, analyzed, or acted on, see:
Tracer/collect
Tracer/tune
Tracer/sweep
## Execution-level observability Tracer observes workloads at the operating system level. It captures how processes execute and use resources, rather than relying on workflow metadata, logs, or application instrumentation. Observes real execution behavior, not configuration intent Works without modifying code, containers, or workflows Does not require tagging or framework-specific integration This execution-first approach is consistent whether Tracer is used for pipeline analysis, optimization, or systemwide cost discovery. ## Supported environments Tracer is designed to work in real production environments with heterogeneous tooling. ### Operating system * Linux hosts only ### Infrastructure * Cloud compute (for example, AWS EC2 and AWS Batch) * On-premises Linux clusters * Hybrid cloud and on-prem environments ### Workload types * Containerized and non-containerized workloads * Batch processing systems * Interactive compute sessions * Pipelines with many short-lived processes * Legacy tools and custom binaries If a workload runs on Linux, Tracer can observe it. ## Framework and workflow support Tracer is framework-agnostic. It does not depend on workflow engine metadata and does not require configuration for specific systems. Commonly used with: * Nextflow * Snakemake * CWL * Slurm * Custom scripts and pipelines * AWS Batch–managed workloads This list is illustrative, not exhaustive. ## Installation guarantees Across all supported environments: Works with existing code as-is No application wrappers required No annotation or tagging required Once Tracer is installed, execution visibility is available automatically on the next run. See [Quickstart](/docs/quickstart) for setup instructions. ## Security and data boundaries Tracer is designed to minimize data exposure. **Across all products, Tracer:** * Collects system-level execution metadata only * Does not inspect application payloads or scientific data * Does not read file contents or memory * Does not collect environment variables or secrets Detailed data-collection limits and privacy boundaries are documented in [Limits and privacy](/docs/technology/limits-privacy). Information about kernel-level safety and isolation is covered in [eBPF and security](/docs/technology/ebpf-security). ## Performance characteristics Tracer is built for continuous use in compute-heavy cloud environments. (Also works locally) General characteristics: * **Low overhead** — Less than 2% runtime overhead * **No re-runs** — No need to re-run pipelines to collect data * **Scales well** — Handles thousands of short-lived processes Specific performance details depend on workload characteristics and are documented in [Tracer/collect](/docs/technology/tracer-collect). ## What this page is for This page answers: * Will Tracer work in my environment? * What are the system requirements and guarantees? * What assumptions does Tracer make about workflows and infrastructure? It intentionally does not describe product-specific behavior or UI features. ## Summary Tracer provides execution-level observability across diverse scientific and compute environments. Its capabilities are defined by what it supports, what it guarantees, and what it deliberately does not require, independent of any specific workflow engine or cloud provider. Tracer/collect
Learn how execution signals are captured at the kernel level Tracer/tune
Analyze and optimize pipeline performance and resource usage Tracer/sweep
Uncover systemwide cloud waste using real execution behavior
# Data model Source: https://opensre.com/docs/technology/data-model How Tracer structures execution data Tracer organizes execution data into a small set of core entities that reflect how real workloads run: runs, tasks, tools, containers, and hosts. This data model allows Tracer to map low-level execution signals to the way teams reason about pipelines and infrastructure, without relying on workflow metadata, logs, or application instrumentation. This page describes those entities and how they relate to each other. ## Overview At a high level: * Tracer/collect observes execution events at the operating system level * These events are correlated into structured entities * Higher-level products (Tracer/tune and Tracer/sweep) operate on this shared model Works with any orchestrator or scheduler Consistent across environments Represents complex, multi-process execution ## Core entities ### Runs A run represents a single execution of a pipeline or workload. A run typically corresponds to: * A workflow execution (for example, a Nextflow or Snakemake run) * A batch job or experiment * A repeated invocation of the same pipeline configuration Runs provide the top-level boundary for grouping execution data and comparing behavior across executions. ### Tasks A task represents a logical unit of work within a run. Tasks often correspond to: * Workflow steps or processes * Batch jobs or array jobs * Scheduled units of execution A task may: * Run on one or multiple hosts * Execute sequentially or in parallel * Spawn multiple tools and subprocesses Tasks are the primary unit used for performance comparison and tuning. ### Tools A tool represents an executable program invoked during a task. Examples include: * Native binaries (for example, bwa, samtools) * Interpreters and scripts (python, bash) * JVM-based tools * Short-lived helper binaries and child processes Tracer identifies tools based on observed process execution, not logs or configuration. Even tools that produce no logs are captured as first-class entities. ### Containers A container represents an execution context defined by container runtimes or Linux namespaces. Containers: * Group related processes * Provide isolation boundaries * May contain multiple tools and subprocesses Tracer does not require containers to be present, but when they are used, container context is preserved and reflected in the data model. ### Hosts A host represents a physical or virtual machine where execution occurs. Hosts include: * Cloud instances (for example, EC2) * On-premises nodes * Batch or HPC worker nodes Host-level data provides the infrastructure context needed to understand scheduling behavior, resource contention, and idle time. ## Relationships between entities The entities form a hierarchy: * A run contains one or more tasks * A task invokes one or more tools * Tools execute within a container or directly on a host * All execution ultimately occurs on a host This structure allows Tracer to: * Attribute resource usage accurately * Compare behavior across runs and tasks * Correlate infrastructure behavior with pipeline execution ## How correlation works Tracer correlates execution events using identifiers exposed by the operating system, including: * Process IDs and parent–child relationships * Cgroups and namespaces * Container runtime metadata (when available) This correlation happens automatically and does not require: * Workflow engine integration * Application instrumentation * Explicit tagging The result is a consistent execution model across heterogeneous environments. ## What the data model enables This data model is the foundation for Tracer's higher-level capabilities. It enables: * Execution timelines organized by run, task, and tool * Resource usage attribution at meaningful boundaries * Detection of idle execution and contention * Cost attribution aligned with real execution behavior * Cross-run comparison and regression detection Tracer/tune and Tracer/sweep operate on this shared structure rather than raw telemetry. ## What the data model does not represent **The data model intentionally excludes:** * Application payloads or scientific input/output data * Source code, function calls, or language-level execution traces * Domain-specific semantics or correctness Tracer models how workloads execute, not what they compute. ## Orchestrator terminology mapping (reference) Tracer's data model is framework- and language-agnostic. The table below shows how Tracer entities typically align with common orchestrator concepts. Exact mappings may vary by workflow engine and configuration. Workflow run, DAG run, execution Process, step, task, op, node Binary, script, container entrypoint Pod, container, namespace Worker node, instance, executor host | Tracer concept | Common equivalents | | -------------- | ------------------------------------ | | Run | Workflow run, DAG run, execution | | Task | Process, step, task, op, node | | Tool | Binary, script, container entrypoint | | Container | Pod, container, namespace | | Host | Worker node, instance, executor host | This mapping is provided for orientation only. Tracer does not depend on orchestrator metadata to build its execution model. ## When to read this page This page is most useful if you: * Want to understand how Tracer structures execution data * Are integrating Tracer data into external systems * Need clarity on attribution boundaries and terminology * Are evaluating Tracer for complex or regulated environments Tracer/collect
Execution capture details Tracer/tune
Optimization and analysis Tracer/sweep
Cloud waste discovery
# eBPF and security Source: https://opensre.com/docs/technology/ebpf-security How Tracer observes execution safely and securely Tracer uses eBPF (extended Berkeley Packet Filter) to observe execution behavior directly from the Linux kernel. This approach enables high-fidelity visibility with minimal overhead, while maintaining strong safety and security guarantees. This page explains why eBPF is used, what Tracer observes, what it does not observe, and how security is enforced. ## Why Tracer uses eBPF Traditional observability approaches depend on logs, metrics exporters, or application-level instrumentation. These methods rely on what software chooses to expose and often miss behavior in short-lived processes, external binaries, or between metric collection intervals. eBPF allows Tracer to observe execution where it actually happens, at the boundary between user processes and the operating system. Using eBPF, Tracer can: * Observe syscalls, scheduling events, and I/O activity * Track process lifecycles across containers and hosts * Capture execution behavior without modifying application code * Work across languages, frameworks, and binaries This makes eBPF well suited for scientific and ML workloads that rely on many heterogeneous tools. How Tracer works: observing execution at the kernel level using eBPF ## Safety model eBPF programs are subject to strict safety constraints enforced by the Linux kernel. Tracer's use of eBPF follows these principles: All eBPF programs are validated by the kernel verifier before execution. Programs that fail verification cannot run. eBPF programs run in a restricted environment and cannot access arbitrary memory or crash the kernel. Tracer does not load kernel modules or modify kernel code. Tracer attaches only to well-defined kernel hooks such as system call boundaries and scheduler events. These guarantees are provided by the kernel itself and apply regardless of how Tracer is configured. ## What Tracer observes Tracer observes execution metadata, not application data. CPU usage and scheduling behavior Memory usage and peak memory Disk and network I/O activity Process start, stop, and parent–child relationships Container, namespace, and cgroup context This data is used to reconstruct execution timelines and resource usage patterns. ## What Tracer does not observe **Tracer explicitly does not collect:** * Application payloads or scientific input/output data * File contents or data values * Source code or function-level execution traces * Environment variables or secrets * Application- or domain-level interpretation of what a command does While Tracer can observe which binaries and commands were executed, it does not inspect the data those commands operate on or infer application semantics. ## Data handling and isolation Tracer separates data collection from analysis. * Execution signals are captured locally and processed into structured telemetry * Only metadata required for analysis is transmitted * Payload data is never inspected or exported * Correlation is based on operating system identifiers (PIDs, cgroups, namespaces) This design minimizes data exposure while preserving execution insight. ## Performance considerations Tracer's eBPF-based collection is designed to minimize overhead: eBPF probes execute in kernel space with low latency Event filtering reduces data volume at the source Collection overhead remains low even with many short-lived processes No pipeline re-runs are required to obtain telemetry Measured overhead depends on workload characteristics but is typically low enough for continuous use in production environments. ## Security boundaries Tracer is intentionally scoped. **It does not:** * Modify application behavior * Control execution or scheduling * Start, stop, or change resources * Replace IAM, RBAC, or cloud security controls Security ownership remains with the operating system, container runtime, and cloud provider. Tracer observes execution behavior within those boundaries. ## When to read this page This page is most relevant if you: * Need to understand how Tracer observes execution safely * Operate in regulated or security-sensitive environments * Evaluate eBPF-based tooling for production systems * Want clarity on what data Tracer does and does not collect Tracer/collect
Implementation details Tracer/tune
Execution analysis and optimization Tracer/sweep
Cloud waste discovery
# How Tracer works, end to end Source: https://opensre.com/docs/technology/end-to-end A shared execution signal for analysis and optimization Tracer is built to make execution behavior visible in compute-intensive environments, without changing workloads or relying on what applications choose to report. At a high level, Tracer works in three layers: * **Tracer/collect**: an open-source eBPF agent that gathers execution signals from the host-layer * **Tracer/datalake**: a shared execution view across pipelines and environments * **Tracer/tune and Tracer/sweep**: use that signal to solve different problems Tracer simplified architecture: collect, datalake, tune and sweep This page explains the architecture. Each product page goes deeper on its specific behavior. ## What Tracer is made of Tracer consists of three components with distinct responsibilities: * **Tracer/collect** gathers execution signals directly from the operating system * **Tracer/tune** uses those signals to analyze and optimize pipeline performance * **Tracer/sweep** uses the same signals to uncover systemwide cloud waste Tracer/collect is the foundation. Tracer/tune and Tracer/sweep are built on top of the execution signal it produces. How Tracer works end to end: from kernel-level signals to analysis and optimization ## Architecture at a glance Tracer's data flow can be understood in four stages: Tracer/collect attaches non-intrusively to running processes and containers on a Linux host using eBPF, a Linux kernel technology for safe, low-overhead instrumentation. No code changes, container restarts, or application modifications are required. Execution events are captured at the kernel boundary, including CPU scheduling, memory activity, disk and network I/O, and process lifecycle events. Only relevant signals are selected through intelligent filtering rules. Low-level events are mapped to higher-level execution context such as containers, tools, tasks, runs, and pipelines. This mapping uses kernel-native identifiers like PIDs, namespaces, and cgroups. Structured telemetry is batched and sent securely to Tracer's backend, where it becomes available for analysis, visualization, and downstream products. Data is buffered locally and retried until successfully delivered. This pipeline is continuous and designed to operate safely in production cloud compute environments. ## The execution signal (single source of truth) Tracer's execution signal is a structured representation of what actually ran on the system. It includes: * CPU usage and scheduling behavior * Memory allocation and pressure * Disk and network I/O activity * Process lifecycles and relationships * Container and host context **It explicitly does not include:** * Application payloads or scientific input/output data * Source code, function calls, or language-level execution traces * Application- or domain-specific interpretation of what a command does The execution signal is derived from kernel-level observation via eBPF, without application instrumentation or code changes. It serves as the shared input for both Tracer/tune and Tracer/sweep. ## How correlation works Raw kernel events are not useful on their own. Tracer/collect correlates them into meaningful execution context. At a high level: * Kernel events are associated with processes * Processes are grouped by containers and cgroups * Containers and processes are mapped to tools, tasks, runs, and pipelines This correlation allows Tracer to answer questions such as: * Which tool generated this I/O? * Which task was idle during this period? * Which pipeline run consumed these resources? All correlation is derived from operating system identifiers and execution context, not from workflow-specific integrations. ## Where Tracer/tune fits Tracer/tune focuses on pipelines that already work, but are slow or inefficient. Using the execution signal, Tracer/tune: * Visualizes actual resource usage at the task and process level * Identifies underutilization, contention, and bottlenecks * Distinguishes compute-bound, memory-bound, and I/O-bound stages * Produces evidence-based recommendations for right-sizing and optimization Tracer/tune answers: **"How do we make this pipeline faster and cheaper?"** Tracer/tune
Learn more about pipeline performance optimization ## Where Tracer/sweep fits Tracer/sweep focuses on systemwide cloud efficiency. Using the same execution signal, Tracer/sweep: * Scans cloud compute based on real execution activity * Identifies idle time, unused capacity, and hidden inefficiencies * Surfaces waste that does not appear in billing reports or dashboards * Avoids predictive shutdown heuristics by relying on observed behavior Tracer/sweep answers: **"Where are we wasting cloud spend right now?"** Tracer/sweep
Learn more about cloud waste detection ## Choose your path Depending on your goal, you can go deeper in different directions: Tracer/collect
Learn how execution signals are captured safely and efficiently at the kernel level. Tracer/tune
Learn how Tracer turns execution data into pipeline performance insights and recommendations. Tracer/sweep
Learn how Tracer uncovers cloud waste using real activity patterns.
# How It Works Source: https://opensre.com/docs/technology/how-it-works Technology overview ## Tracer Technology Benefits Tracer offers a set of core capabilities that improve visibility and analysis across scientific workflows compared to traditional monitoring tools: * Deep information: enabling faster and more accurate issue detection. * Workflow-agnostic: Works with any workflow without required modifications. * Advanced Features: driving up value creation such as resource and cost optimization
## How Tracer Can Offer This Tracer is built on top of eBPF (extended Berkeley Packet Filter), a Linux kernel technology that allows safe, high-performance instrumentation without kernel modifications. eBPF programs run inside the kernel and expose detailed telemetry to Tracer’s runtime, which correlates system events with pipeline steps, tools, and samples. Below is a high-level view of how Tracer uses eBPF during development and runtime: eBPF Overview eBPF Overview ### Development eBPF programs are compiled into safe bytecode and validated by the kernel’s verifier. Tracer distributes precompiled, architecture-compatible eBPF modules through its Go-based agent, so no user compilation is required. ### Runtime At runtime, Tracer’s eBPF modules attach to system call boundaries, network operations, scheduling events, and other kernel hooks. Using eBPF Maps, Tracer aggregates telemetry efficiently and streams enriched metrics to its backend. This enables: * Per-tool and per-task CPU/I/O visibility * Real-time failure attribution * Identification of idle or stuck processes * Network and storage performance insights * Cost and resource usage mapping at sample, step, and pipeline levels All instrumentation operates with minimal overhead and without requiring privileged kernel modules. ## Why eBPF Understanding why Tracer uses eBPF for observability Traditional approaches rely on logs, metrics exporters, or code instrumentation. eBPF serves four main purposes for Tracer:


**See everything**
System calls, process lifecycle, I/O, and scheduling events

**Stay lightweight**
Sampling at kernel level without copying large data


**Stay safe**
Verified, sandboxed bytecode that cannot crash your node

**Stay universal**
Works with any container, binary, or programming language
This provides low-overhead visibility into black-box tools, which is particularly valuable for bioinformatics and computational biology pipelines. ## How Tracer works, next to eBPF Tracer observes pipeline execution directly at the OS level using eBPF and a multi-layer data processing architecture. The workflow consists of four main stages: The Tracer agent attaches non-intrusively to running processes and containers. No restarts, code changes, or wrapper scripts are required. Using eBPF, the agent captures granular system-level signals, including CPU scheduling delays, I/O operations, memory activity, GPU context switches, and other kernel events, while maintaining low overhead. Captured events are mapped back to the structure of the pipeline, including jobs, tasks, and steps. This provides context for understanding runtime behavior across diverse tools and workflow engines. Structured telemetry is sent to the Tracer backend (on-premises or cloud). Data is aggregated, visualized in dashboards, and made available for export through standard interfaces such as APIs. Tracer Functioning Overview Tracer Functioning Overview # Limits and privacy Source: https://opensre.com/docs/technology/limits-privacy What Tracer does and does not collect Tracer is designed to provide execution insight while minimizing data exposure. It observes how workloads run, not what they compute or the data they process. This page explains Tracer's intentional limits, privacy boundaries, and data handling principles. ## What Tracer collects Tracer collects execution metadata derived from operating system–level signals. CPU usage and scheduling behavior Memory usage and peak memory Disk and network I/O activity Process start and stop times Parent–child process relationships Container, namespace, and cgroup context Cloud cost and usage identifiers (from supported providers) This data is used to reconstruct execution timelines and resource usage patterns. ## What Tracer does not collect Tracer explicitly does not collect or inspect: * Input data files * Output data or results * Sample, patient, or experimental data * File contents or payloads Tracer may observe that a file was accessed, but never reads or captures file contents. This behavior can be verified in the open-source Tracer/collect implementation. * Source code or scripts * Function calls or call stacks * Variables, objects, or in-memory data * Language-level execution traces Tracer operates at the process and kernel level, not inside language runtimes. * Environment variables * Credentials or API keys * Tokens, passwords, or certificates Tracer does not inspect process memory or application configuration. * Biological meaning or correctness * Algorithmic intent * Business or scientific interpretation of results While Tracer can observe which binaries or commands were executed, it does not infer what those commands mean within an application or domain. ## Command visibility (clarification) Tracer may observe: * Which binaries were executed * Command-line arguments passed to those binaries This visibility is limited to execution metadata and is required to correlate processes to tools and pipeline steps. Tracer does not: * Inspect data passed through those commands * Parse command arguments for domain meaning * Access application payloads ## Data minimization Tracer follows a data-minimization approach: Only metadata required for execution analysis is collected Filtering occurs as early as possible to reduce volume No payload inspection or deep packet capture is performed Collection focuses on resource behavior, not content This keeps the data footprint small and purpose-limited. ## Maintained allowlists and denylists Tracer maintains a small set of internal allowlists and denylists to focus collection on meaningful execution activity and reduce unnecessary data. These lists are used to: * Include known scientific tools, workflow binaries, and execution patterns relevant for pipeline observability * Exclude generic system activity that does not contribute to understanding workload execution (for example, background OS services) The purpose of these lists is signal quality and data minimization, not access control. ### What these lists contain Depending on configuration and environment, the lists may include: * Common scientific and ML tools and runtimes * Workflow-related binaries and schedulers * Known helper processes that are part of pipeline execution These identifiers are used only to classify execution activity and improve correlation. ### What these lists do not contain The lists do not include: * File contents or data values * User-defined secrets or identifiers * Sample, patient, or experiment metadata * Application payloads or outputs They are not used to inspect, filter, or interpret application data. ### How the lists are used * Lists are applied early in the collection process to reduce event volume * Classification happens at the level of process metadata, not data content * The lists do not change application behavior or execution outcomes In environments with custom tools or binaries, these lists can be extended or refined without redeploying workloads. ### Why this matters Maintaining explicit allowlists and denylists helps Tracer: * Minimize data collection to what is operationally relevant * Reduce overhead in high-throughput environments * Avoid collecting noisy or unrelated system activity * Preserve clear privacy and security boundaries This approach supports accurate execution insight while keeping collection conservative and purpose-limited. ## Data handling and storage * Execution signals are captured locally and aggregated into structured telemetry * Only derived metadata is transmitted to the Tracer backend * Payload data is never exported * Data retention and access are governed by account-level configuration Tracer separates collection, correlation, and analysis to reduce exposure. ## Product boundaries Tracer is intentionally scoped. **It does not:** * Modify application behavior * Control execution or scheduling * Start, stop, or terminate workloads * Replace IAM, RBAC, or cloud security controls Tracer observes execution within the boundaries enforced by the operating system, container runtime, and cloud provider. ## Transparency and open source The core Tracer agent (Tracer/collect) is open source. The repository documents how execution signals are collected, filtered, and structured, and makes it possible to independently review what data is gathered and what is explicitly excluded. This transparency supports security reviews and helps teams verify Tracer's data-collection boundaries. Review the open-source implementation ## When this matters This page is especially relevant if you: * Operate in regulated or security-sensitive environments * Need to complete security or privacy reviews * Evaluate Tracer's suitability for production workloads * Want clarity on data collection boundaries eBPF and security
How execution is observed safely Data model
How execution data is structured ## Summary Tracer provides execution visibility without inspecting application data. By limiting collection to system-level execution metadata, applying conservative filtering, and enforcing clear boundaries, Tracer delivers performance and cost insight while preserving privacy and security.
# Features Source: https://opensre.com/docs/technology/our-features Explore the features that gives developers a direct view into how scientific and ML pipelines actually run on their systems. Tracer observes what your jobs do at the operating system level and surfaces the signals that normally take weeks to piece together from logs, cloud consoles, Batch systems, and workflow engines. Tracer does not change your code, nor does it rewrite your pipelines. It doesn't require tagging, and doesn't require you to change the way you work. ## Getting started ### Installation requires no code changes Tracer installs with a single command and takes another command to run. * No changes to your scripts, workflow definitions, containers, or environments. * Once installed, your next run is visible automatically. ## Observability ### Runtime view Runtime view showing pipeline execution organized by pipeline, run, step, and tool Tracer reconstructs any pipeline execution directly from the kernel. Tracer organizes telemetry by pipeline, run, step, and tool. I.e. It mirrors how research workflows are structured. You see: * Every run currently executing * Steps, tools, and subprocesses * Start and stop times * Resource usage over time * Container lifecycle events This view does not depend on workflow metadata or logs. Because Tracer runs close to the metal it accurately reflects everything that ran. ### Tool and binary detection Tool and binary detection showing all binaries, scripts, and containers active on the cluster Instantly identify every binary, script, or container active on your cluster, including hidden dependencies or subprocesses. See the binaries inside each pipeline step, including: * Native executables * Python-spawned processes * Java tools * Shell commands * Long-lived and short-lived child processes Even tools that don't produce logs will still appear with complete runtime information. ### Kernel-level telemetry Kernel-level telemetry using eBPF for low-level performance observation
Kernel-level telemetry showing CPU, memory, disk I/O, and network activity metrics Tracer uses eBPF to observe low-level performance safely and efficiently at the kernel-level. It captures: * CPU usage * Memory usage and peak memory * Disk I/O * I/O wait * Network activity * Syscall-level metadata * Out-of-memory kill events These signals enable Tracer to report what happened without depending on user logs, and you can see any change to every metric in real-time. ### Full pipeline overview in real-time View every pipeline run as it happens. See which steps, tools, and samples are running, queued, or completed, along with their current resource usage, expected runtime, and recent progress. Tracer highlights steps that are stalled, making no forward progress, running significantly slower than usual, or consuming abnormal CPU, memory, or I/O. You can follow each step from start to finish, watch new subprocesses appear in real time, and understand how work is distributed across your nodes, without checking logs, SSH'ing into machines, or waiting for workflows to finish. ### Automatic logging Automatic logging showing structured execution timelines from kernel-level signals Tracer generates complete, structured execution timelines directly from kernel-level signals, even for tools that produce minimal logs, logs that disappear after a failure, or no logs at all. For every tool and subprocess, Tracer reconstructs when it started, when it stopped, how much CPU and memory it consumed, what files it touched, and whether it progressed. This gives you a reliable view of what happened inside steps that would otherwise be opaque, including legacy bioinformatics tools, short-lived helper binaries, child processes spawned by Python, and tools whose stderr/stdout output provides no useful context. Because the logging is derived from the operating system rather than the application, you always get complete runtime information without the need for instrumentation, wrappers, or re-running pipelines. ## Debugging ### Failure signals Failure signals showing OOM kills, stalled tools, and I/O wait issues Tracer surfaces failure conditions observed at the OS level: * Accurate OOM kills * Tools that start but make no progress * Unusually long I/O wait * Unusually long Network I/O wait * Steps that run significantly slower than typical These are tied directly to the tool or subprocess that triggered them and makes it a lot easier for you to debug and optimize your pipeline. ### Root-cause insights Root-cause insights correlating slowdowns and failures with resource behavior Tracer correlates slowdowns, queue delays, and task failures with the underlying resource behavior observed at the operating system level, including I/O wait, disk contention, network stalls, CPU oversubscription, and memory pressure. Tracer links all low-level signals to the exact tool or subprocess that triggered them, and makes it clear why a step is running slowly or failing, not just that it did. You can see when a tool stops making forward progress, when a subprocess is blocked on file I/O, when network throughput drops unexpectedly, or when a step is repeatedly retrying due to resource starvation. Instead of manually piecing together logs, cloud metrics, stderr output, and orchestrator messages, Tracer provides a precise, real-time explanation of performance issues directly in the context of the pipeline run you're debugging. ## Cost optimization ### Cost and usage tracking Cost and usage tracking broken down by pipeline, run, step, tool, and instance Tracer aggregates compute usage and cost for your cloud accounts and breaks them down by: * Pipeline * Run * Step * Tool * Instance * User * Cost center Cost is calculated using the same metrics used by the cloud provider and therefore always maps 1:1 to the cloud provider you use to run your pipeline. AWS is currently the main cost integration; GCP telemetry is supported. ### Instance rightsizing Instance rightsizing recommendations based on real resource usage Tracer analyzes real resource usage for each step and recommends smaller or more appropriate instance types when workloads are overprovisioned or better instance alternatives for the type of pipeline. ### Regional and instance family optimizations Tracer will recommend instance families that suit your task better and regions that support them. Tracer will also recommend cost optimizations across on-demand and spot pricing that's optimized by region. Recommendations are based only on observed runtime behavior. ### Idle resource detection Idle resource detection showing inactive EC2 nodes and batch workers Tracer detects: * Active EC2 nodes that aren't doing any work * Batch workers stuck idle * Instances with negligible CPU activity over time These signals help teams understand which instances can be shut down and avoid unnecessary spend. We heard from a bioinformatician that one of their interns had forgotten to turn off an instance over a weekend. The intern unfortunately managed to burn through the next six months of their cloud budget. ## Compatibility ### Framework and cloud support Tracer is framework agnostic and works with all the frameworks you use: * Nextflow * Snakemake * CWL * Argo * Custom scripts * AWS Batch * EC2-based compute * GCP (telemetry) * On-prem Linux environments * And many others Because Tracer runs at the kernel-level, you don't need any configuration for any of these systems, it all just works straight-out-of-the-box. ### Real-world environments Tracer is built to handle the complexity of production scientific computing environments. Whether you're running on cloud infrastructure, on-premises clusters, or a hybrid of both, Tracer adapts to your setup. Tracer supports: * Mixed cloud and on-prem environments * Legacy tools and custom binaries * Pipelines with many short-lived steps * Custom AMIs and machine images * Containerized and non-containerized workloads * Batch processing systems * Interactive compute sessions If the workload runs on Linux, Tracer observes it. ## Security and performance ### Security model All your data is safe and secure as Tracer does not inspect or collect: * Input data * Output data * Sample or patient data * Code * Environment variables * Secrets The Tracer agent only collects: * System-level performance telemetry * Tool and process metadata * Cloud cost and usage identifiers (From your Cloud provider) ### Overhead Tracer's eBPF-based collection runs inside the kernel with negligible overhead. The agent is designed to have minimal impact on your pipeline's performance, even when monitoring complex, multi-step workflows with thousands of subprocesses. You do not need to re-run pipelines to obtain telemetry. Key performance characteristics: * eBPF probes execute in kernel space with nanosecond-level latency * No modification to your application code or binaries * Constant memory footprint regardless of pipeline complexity * Zero network overhead for local telemetry collection # Product Benefits Source: https://opensre.com/docs/technology/product-benefits Understanding the value and opportunity of Tracer Tracer provides structured visibility into scientific and compute-intensive pipelines.
It helps teams understand where resources are used, how performance changes over time, and how workflow reliability can be improved across environments. ## Value for Developers and Engineers **Tracer was designed for scientists, DevOps engineers, and compute-heavy pipeline developers to solve real-world problems.
** The questions below reflect recurring challenges teams have shared during customer calls, the everyday issues Tracer is designed to address. | Question | How Tracer Solves It | | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | **How can I see what job is actually running right now?** | Tracer gives you real-time metrics and costs at the tool, sample, and pipeline levels, without manual digging. | | **How can we debug failures faster or even predict them?** | Tracer auto-logs every step, even tools without logs, and ties failures to system causes instantly. | | **How do we catch idle instances before they burn money?** | Tracer flags idle jobs and over-allocated compute in real time, preventing cloud burn. | | **How do we simplify our complex setups?** | Installed with one line and zero code, Tracer brings centralized observability across all environments. | Tracer helps teams move from trial-and-error debugging to insight-driven performance engineering, without rewriting pipelines. ## Cost Optimization Tracer helps organizations uncover and eliminate hidden inefficiencies in their compute infrastructure. By continuously profiling resource use at every layer, it enables teams to: * Detect over-provisioned or idle compute resources * Identify tasks that underutilize assigned CPUs or memory * Optimize job scheduling and instance types * Cut cloud and HPC costs by 20–40% on average ## Performance Improvement Beyond cost, Tracer improves scientific throughput by exposing the "why" behind pipeline slowness. * Pinpoint slow-running tasks, filesystem bottlenecks, or I/O contention * Visualize CPU, GPU, and memory utilization per container, node, or tool * Identify scheduler delays and imbalance across parallel jobs * Reduce total pipeline execution time through data-driven tuning Tracer bridges the gap between pipeline logic (WDL, Nextflow, Snakemake, etc.) and underlying system performance. ## Operational Efficiency Tracer consolidates observability across environments, helping teams manage complex scientific operations without overhead. * Unified dashboard for all pipeline runs, across cloud, on-prem, and hybrid HPC * Automatic correlation of distributed, multi-node workflows * Real-time alerts for failures or anomalies * Historical performance timelines for trend and root-cause analysis This results into fewer blind spots, faster debugging, and smoother collaboration between research and infrastructure teams. ## Next Steps Ready to see Tracer in action?
[Get started for free](https://sandbox.tracer.cloud) or [book a demo](https://www.tracer.cloud/demo) to learn more. Or dive deeper into our groundbreaking technology: **How It Works**
High-level overview of the technology eBPF eBPF **eBPF**
Revolutionary Linux kernel technology Tracer Agent Tracer Agent **Tracer Agent**
Architecture and capabilities # Tracer Agent Source: https://opensre.com/docs/technology/tracer-agent Learn about the Tracer Agent architecture and capabilities Why Tracer’s Architecture Is Unique The Tracer Agent is the heart of Tracer’s platform, a lightweight process that runs on your compute nodes and observes pipeline execution from the kernel upward. Most monitoring systems rely on application-level hooks or SDKs that require code changes and introduce overhead. In scientific workloads, where thousands of short-lived processes execute complex binaries (BLAST, BWA, GATK, TensorFlow, etc.), this approach simply doesn’t scale. Tracer solves this by leveraging eBPF (Extended Berkeley Packet Filter) technology to monitor workloads directly from the Linux kernel, safely and with minimal overhead. # Tracer/collect Source: https://opensre.com/docs/technology/tracer-collect The core data collection daemon that powers Tracer observability Tracer/collect is the core data collection daemon that powers Tracer's observability platform. It runs on Linux hosts, captures execution signals directly from the operating system, and streams structured telemetry to Tracer's backend for analysis and visualization. Unlike tools that rely on framework integrations, logs, or application instrumentation, Tracer/collect observes execution where it actually happens, at the kernel boundary between user processes and the operating system. ## Overview Tracer/collect provides low-overhead, high-fidelity visibility into scientific and cloud-based high-compute workflows. Works with any workflow engine, scheduler, language, or container runtime. Uses kernel tracing without modifying system binaries or requiring code changes. Produces data that describes what ran, where it ran, and how resources were consumed. ## How Tracer/collect works Tracer/collect is built as a static Rust daemon that orchestrates eBPF programs to observe the Linux kernel. ### Kernel-level observability with eBPF Tracer/collect uses eBPF (extended Berkeley Packet Filter) to attach probes at key points in the operating system. eBPF is a verified, sandboxed execution environment in the Linux kernel that lets user-space programs monitor kernel events safely and efficiently. Using eBPF, Tracer/collect observes: * **System calls and scheduling events**: when processes start, stop, or yield CPU * **Resource activity**: CPU usage, memory usage, I/O operations, and network activity * **Process lifecycles**: which binary ran, with what arguments, and in which container context These signals form the basis of Tracer's visibility into pipelines. eBPF allows this without inserting code into user binaries or requiring any language-specific hooks. Tool and binary detection showing all binaries, scripts, and containers active on the cluster Tracer/collect identifies every binary, script, or container active on your cluster, including hidden dependencies or subprocesses. This includes native executables, Python-spawned processes, Java tools, shell commands, and short-lived child processes. Even tools that don't produce logs appear with complete runtime information. How Tracer/collect captures execution signals from the kernel level Tracer/collect processes these kernel events and emits structured telemetry that maps to pipeline steps, tools, and tasks. ### Correlation through identifiers Tracer/collect correlates kernel events with higher-level execution semantics by using identifiers that the Linux kernel exposes: * Process IDs (PIDs) and parent PIDs * Namespaces and cgroups that define container boundaries This mapping lets Tracer distinguish between processes running concurrently on the same host and assign each observation to the correct tool, container, or pipeline step. ## Data collection stages Tracer/collect's internal workflow can be understood in four stages: eBPF probes attach to kernel trace points such as system call entry/exit and scheduler events. No application changes or container restarts are required. Kernel events are streamed into the daemon. Only relevant events, those associated with processes running scientific tools or workflow steps, are selected through filtering rules. Events are linked to processes, containers, and pipelines using kernel metadata and optional scheduler-provided trace identifiers. Aggregated telemetry is batched and sent to Tracer's backend for indexing, visualization, and analysis. ## Filtering and relevance Kernel probes can generate a high volume of events. Tracer/collect minimizes overhead and focuses on meaningful signals by applying filter rules at the source. Events that correspond to known tools, binaries, or workflow steps Generic system activity that is not useful for pipeline observability Clients can extend or refine these rules over time without redeploying the daemon. ## Deployment and lifecycle Tracer/collect is designed for modern compute environments: * It runs as a daemon on each host. * It can be installed via launch scripts, imaging systems, or orchestration tools (cloud init, configuration management, container DaemonSets). * The daemon is self-updating and resilient to intermittent network issues. Telemetry is buffered locally and retried until successfully delivered to the backend. ## Performance Tracer/collect is optimized for minimal interference with workloads: Typically under 2% for most workloads Event sampling and filtering reduce data volume early Rust's safety guarantees reduce crash risk on production systems Typical overhead measurements come from controlled benchmarks on pipeline runs. ## When to use Tracer/collect Use Tracer/collect when you want accurate, system-level insight into pipeline execution without modifying your workloads. You should deploy Tracer/collect when: * Your pipelines are stable but slow or expensive, and you need to understand resource and time usage. * You run heterogeneous workflows that span multiple tools, languages, and schedulers. * You need visibility inside containers and host processes at the execution boundary. * Application-level instrumentation is impractical or unavailable. * You want out-of-the-box insights that point you to the right optimization targets. Tracer/collect is not a language-specific profiler. If you need function-level or code-line profiling, combine Tracer's process-level signals with a dedicated profiling tool. ## Limits and boundaries **Tracer/collect does not:** * Inspect or capture application payload data (input data or scientific output) * Provide source-code level traces (e.g., function call stacks) by default * Replace language-specific profilers (for deep sampling within process logic) Instead, it surfaces resource and execution patterns that help you decide where deeper investigation belongs. ## Why Rust? Rust enables Tracer/collect to be: Reducing the risk of crashes or undefined behavior Low runtime overhead A single static binary that runs consistently across environments These properties make it suitable for high-compute systems where stability and performance are essential. ## Summary Tracer/collect is a system-level observability agent that delivers truthful execution data from the Linux kernel to Tracer's analysis backend. It provides process-level visibility without requiring code changes, supports heterogeneous pipelines, and produces data that helps you understand and optimize complex workflows. This design makes it especially useful in bioinformatics, machine learning, and scientific computing environments where tooling diversity and performance transparency are critical.
# Tracer/sweep Source: https://opensre.com/docs/technology/tracer-sweep Systemwide cloud waste detection based on real execution activity Tracer/sweep analyzes runtime behavior across your cloud compute environment to identify resources that are running but not performing useful work. It uses execution-level signals collected from each instance to locate idle machines, underutilized compute, and other forms of waste that are not visible through billing or utilization summaries. Tracer/sweep does not modify workloads or infrastructure. It only reads execution metadata and cloud billing information to provide findings. ## What Tracer/sweep does Tracer/sweep performs a read-only scan of your compute environment and: * Map it to pipeline and execution activity from Tracer/collect (if setup) * Evaluates CPU, memory, process, I/O, and container-level behavior * Detects periods where machines are running without active work * Identifies resources that appear active but have no meaningful execution * Prioritizes findings by estimated impact The analysis requires minimal setup and does not depend on tagging rules or external configuration. ## Idle Instance Scanning ### Overview Idle Instance Scanning identifies EC2 instances that remain online without running active processes or tasks. This helps teams locate compute resources that can be shut down, consolidated, or resized. Idle resource detection showing unutilized compute ### How it works Tracer/sweep relies on execution signals collected from each instance. These include: * CPU scheduling activity * Memory activity and pressure * Process start and stop events * Disk and network I/O * Container and namespace activity * GPU job activity when present Using these signals, Tracer/sweep identifies: * Instances with no running processes * Machines that remain idle after a workflow or task completes * Instances where tasks have exited or stalled but the machine continues running * GPU nodes with no scheduled workloads * Instances with periodic spikes but no sustained execution When an instance meets the idle criteria, Tracer/sweep records: * The detection timestamp * The reason for classification * The duration of the idle period (when available) These findings appear in the Tracer UI as part of the waste summary. ## What Tracer/sweep finds Tracer/sweep highlights several categories of unused or inefficient compute, including: Instances running without active processes or useful work. Machines that appear busy in high-level metrics but show little or no execution activity inside the OS. Instances left running after workflow changes, testing, or deployments. GPU servers without active kernels or workloads. All findings are based on observed execution behavior, not estimates or predictive heuristics. ## Outputs Tracer/sweep provides: * A ranked list of potential cost-saving opportunities * The detection reason for each idle or underutilized instance * Estimated impact ranges based on observed activity * Suggested next steps such as resizing or shutting down instances Findings are designed to be auditable. Each result is backed by measurable signals. ## What Tracer/sweep does not do Tracer/sweep is read-only. It does not: * Start, stop, or terminate instances * Apply predictive shutdown rules * Modify workloads or configurations * Replace cloud billing tools It surfaces evidence. Decisions remain with the user. ## Requirements Tracer/sweep requires: * Tracer/collect installed on instances to provide execution signals * A read-only IAM role to access cloud cost and usage data (AWS supported) * Access to the Tracer UI No application changes, tagging, or instrumentation are needed. ## Installation Follow these steps to connect your AWS account. ### Before you begin Ensure you have: * An AWS account with permission to create IAM roles * Access to the Tracer UI In the Tracer UI, open the provided CloudFormation link. 1. Deploy the CloudFormation stack. 2. The External ID is pre-filled. The stack creates a read-only role with the permissions required by Tracer/sweep. After deployment, locate the `TracerReadRoleArn` output. 1. Copy the Role ARN. 2. Paste it into the Tracer UI and save. Tracer will use this role to read cost and usage metadata. Once connected, Tracer/sweep analyzes your environment so you can: * Review historical cloud usage (up to 90 days) * Identify idle or low-activity instances * View current execution activity * Inspect estimated waste Findings appear automatically in the Tracer UI. ## Security model * The IAM role is read-only * No application data or secrets are accessed * Tracer does not modify infrastructure * You can revoke access at any time by deleting the IAM role ## What happens next After setup, Tracer/sweep continues to evaluate execution activity and update findings. Idle Instance Scanning runs continuously in the background. For workload-specific optimization, see Tracer/tune. ## Summary Tracer/sweep detects unused or underutilized compute by analyzing real execution activity from each instance. This makes it possible to locate idle machines and other forms of waste that do not appear in billing summaries or coarse utilization metrics. The tool is read-only, requires minimal setup, and provides evidence-backed findings that are easy to verify. Tracer/collect
Learn how execution signals are captured Tracer/tune
Optimize specific pipelines and workloads
# Tracer/tune Source: https://opensre.com/docs/technology/tracer-tune Turn "working" pipelines into fast, efficient pipelines Tracer/tune is where execution visibility becomes optimization. It answers what actually happened at runtime and turns that understanding into concrete recommendations to improve pipeline performance, stability, and cost efficiency. Tracer/tune is built on execution signals captured by Tracer/collect. It does not change pipeline code, rewrite workflows, or rely on heuristics. All recommendations are based on observed execution behavior. ## What Tracer/tune does Tracer/tune analyzes how pipelines actually ran and translates that behavior into actionable guidance. Specifically, it: * Reconstructs execution at the level of pipelines, runs, steps, tools, and subprocesses * Visualizes real CPU, memory, disk, and network usage over time * Identifies bottlenecks, idle execution, and over-allocation * Produces right-sizing and optimization recommendations grounded in runtime data Tracer/tune focuses on execution reality, not configuration intent. Runtime view showing pipeline execution organized by pipeline, run, step, and tool ## What Tracer/tune is optimized for Tracer/tune is designed for teams with pipelines that fail or already run successfully but are inefficient, unstable, or expensive. Understand patterns across many runs, not single outliers Pinpoint what actually limits progress Align resources with observed usage Reduce retries, stalls, and intermittent failures Detect performance drift over time These problems are difficult to solve with logs, dashboards, or orchestration metadata alone. ## How Tracer/tune produces recommendations Tracer/tune operates entirely on execution signals derived from the host-layer. ### Inputs Tracer/tune analyzes: * CPU utilization and scheduling behavior * Memory usage, peak memory, and pressure * Disk and network I/O throughput and wait time * Idle execution and blocked subprocesses * Variance in resource usage across runs and steps These signals come from kernel-level telemetry and reflect what actually happened during execution. Kernel-level telemetry showing CPU, memory, disk I/O, and network activity metrics ### Outputs Based on these observations, Tracer/tune produces recommendations such as: * Lowering CPU or memory requests for underutilized steps * Increasing peak memory to prevent OOM retries * Changing storage type or data locality for I/O-bound stages * Selecting more appropriate instance or node families * Highlighting steps that stall, make no forward progress, or run abnormally slow Recommendations are advisory, explainable, and based only on observed behavior. ## What you see in the Tracer UI Tracer/tune is driven by a shared execution view in the Tracer UI. You can: * Follow pipeline runs in real time, step by step * See which tools and subprocesses are active, queued, or stalled * Inspect resource usage over time for each step or tool * Compare behavior across runs to identify regressions or improvements * Understand how work is distributed across nodes and instances Root-cause insights correlating slowdowns and failures with resource behavior Automatic logging showing structured execution timelines from kernel-level signals Tracer/tune generates complete, structured execution timelines directly from kernel-level signals, even for tools that produce minimal logs, logs that disappear after a failure, or no logs at all. Because logging is derived from the operating system rather than the application, you always get complete runtime information without instrumentation, wrappers, or re-running pipelines. This view is reconstructed directly from execution signals and does not depend on workflow metadata or application logs. ## Examples These examples are framework-agnostic and apply across workflow engines and environments. ### CPU underutilization A step requests many cores but consistently uses only a small fraction. → Tracer/tune recommends lowering CPU allocation without affecting runtime. Kernel-level telemetry using eBPF for low-level performance observation ### High I/O wait A task spends most of its time blocked on disk or network I/O. → Tracer/tune recommends storage or locality changes, not additional cores. Failure signals showing OOM kills, stalled tools, and I/O wait issues ### Memory spikes and retries A step occasionally exceeds memory limits and retries. → Tracer/tune recommends right-sizing peak memory to stabilize execution. In each case, the recommendation is tied directly to observed runtime behavior. ## Cost-aware optimization Tracer/tune links execution behavior to actual cloud cost. It: * Breaks down usage and cost by pipeline, run, step, tool, and instance * Uses cloud-provider billing metrics for accurate cost attribution * Highlights over-provisioned resources that drive unnecessary spend * Supports instance rightsizing and instance family recommendations Cost and usage tracking broken down by pipeline, run, step, tool, and instance Instance rightsizing recommendations based on real resource usage Cost optimization is derived from execution data, not estimates or tagging. ## What Tracer/tune does not replace **Tracer/tune is not:** * A workflow orchestrator or scheduler * A language-level or function-level profiler Tracer/tune provides process-level execution truth. For code-level optimization, it complements traditional profilers rather than replacing them. ## Requirements Tracer/tune requires Tracer/collect to be installed and running. It supports: * AWS Batch and other Linux-based cloud compute * On-prem and hybrid HPC environments * Containerized and non-containerized workloads * Any workflow engine, scheduler, language, or binary supported by Tracer/collect No code changes, instrumentation, or tagging are required. Once installed, your next run is visible automatically. See [Quickstart](/docs/quickstart). ## Summary Tracer/tune turns execution visibility into optimization. By analyzing what actually happened at runtime, it helps teams make informed decisions about resource allocation, performance tuning, and cost control, without rewriting pipelines or changing how they work. Tracer/collect
Learn how execution data is captured Tracer/sweep
Explore systemwide cloud cost discovery
# Why eBPF Source: https://opensre.com/docs/technology/why-ebpf Understanding why Tracer uses eBPF for observability Traditional approaches rely on logs, metrics exporters, or code instrumentation.
With eBPF, Tracer: **See everything**
System calls, process lifecycle, I/O, and scheduling events **Stay lightweight**
Sampling at kernel level without copying large data **Stay safe**
Verified, sandboxed bytecode that cannot crash your node **Stay universal**
Works with any container, binary, or programming language The result: zero-overhead visibility into black-box tools, a critical advantage for bioinformatics and computational biology pipelines. # Grafana Tempo Source: https://opensre.com/docs/tempo Connect Grafana Tempo so OpenSRE can query distributed traces during investigations OpenSRE uses Grafana Tempo to investigate trace-related alerts — searching spans by service, fetching full traces by ID, listing instrumented services, and filtering by error status or latency. This integration talks to Tempo **directly** via its HTTP API. It does not require a Grafana instance or datasource proxy. If you run the full Grafana stack, the [Grafana](/docs/grafana) integration already surfaces Tempo through the datasource proxy — use this integration when you run Tempo standalone. ## Prerequisites * Grafana Tempo 1.4+ * Network access from the OpenSRE environment to your Tempo instance * Auth credentials only if your deployment requires them (many run without auth behind a gateway) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup tempo ``` You will be prompted for the Tempo URL and optional auth. Leave auth fields blank if your Tempo runs without authentication. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} TEMPO_URL=http://localhost:3200 TEMPO_API_KEY= # optional TEMPO_USERNAME= # optional (basic auth) TEMPO_PASSWORD= # optional (basic auth) TEMPO_ORG_ID= # optional (X-Scope-OrgID for multi-tenant) ``` | Variable | Default | Description | | ---------------- | --------- | -------------------------------------------------------------- | | `TEMPO_URL` | — | **Required.** Tempo HTTP API base URL | | `TEMPO_API_KEY` | *(empty)* | Bearer token for auth | | `TEMPO_USERNAME` | *(empty)* | Username for basic auth | | `TEMPO_PASSWORD` | *(empty)* | Password for basic auth | | `TEMPO_ORG_ID` | *(empty)* | Tenant ID sent as `X-Scope-OrgID` for multi-tenant deployments | ### Option 3: Persistent store Integrations are automatically persisted to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "tempo-prod", "service": "tempo", "status": "active", "credentials": { "url": "http://localhost:3200", "api_key": "" } } ] } ``` ## Investigation tools OpenSRE exposes a single `query_tempo` tool with an `action` parameter: ### search Searches traces by service, span name, duration, and tags using TraceQL. Returns one summary row per trace. ```text theme={null} query_tempo(action="search", service="checkout-service", min_duration_ms=500) query_tempo(action="search", tags={"http.status_code": "500"}) ``` | Parameter | Description | | -------------------- | --------------------------------- | | `service` | Filter by `resource.service.name` | | `span_name` | Filter by span name | | `min_duration_ms` | Minimum trace duration | | `max_duration_ms` | Maximum trace duration | | `tags` | Key/value span attributes | | `time_range_minutes` | Lookback window (default 60) | | `limit` | Max traces to return (default 20) | ### get\_trace Fetches a full trace by ID and flattens its spans. ```text theme={null} query_tempo(action="get_trace", trace_id="4f8c...e21") ``` ### list\_services Lists all services registered in Tempo. ```text theme={null} query_tempo(action="list_services") ``` ### list\_span\_names Lists all span names registered in Tempo. ```text theme={null} query_tempo(action="list_span_names") ``` ## Verify ```bash theme={null} opensre integrations verify tempo ``` Expected output: ``` Service: tempo Status: passed Detail: Connected to Grafana Tempo HTTP API (/api/search, /api/traces). ``` ## Troubleshooting | Symptom | Fix | | ---------------------- | --------------------------------------------------------------------------------------------------- | | **Connection refused** | Verify the URL and port. Default Tempo HTTP port is `3200`. | | **HTTP 401** | Set `TEMPO_API_KEY` or `TEMPO_USERNAME`/`TEMPO_PASSWORD` if your deployment requires auth. | | **HTTP 404 on verify** | Check your Tempo version — the `/api/search/tags` endpoint requires Tempo 1.4+. | | **No traces returned** | Confirm your services are sending traces to Tempo and the time range covers the period of interest. | | **Multi-tenant 400** | Set `TEMPO_ORG_ID` to the correct tenant ID. | ## Security best practices * Use a **read-only** token or service account if your Tempo deployment supports auth. * Store credentials in `.env`, never in code. * Restrict network access to Tempo — OpenSRE only needs the HTTP API port (`3200` by default). # Temporal Source: https://opensre.com/docs/temporal Connect Temporal so OpenSRE can inspect workflow executions, event history, and worker health during investigations OpenSRE queries Temporal's HTTP API to retrieve workflow executions, event history, task queue health, and namespace-level metrics — helping diagnose workflow failures, activity retries, and worker outages. OpenSRE connects to Temporal's **HTTP API** (the `/api/v1/...` REST interface served by the frontend service). This is a **self-hosted** server feature, enabled with the `--http-port` flag (dev server) or `frontend.httpPort` config. **Temporal Cloud is not currently supported**: Cloud exposes only gRPC/mTLS endpoints for workflow data and an HTTP *Ops API* for control-plane management — neither is the frontend HTTP API this integration uses. Point OpenSRE at a self-hosted Temporal deployment. ## Prerequisites * A self-hosted Temporal Server with the HTTP API enabled * The HTTP API base URL (and an API key only if your deployment requires bearer auth) Port `7233` is the **gRPC** frontend port and will **not** work as `base_url` — the HTTP API listens on a **separate** port. On the dev server it is set with `--http-port` (it otherwise defaults to a random free port). The examples below use `7243`. ## Setup ### Option 1: Environment variables ```bash theme={null} export TEMPORAL_API_URL="http://localhost:7243" export TEMPORAL_NAMESPACE="default" export TEMPORAL_API_KEY="" # only if your deployment requires bearer auth ``` ### Option 2: Persistent store Add to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "temporal-prod", "service": "temporal", "status": "active", "credentials": { "base_url": "http://temporal-frontend:7243", "namespace": "default", "api_key": "" } } ] } ``` | Field | Default | Description | | ----------- | --------- | --------------------------------------------------------------------------------------------- | | `base_url` | — | Temporal **HTTP API** base URL (the `--http-port` listener, not the gRPC `7233` port) | | `namespace` | `default` | Temporal namespace to query | | `api_key` | — | Bearer token, sent as `Authorization: Bearer `. Leave empty for unauthenticated clusters | ## Self-hosted Temporal Server Set `base_url` to the frontend's HTTP API endpoint. Ensure the HTTP API is enabled — it is a distinct listener from the gRPC frontend (`frontend.httpPort` in static config, or `--http-port` on the dev server). ### Quick local test with Docker The `temporalio/temporal` image bundles the CLI and an embedded dev server. Pin the HTTP port explicitly (it is random by default) and bind to all interfaces so it is reachable from the host: ```bash theme={null} docker run --rm \ --name temporal-dev \ -p 7233:7233 \ -p 8233:8233 \ -p 7243:7243 \ temporalio/temporal:latest \ server start-dev \ --ip 0.0.0.0 \ --http-port 7243 ``` | Port | Purpose | | ------ | ------------------------------------- | | `7233` | gRPC frontend (SDKs, `temporal` CLI) | | `8233` | Web UI (`http://localhost:8233`) | | `7243` | **HTTP API** — set this as `base_url` | Confirm the HTTP API is answering before configuring the integration: ```bash theme={null} curl -s http://localhost:7243/api/v1/namespaces/default ``` Then verify the integration end to end: ```bash theme={null} opensre integrations verify temporal ``` ## Investigation tools When OpenSRE investigates a Temporal-related alert, four diagnostic tools are available: | Tool | What it does | | -------------------- | ---------------------------------------------------------------------------------------------------------------- | | **Namespace info** | Retrieves namespace state and workflow execution counts grouped by status (Running, Failed, TimedOut) | | **Workflows** | Lists recent workflow executions with status, type, task queue, and timing | | **Workflow history** | Fetches the event history for a specific execution — shows the sequence of started, failed, and completed events | | **Task queue** | Describes a task queue's active pollers and backlog stats (queue depth, add/dispatch rates) | ### Typical investigation flow 1. **Namespace info** — get a high-level picture: how many workflows are running vs failed? 2. **Workflows** — filter to failed/timed-out executions, identify the affected workflow type and task queue 3. **Workflow history** — drill into a specific execution to find which activity failed and why 4. **Task queue** — check if workers are polling and whether the queue has a growing backlog ## Troubleshooting | Symptom | Fix | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | **Connection refused / protocol errors** | You may be pointing at the gRPC port. Use the HTTP API port (`--http-port`, e.g. `7243`), not `7233` | | **404 on `/api/v1/...`** | The HTTP API may not be enabled — confirm `--http-port` (dev) or `frontend.httpPort` (static config) is set | | **401 Unauthorized** | The cluster requires auth — set `api_key` to a valid bearer token | | **404 Namespace not found** | Confirm the `namespace` value matches exactly (case-sensitive) | | **Empty workflow list** | Workflows may have passed retention — check namespace retention settings | | **No pollers on task queue** | Workers may be down — check worker deployment health | ## Security best practices * Use a **read-only API key** where your deployment supports scoped auth — OpenSRE never writes to Temporal. * Restrict network access to the HTTP API to trusted IPs. * Store credentials in `~/.opensre/integrations.json` or environment variables, not in source code. # Terminal metrics runbook Source: https://opensre.com/docs/terminal_metrics_runbook # Terminal Metrics Runbook This runbook defines how to interpret the interactive-terminal analytics emitted by the CLI. ## Event Groups * Test execution lifecycle: `test_run_started`, `test_run_completed`, `test_run_failed`, `test_synthetic_started`, `test_synthetic_completed`, `test_synthetic_failed` * Interactive terminal behavior: `terminal_actions_planned`, `terminal_actions_executed`, `terminal_turn_summarized` ## Core KPIs * `terminal_action_execution_success_rate`: successful deterministic action executions * `terminal_fallback_rate`: share of turns that required LLM fallback ## Operational Guidance * High `terminal_fallback_rate` with low `planned_count` indicates missing deterministic action coverage; improve action recognizers before changing LLM prompts. * High `planned_count` but low execution success suggests command execution reliability issues (shell failures, missing dependencies, timeout thresholds). ## Data Contract Source of Truth * Event enum: `platform/analytics/events.py` * Capture helpers and KPI query specs: `platform/analytics/cli.py` * Provider type constraints and coercion: `platform/analytics/provider.py` # Tool placement policy Source: https://opensre.com/docs/tool-placement-policy # Tool placement policy (T-20) Where a piece of agent-callable capability lives is decided by **how many vendor integrations its domain logic touches**, not by convenience or file size. This is the decision rule referenced from [ARCHITECTURE.md](ARCHITECTURE.md#tier-2--tools-and-integrations). ## Decision rule Ask: does this tool's *purpose* depend on a specific external vendor or SaaS backend (an `integrations//` package)? 1. **Single vendor** → `integrations//tools/`. The tool only makes sense in terms of one vendor's domain (a Datadog query, a GitHub mutation, a Sentry issue lookup). This is the default, most common case — see "Adding a Tool" in `AGENTS.md`. 2. **No vendor at all** → `tools/system/`. The tool's domain purpose has nothing to do with an external vendor — local process introspection, sandboxed code execution, RAG-style guidance retrieval. An *incidental* import of a vendor client for a side concern (e.g. resolving one of several possible credential sources, or picking a notification channel for delivery) does not make a tool vendor-specific; the test is the tool's reason to exist, not every import in the file. 3. **Genuinely spans 2+ vendor integrations** → `tools/cross_vendor/`. The tool's logic itself correlates or orchestrates across multiple `integrations//` packages — e.g. `fix_sentry_issue` reads from `integrations.sentry` and hands the fix to `integrations.pi`. This is the narrow bucket; don't reach for it just because a tool happens to format its output for a second vendor (e.g. "Slack-ready" report text is still single-vendor logic, not cross-vendor). 4. **True surface-level (CLI + REPL) duplication, not tool logic at all** → `surfaces/shared/` (see T-21). This is a different axis entirely — it's about presentation code two *surfaces* both need, not about where an agent-callable tool's business logic lives. `tools/` also holds framework subsystems that aren't individual tools — `tools/investigation` (the investigation pipeline), `tools/interactive_shell` (REPL action tools), `tools/registry.py` (the tool registry itself). These stay at the top level of `tools/`; the `system` / `cross_vendor` split only applies to individual tool packages. ## Current state (as of T-19) Applied to the pre-existing top-level `tools/` packages: | Package | Placement | Why | | -------------------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `tools/system/fleet_monitoring/` | system | Local AI-agent fleet monitoring; no vendor. | | `tools/system/python_execution_tool/` | system | Generic sandboxed Python execution; the GitHub token import is one of several optional credential sources, not the tool's purpose. | | `tools/system/sre_guidance_tool/` | system | Local knowledge-base retrieval; no vendor. | | `tools/system/watch_dog/` | system | CLI/REPL process monitoring; Telegram is only the alarm-delivery channel, not the tool's domain. | | `tools/cross_vendor/fix_sentry_issue/` | cross\_vendor | Reads a Sentry issue and hands the fix to the Pi coding agent — two `integrations/` packages in one tool's logic. | **Migrated to their vendor packages** — every single-vendor tool now lives under `integrations//tools/`, so rule 1 has no exceptions left: * `integrations/github/tools/` — `architecture_issue_tool`, `community_followup_tool`, `git_deploy_timeline_tool`, `github_cli`, `work_status_report_tool`. * `integrations/pi/tools/pi_coding_tool/` — Pi-only. * `integrations/slack/tools/slack_send_message_tool/` — Slack-only. A vendor's tool package is walked once its dotted path is listed in `INTEGRATION_TOOL_PACKAGES` (`tools/registry_discovery.py`); nested tool packages under it are discovered without further wiring. ## Registry mechanics `tools/system/` and `tools/cross_vendor/` are ordinary packages discovered by `tools/registry.py`'s top-level walk of `tools/`. Each declares a `TOOL_MODULES` tuple in its `__init__.py` (the same manifest mechanism `integrations/slack/tools/slack_send_message_tool/__init__.py` already uses for its own `tool` submodule) listing the tool packages nested one level inside it. Adding a new system or cross-vendor tool means adding its package name to the relevant `TOOL_MODULES` tuple — no registry code changes required. # Tracer FAQ Source: https://opensre.com/docs/tracer-faq Frequently asked questions about Tracer Yes. In most cases, this is not a problem because observability platforms like Grafana and Datadog already aggregate data across multiple cloud environments, and Tracer works on top of that information. Tracer also supports direct cloud integrations for teams that want more control, lower cost, or a more self-hosted setup. Currently, we support AWS connectors. No. We started in data pipelines, and Tracer is still very strong there, but we have already expanded to a much broader set of use cases, including Kubernetes and web applications. Yes. We plan to add more direct cloud-provider connectors over time. # Turn scenario test gap Source: https://opensre.com/docs/turn-scenario-test-gap # Memorandum: Turn Scenario Test Infrastructure Gap **Date:** 2026-06-18\ **Concerns:** `complex_shell_prompts` scenario class; oracle coverage of conversational tool-gathering\ **Status:** Partially addressed (2026-06-26) — gather recording, `tool_actions`, fixture `resolved_integrations`, and `@live` fail-closed CI are in place; many handoff scenarios still rely on text-only contracts > **Update (2026-06-26):** Natural-language investigation dispatch is re-enabled > (`INTERACTIVE_SHELL_INVESTIGATION_ENABLED = True`). Scenarios **314**, **338**, > **339**, and **315** assert gather dispatch via `tool_actions` with fixture > integrations; **333–335** and **337** use `@live` for canonical per-integration > gather. Handoff-only **313** lives under `chat_handoff/`. Remaining gap: > scenarios without `tool_actions` gather entries still pass on hallucination-satisfiable > text contracts only. > **Update (2026-06-19):** The scenario schema has since been trimmed and the > oracle's capability defaults realigned with production. `available_capabilities` > is now a three-state knob (omit = enabled/production default, `[]` = disabled, > non-empty = allowlist) instead of disabling slash/cli/synthetic by default, and > the dead `risk_level`/`tier`/`remote_connected`/`surface` fields were removed. > See the "Scenario schema and `available_capabilities` semantics" section of > `core/agent_harness/AGENTS.md` for the canonical contract. *** ## Summary The turn scenario oracle (`_oracle_runtime.py`) does not observe, assert on, or control the conversational tool-gathering path (`gather_tool_evidence` → `Agent.run`). Every `complex_shell_prompts` scenario passes in CI even when zero integrations are queried and the response is entirely hallucinated text. The test infrastructure provides confidence that does not exist. *** ## 1. The Two Execution Paths — Only One Is Tested When a REPL turn enters `run_agent_prompt`, two independent paths can fire: | Path | What it does | Oracle coverage | | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | **Action agent → AgentTool execution** | LLM proposes shell action tool calls (slash, investigation, shell, etc.); the oracle observes the terminal side effects recorded by the action tools | **Fully observed and asserted** | | **`gather_tool_evidence` → shared runtime loop** | A bounded ReAct loop queries registered tools (Sentry, GitHub, PostHog, etc.) to ground a conversational answer | **Completely unobserved** | The oracle observes the action-agent execution path. It does not patch `gather_tool_evidence`, the shared tool-gathering harness, or `_resolve_session_integrations`. Tool calls made during the gather pass are invisible to the test. *** ## 2. `configured_integrations` in Fixtures Does Not Isolate the Store `fresh_session` applies the fixture's `configured_integrations` list to `session.configured_integrations`. This field controls the LLM system-prompt copy and the REPL status bar. It does not control which integrations are **actually loaded** for the gather loop. `_resolve_session_integrations` ignores `session.configured_integrations` entirely: ```python theme={null} # interactive_shell/runtime/integration_tool_gathering.py def _resolve_session_integrations(session: Session) -> dict[str, Any]: if session.resolved_integrations_cache is not None: return session.resolved_integrations_cache resolved = resolve_integrations({}) # hits the real env and ~/.opensre/integrations.json session.resolved_integrations_cache = resolved return resolved ``` `resolve_integrations({})` reads the developer's live `~/.opensre/integrations.json` and environment variables. This produces three distinct, silent behaviours across environments: | Environment | What `resolve_integrations` returns | Tool-gathering outcome | | ---------------------------------------- | ----------------------------------------- | ---------------------------------------------------------- | | CI (no store, no env keys) | `{}` — no tools available | Gathering is a no-op; text-only answer | | Developer machine, no keys | `{}` | Same no-op | | Developer machine with real integrations | Real configs (PostHog, GitHub, Sentry, …) | Real tool calls fire with the developer's live credentials | A scenario that declares `configured_integrations: [sentry, github, posthog]` in CI runs exactly the same code path as one that declares `configured_integrations: []`. The field is decoration, not isolation. *** ## 3. Response Contracts Are Satisfiable by Hallucination Alone Because the gather loop is a no-op in CI, the response evaluated against the contract is produced by the LLM from its training data, not from any live integration. The contracts for the two `complex_shell_prompts` scenarios are: **313** (`configured_integrations: []`) ```yaml theme={null} must_contain_any: [GitHub, issues, Windows, crash] ``` Any response that mentions "GitHub" passes. The LLM always mentions GitHub when asked about GitHub issues. **314** (`configured_integrations: [sentry, github, posthog]`) ```yaml theme={null} must_contain_all: [Sentry, GitHub, PostHog] must_contain_any: [Windows, crash] ``` Any response that mentions all three names passes — including a response that says "I cannot access Sentry, GitHub, or PostHog right now." The scenario explicitly notes the agent "must commit to checking the connected sources," but the contract cannot verify this because it cannot observe whether any source was actually checked. *** ## 4. The Behaviour Proven by Current Tests Across all 54 turn scenarios, what passes in CI is: * **Turn-entry correctness** — every turn is handed to the agent entrypoint. This is intentionally static; the valuable behavior is downstream dispatch and planning. * **Deterministic command-text detection** — slash commands and aliases resolve correctly for UI policy decisions. This is genuine and valuable. * **Planned terminal action shape** — when a planner action fires (slash, shell, investigation), the oracle records and asserts it correctly. This is genuine and valuable. * **Hallucination-satisfiable text contracts** — for all conversational turns, the contract is met by the LLM generating plausible text that mentions the right words. **This is not a meaningful signal.** What is not proven: * Whether the gather loop fires at all. * Whether any specific tool was called. * Whether any integration returned data. * Whether the response is grounded in integration data vs. generated from training knowledge. * Whether a broken integration (validation error, auth failure, timeout) prevents the response from being useful. *** ## 5. Why This Is a Large Correctness Risk The `complex_shell_prompts` class exists specifically to cover the integration data-gathering surface — the tests are named and described as covering exactly what they do not cover. This creates three concrete risks: **Risk 1: Broken tool extraction goes undetected.**\ If `_posthog_mcp_extract_params` starts returning bad config fields (as happened: live PostHog calls received `posthog_mode="mcp"` from the LLM), the scenario passes. The broken extraction is only discovered when a user exercises the feature interactively. **Risk 2: Integration registration silently drops.**\ If a tool's `is_available` check starts returning `False` for all sessions, or if the tool is accidentally deregistered, every `complex_shell_prompts` scenario still passes. A regression that stops the agent from ever querying GitHub or PostHog cannot be caught by the current test suite. **Risk 3: The no-mocks policy blocks the obvious fix.**\ `AGENTS.md` and `test_turn_fixture_integrity.py` enforce a hard no-mocks rule on the turn oracle: > "Do not use `unittest.mock`, `patch`, `MagicMock`, or equivalent mocking > primitives in turn tests." The intent of this rule is correct — it prevents tests from faking the LLM and making action-planning assertions against synthetic planner output. But it accidentally also blocks injecting a controlled integration config into the gather loop, which does not involve the LLM at all. The rule currently prevents the fix. *** ## 6. Root Cause: Architectural Seam Is Missing The docstring in `scenario_loader.py` acknowledges the gap explicitly: ```python theme={null} # Answer docstring, path 2: # "Deeper 'did it actually query the integration?' assertions belong in # execution-layer tests, not these turn fixtures." ``` That execution-layer test does not exist. `tests/interactive_shell/runtime/ test_answer_with_tools.py` patches both `gather_tool_evidence` and `generate_response` entirely, so it tests the wiring between them (gather output flows to answer), not whether the gather loop calls the right tools with the right config. The gap noted in the docstring has never been closed. *** ## 7. Proposed Remediation Three changes are required, in dependency order. ### 7.1 — Add a stable test seam for integration injection Add `resolved_integrations_override` support to `fresh_session` in the oracle. When set, `_resolve_session_integrations` returns the override instead of hitting the real store. This does not mock the LLM, does not mock any tool, and does not violate the spirit of the no-mocks rule — it controls the integration config the tool is called with, which is fixture input, not LLM output. ```python theme={null} # _oracle_runtime.py def fresh_session( *, with_prior_state: bool, configured_integrations: tuple[str, ...] = (), available_capabilities: dict[str, tuple[str, ...]] | None = None, resolved_integrations_override: dict[str, Any] | None = None, ) -> Session: session = Session() ... if resolved_integrations_override is not None: session.resolved_integrations_cache = resolved_integrations_override return session ``` `run_oracle_once` reads this from `case.scenario.session.resolved_integrations` when present, and uses `{}` (no-op gather) otherwise. CI remains fast because no fixture currently sets this field. ### 7.2 — Track gather-loop tool calls in the oracle Wrap `Agent.run` with a thin recorder inside `run_oracle_once` so tool calls made during gathering are captured alongside planned terminal actions. This does not mock the tools themselves; it records which ones fired. ```python theme={null} # _oracle_runtime.py gathered_calls: list[str] = [] original_run = Agent.run def _recording_run(self, initial_messages): result = original_run(self, initial_messages) for tc, _ in result.executed: gathered_calls.append(tc.name) return result monkeypatch.setattr(Agent, "run", _recording_run) ``` The oracle result gains `gathered_tool_calls: list[str]` and the OracleRunResult exposes this for contract assertions. ### 7.3 — Add `tool_actions` gather entries to the scenario schema Fixtures now use a unified `tool_actions` list with `surface: gather` and `expect` modes instead of a separate `gathered_tools_contract` block. Example: ```yaml theme={null} tool_actions: - surface: gather tool: search_sentry_issues expect: valid_data - surface: gather tools: [search_github_issues, list_posthog_tools] expect: not_called ``` Extend the YAML schema with an optional section that the scenario loader validates and the oracle asserts: ```yaml theme={null} gathered_tools_contract: must_call_any: # at least one of these tool names must appear - list_github_issues - search_github_issues must_not_call: # none of these must appear - run_investigation - execute_shell_command ``` Updated `314-windows-crash-multisource-query.yml`: ```yaml theme={null} session: configured_integrations: [sentry, github, posthog_mcp] resolved_integrations: # injected into session cache; tool calls run for real sentry: connection_verified: true auth_token: "test-token" ... github: connection_verified: true ... posthog_mcp: connection_verified: true mode: streamable-http ... gathered_tools_contract: must_call_any: - search_sentry_issues - list_sentry_issues - search_github_issues - list_github_issues - list_posthog_tools ``` With the override in the session cache, the tools run with the fixture config (no live credentials needed). With the gather recorder active, the contract is asserted. A broken `is_available` check or bad `extract_params` now fails the test immediately. ### 7.4 — Update the no-mocks rule scope Amend the "no mocks" policy in `AGENTS.md` and `test_turn_fixture_integrity.py` to distinguish between two separate things: * **Mocking the LLM** — prohibited. Turn oracle must exercise the real LLM. * **Injecting fixture integration configs** — permitted. This is equivalent to providing test credentials and does not involve the LLM. Add an AST check that specifically permits `monkeypatch.setattr` on `integration_tool_gathering._resolve_session_integrations` and `core.agent.Agent.run` while continuing to prohibit `patch`, `MagicMock`, and LLM client stubs. ### 7.5 — Rename or reclassify misleading existing scenarios **313** (`configured_integrations: []`) is now under `chat_handoff/` with `tool_actions` gather `not_called` assertions. It covers the no-integration handoff path, not live data gathering. **338** and **339** assert gather `call_any` with fixture `resolved_integrations`. *** ## 8. Migration Path and Priority | Step | Effort | Risk | Priority | | ---------------------------------------------------------------- | ------------------ | ---- | --------------------------------------------- | | 7.1 — `resolved_integrations_override` seam | Small (20 lines) | Low | **P0** — unblocks everything else | | 7.2 — gather-loop tool call recorder | Small (30 lines) | Low | **P0** — required for assertions | | 7.3 — `gathered_tools_contract` schema + assertions | Medium (100 lines) | Low | **P1** — makes contracts meaningful | | 7.4 — Update no-mocks rule scope | Trivial | None | **P1** — prevents the fix from being reverted | | 7.5 — Reclassify 313 | Trivial | None | **P2** — clarity, not correctness | | Write new `complex_shell_prompts` scenarios with fixture configs | Medium | Low | **P1** — actual test coverage | Items 7.1 and 7.2 can land in one PR. Items 7.3 and 7.4 land together. New scenario fixtures follow. *** ## 9. What Does Not Change * The no-mocks policy on the LLM path. The planner, classifier, and conversational assistant all continue to hit the real LLM in turn tests. * The turn-execution oracle still invokes `run_agent_prompt` directly. * Any existing passing scenario. The `resolved_integrations_override` is opt-in; existing scenarios without it keep the current no-op gather behaviour and continue to pass. * CI runtime budget. Fixture-injected integration configs do not make live network calls (tools check `is_available` against the resolved dict, not a live endpoint), so test time stays flat. *** ## 10. Acceptance Criteria for "Fixed" 1. A scenario in `complex_shell_prompts` with `resolved_integrations` injected and `gathered_tools_contract` defined **fails** when the named tools do not fire. 2. The same scenario **passes** when the tools fire and return data. 3. Introducing a bug in `_posthog_mcp_extract_params` (e.g. passing `mode="mcp"`) causes the affected scenario to **fail** in CI. 4. A tool whose `is_available` is patched to always return `False` causes the scenario to **fail** if it is in `gathered_tools_contract.must_call_any`. 5. No existing scenario changes its pass/fail status. 6. CI runtime increases by less than 10 seconds per shard. # Debugging nf-core demo pipeline Source: https://opensre.com/docs/tutorials/fastquorum-debugging Using Tracer to diagnose and optimize UMI-based consensus sequencing This tutorial walks a bioinformatics engineer through real-time observability of the nf-core/fastquorum pipeline using Tracer's eBPF-powered monitoring. We simulate a small but realistic UMI-based duplex sequencing workflow on a single chromosome (chr17.fa), run it in a GitHub Codespace, and use Tracer to detect resource bottlenecks, identify redundant I/O, and explain why the pipeline completed in 1m 36s despite only 12 processes. ## What You'll Learn * Connect a live Codespace to the Tracer sandbox * Auto-instrument a Nextflow pipeline with zero code changes * Visualize per-process CPU, memory, and I/O in real time * Extract actionable optimization insights **Why this matters:** fastquorum is complex (UMI grouping, consensus calling, dual alignment). Without OS-level visibility, engineers guess where time is spent. Tracer shows exactly which process is the bottleneck — no logs, no profiling flags. ## Tools Used * **Pipeline:** nf-core/fastquorum v1.0.0+ * **Environment:** GitHub Codespaces (Ubuntu 22.04, 4-core, 16GB RAM) * **Observability:** Tracer.bio (eBPF) * **Container:** Docker * **Genome:** chr17.fa (subset of GRCh38) ## 1. Login & Setup: Tracer Sandbox + GitHub Codespaces We begin in a GitHub Codespace — a reproducible, cloud-based dev environment that mimics a local VM. Tracer's eBPF agent runs natively here and streams metrics to the Tracer Sandbox Dashboard ([https://dev.sandbox.tracer.cloud](https://dev.sandbox.tracer.cloud)) in real time. 1. Go to [GitHub Codespaces](https://github.com/codespaces) 2. Click **"New codespace"** 3. Select **"Create your own"** → Paste this repo: `https://github.com/yourusername/nfcore-fastquorum-tracer-demo` 4. Choose machine: **4-core, 16GB RAM** (required for Docker + Nextflow) 5. Click **Create codespace** Codespace display with the cloned nf-core pipeline repository *Fig 1: Codespace display with the cloned nf-core pipeline repository* In the Codespaces terminal, run: ```bash theme={null} curl -sSL https://install.tracer.cloud | CLI_BRANCH=dev sh -s user_35Fukh3QxSAxJLgfyE9SwPoPy9K ``` To start tracking a pipeline, run the following command: ```bash theme={null} tracer init --token eyJh---- (your token) ``` Successful connection snapshot 1 Successful connection snapshot 2 Successful connection snapshot 3 *Fig 2: You will see something like this upon successful connection (Snapshot of tracer init command which connecting to tracer)* With the Tracer agent connected, input validated, and genome indexed, we now execute the full nf-core/fastquorum pipeline. No code changes are required — Tracer's eBPF hooks automatically detect nextflow launches, label processes, and stream OS-level metrics (CPU, RAM, I/O, syscalls) to your sandbox dashboard in real time. ## 2. Dataset Preparation This section is critical — nf-core/fastquorum enforces strict requirements on input format, UMI placement, and file integrity. ### Key Preparation Steps We begin by downloading real test data directly from the nf-core test-datasets repository, ensuring authenticity and compatibility. Confirm UMI structure — in this case, a 6-base inline UMI (NNNNNN) embedded at the start of Read 1, which matches the expected pattern for duplex consensus sequencing. Ensure all FASTQs are properly gzipped and accessible via relative paths to avoid runtime errors. A correctly formatted `samplesheet.csv` is constructed with mandatory columns: `sample`, `fastq_1`, `fastq_2`, `umi_read`, and `umi_pattern`, adhering to the pipeline's JSON schema. To eliminate I/O noise during the observed run, the genome index (BWA-MEM1, SAMtools FAIDX, and DICT) is pre-built locally and stored for reuse, ensuring clean, reproducible eBPF telemetry from Tracer. ## 3. Launch the Pipeline From the pipeline root: ```bash theme={null} nextflow run . \ --input samplesheet.csv \ --fasta data/chr17.fa \ --outdir results \ --duplex_seq true \ -profile test,docker \ -with-trace \ -with-report results/report.html ``` ### Parameters | Flag | Purpose | | ---------------------------------- | -------------------------------- | | `--input samplesheet.csv` | Validated manifest | | `--fasta data/chr17.fa` | Local reference | | `--duplex_seq true` | Enable duplex consensus | | `-profile test,docker` | Use test config + containers | | `-with-trace` | Nextflow-native trace (optional) | | `-with-report results/report.html` | HTML execution report | ## 4. Live Visualization: Tracer Dashboard During Execution With the nf-core/fastquorum pipeline launched and Tracer's eBPF agent actively streaming OS-level events, the Tracer Sandbox Dashboard becomes a real-time observability cockpit. No polling, no logs — just continuous, kernel-level telemetry delivered via WebSocket every 2 seconds. ### Dashboard Entry Point: Run Overview Upon launching `nextflow run .`, a new run card appears instantly: **Run Overview Card:** * **Run Name:** run\_1 * **Status:** Running (blue dot) * **Elapsed:** 45s and counting * **Max RAM:** 12 / 100% → 12 GB peak (of 16 GB available) * **Avg. CPU:** 36 / 100% → 36% average across 4 cores * **Disk I/O:** 17 / 100% → 17% of max bandwidth Run Overview Snapshot *Fig 3: Run Overview Snapshot* Compact Summary This compact summary is the first signal that Tracer has auto-detected the Nextflow executor and attached to all child processes — no `-with-trace` or config changes needed. The progress bar fills as tasks complete, and resource meters update in real time. ### System Specs & Cost Panel | Metric | Value | Status | | -------------- | ----------------------- | ---------------------- | | **RAM** | 2.97 GB used / 15.62 GB | HEALTHY | | **CPU** | 1.81 cores / 4 cores | HEALTHY | | **DISK** | 42.90 GB / 207.35 GB | HEALTHY | | **GPU** | Not detected | — | | **TOTAL COST** | \$0.00 | Free tier (Codespaces) | System Specs & Cost Panel This panel confirms the GitHub Codespaces environment: a 4-core, 16 GB VM with ample headroom. The cost meter at \$0.00 reflects that this is a non-billable sandbox run, but in production (e.g., AWS EC2), Tracer would estimate hourly cost based on instance type and utilization. ### Tool Table: Real-Time Process Monitoring **Table Observations:** * `bwa index` is still running — expected: indexing chr17.fa (\~80MB) is CPU-heavy * FastQC hit 118% CPU → Java thread burst (common in multi-threaded mode) * `samtools faidx` is I/O-light — just reads the FASTA once * Status badges update live: Running → Success as tasks finish **Visual Insights:** * **Critical path:** bwa index → FastqToBam → GroupReadsByUmi * **Parallelism:** samtools faidx and dict run concurrently with FastQC * **Tail latency:** Final MultiQC runs alone This Gantt view is interactive — hover to see exact command, stdout, and resource curve. | Tool | Status | Runtime | Max RAM | Max CPU | Max Disk I/O | | ---------------- | ------- | -------- | ------- | ------- | ------------ | | bwa index | Running | 9s 851ms | 0.12 GB | 115.49% | 0.04 GB | | samtools faidx | Success | 482ms | 0.00 GB | 38.10% | 0.00 GB | | samtools dict | Success | 1s 111ms | 0.08 GB | 54.63% | 0.08 GB | | FastQC | Success | 5s 775ms | 0.30 GB | 118.23% | 0.01 GB | | fgbio FastqToBam | Success | 4s 813ms | 0.14 GB | 120.60% | 0.00 GB | Timeline view Table and visual insights for the tools running in pipeline at real-time *Fig 4: Table (detailed) and visual insights for the tools running in pipeline at real-time* ### Metrics Over Time: System-Level Trends **CPU Usage:** * Avg: 91.4% * Max: 115.5% (burst during bwa index) * Pattern: High at start (indexing), drops to \~70% during alignment **Memory Usage:** * Avg: 99.8 MB * Max: 121.5 MB * Spike at 6s: fgbio FastqToBam loads both FASTQs into memory **Disk I/O:** * Avg: 0.08 GB * Max: 0.18 GB * Burst at 40s: Writing intermediate BAM files **Network I/O:** * Avg: 81.42 MB * Max: 180.80 MB * Cause: Docker pulling nf-core/fastquorum:1.2.0 layers (first run) CPU, Memory, Disk, Network Over Time Metrics Over Time 2 *Fig 5,6: System level trend* ## 5. Post-Run Analysis: Resource Heatmap & Bottleneck Detection The pipeline completes in **1m 36s** with **12 successful tasks**. Now we analyze the full trace. ### Resource Analysis | Process | CPU (avg) | RAM (peak) | I/O (total) | Duration | | -------------------- | --------- | ---------- | ----------- | -------- | | BWAMEM1\_INDEX | 95% | 1.4 GB | 180 MB | 53s | | GROUPREADSBYUMI | 99% | 3.1 GB | 42 MB | 24s | | CALLDDUPLEXCONSENSUS | 60% | 1.8 GB | 28 MB | 16s | | FASTQTOBAM | 75% | 1.2 GB | 35 MB | 18s | ### Key Insights BWAMEM1\_INDEX (53s) is the bottleneck — accounts for 55% of total runtime GROUPREADSBYUMI peaks at 3.1 GB — consider increasing memory allocation for larger datasets Most processes utilize >75% CPU — good parallelization Total I/O: 285 MB — minimal disk bottleneck detected ## 6. Conclusion In the fast-evolving landscape of bioinformatics, where pipelines demand precision amid mounting computational complexity, **Tracer emerges as an indispensable ally** for bioinformaticians seeking deeper, actionable insights without the burden of invasive instrumentation. ### Key Benefits By harnessing **eBPF technology** at the operating system level, Tracer delivers: * **Real-time observability** into every facet of your workflows (Nextflow, WDL, Bash, or CWL) * **Automatic detection** of hangs, crashes, and silent failures that traditional logs often overlook * **One-minute setup** with zero code modifications ### Real-World Impact Imagine pinpointing the exact genome file or tool process causing a crash in a duplex sequencing run, or uncovering memory oversizing in dependency updates that could shave weeks off troubleshooting. Tracer excels in resource orchestration, spotlighting inefficiencies like redundant I/O in alignment steps or overprovisioned instances. AI-driven recommendations enable right-sizing of compute environments in mere clicks, potentially slashing costs by 30% or more on cloud platforms, paying only 5% of your pipeline's compute expenses without upfront fees. ### Your Next Steps For bioinformaticians juggling high-throughput NGS data, evolving dependencies, and the pressure to derive reproducible insights from vast datasets, **Tracer isn't just a monitoring tool — it's a superpower** that shifts focus from infrastructure headaches to scientific discovery, fostering scalable, cost-effective workflows that accelerate breakthroughs in genomics, proteomics, and beyond. Dive into the Tracer sandbox today and experience how effortless observability can redefine your pipeline mastery. ## Related Tutorials Learn how to monitor task execution in real-time Debug and resolve failures with diagnostic tools # Investigating Task Failures Source: https://opensre.com/docs/tutorials/investigating-task-failures Debug and resolve failures with Tracer powerful diagnostic tools ## Overview Tracer automatically captures logs, resource metrics, and system call data for every task, even those that fail. Therefore, when tasks fail, Tracer provides detailed information to help you understand what went wrong and how to fix it. Find out how to do this easily below. ## Previous Knowledge Before diving in how we can investigate task failures, it is recommended to have a basic understanding of the following concepts: See the run details Monitoring your tasks ## Identifying Failed Tasks When you see the run overview and notice a failed tool, you can click on the "Logs" tab to see the error summary and exit code.
Run Overview As you can see in the image above, multiple tasks failed and therefore need to be investigated. Here, all logs of the failed run are shown. Select the log you want to investigate deeper. Log Overview This page displays your log and its insights. It includes an error summary, plus automatic logs with warning and failure indicators.
Based on this data, our AI identifies likely root causes and tells you what happened, so you know exactly what needs attention. It also provides recommended solutions to resolve the issue. Log Details Our AI Log Analysis is divide up into three main sections: 1. **Critical Issue Section** - What exactly went wrong 2. **Next steps/Solution Suggestions** - How to resolve the issue and refrain from making the same mistake again 3. **Error Entries** - The specific lines in the log that caused the error AI Insights If you want to dig deeper into the logs, in this section you can see the full log with highlighted error lines.
On the right side, you can see the specific error entries that caused the task to fail together with the warning indicators. This also gives you the opportunity to download the full report. Log Details As logs can be very extensive, there are multiple ways of searching through the logs. You can use the search bar to search for specific keywords, or you can use the filter bar to filter for specific error types and you can filter on time as well. ## Common Failure Patterns ### Resource Exhaustion Tasks may fail due to insufficient resources: * **Out of Memory (OOM)** - Task exceeded available RAM * **Disk Space** - Insufficient storage for outputs * **CPU Timeout** - Task exceeded maximum execution time Tracer's eBPF monitoring captures resource usage leading up to failures, helping you identify resource constraints. ## Next Steps Learn how to monitor task execution # Viewing Task Status Source: https://opensre.com/docs/tutorials/viewing-task-status Learn how to monitor and view the status of your tasks in Tracer On this page, we will explain how you can see the status of your tasks within a specific pipeline run.
If you have not yet run a pipeline, please refer to the [quickstart](/docs/quickstart) to get started. When executed correctly, you should be able to see the following screen in the Tracer dashboard: Tracer updates task status in real time, so you can monitor progress without refreshing the page. Tracer provides detailed visibility into task execution, allowing you to track progress, identify bottlenecks, and ensure your workflows are running smoothly. ## Accessing Task Status Open the Tracer dashboard and select the **"Pipelines"** tab next to "Overview". This will show you all the pipelines that have been run. General Overview When hovering over the pipeline runs, you get more information about what pipelines have been run or are currently running. Access your specific pipeline by clicking on the pipeline name. You can recognize the pipelines which are currently running by the blue dot next to the pipeline name. Pipeline Overview As you will most likely check the current or latest run of the selected pipeline, you can click the blue link **"Latest Run"** to go to the latest run. If you are searching for an older run, you can click on the **"Pipeline Runs"** box in the bottom left corner of the screen. This will show you all runs, where you can select the run you are looking for. Pipeline Runs Here, you get the first high-level overview of the pipeline run. You can see the main metrics, runtime and cost as well as the automatic logs with warnings and failure indicators. To go to the task status, click on the **"Tools"** tab next to "Overview". This will show you the status of all tasks within this pipeline run. Runs Overview The tool visualizer shows you the status of all tasks within this pipeline run: * **Bar width** indicates the runtime of the task * **Bar color** indicates the status: Grey (running), Green (completed), Red (failed) Tool Visualiser In the image above, all tools are completed as indicated by the green bars. **When hovering over a specific task**, you can see the status, runtime, and the exact moment the task started. By clicking the bar, you can dig one level deeper to see the metrics, runtime, cost, and logs with warnings and failure indicators. Running Hover In the image above, the task is still running as indicated by the grey bar and has been running for exactly 30.591s. **For a more detailed overview**, click on the **"Detailed"** button in the top right corner of the screen. This will list all tools in a table showing status, command, start time, runtime, max RAM, max CPU, max Disk I/O, and end time. Detailed View In the image above, you can see that one task is still running as indicated by the "Running" status, while the others are successfully completed. Here, you can see the run-specific metrics including: * **Tool ID** - Unique identifier for the task * **Run Name** - Name of the current run * **Pipeline Name** - Name of the parent pipeline * **AI Insights** - Intelligent analysis and recommendations * **Runtime Metrics** - CPU, memory, and I/O usage You can also compare this task's performance to previous runs. Tool Metrics **After these steps, you’ll have live visibility into task progress, metrics, and logs for your pipeline run.** ## Task Status Indicators Tracer displays tasks with the following status indicators: | Status | Description | Color Used to Indicate | | ------------- | --------------------------- | ---------------------- | | **Running** | Task is currently executing | Grey | | **Completed** | Task finished successfully | Green | | **Failed** | Task encountered an error | Red | ## Task Metrics When viewing a task, you'll see detailed information including: | Category | Metric | Description | | -------------------------- | --------------------- | ------------------------------------------------------ | | **Status & Timing** | Status | Current state of the task (Running, Completed, Failed) | | | Runtime | Total execution time | | | Start Time | When the task began | | | End Time | When the task completed (if applicable) | | **Resource Usage** | CPU | Processor utilization | | | RAM Memory | Memory consumption | | | Disk I/O | Read/write operations | | | Network I/O | Network traffic | | **Logs & Debugging** | Automatic Logs | Standard output and deep system-level logs captured | | | Warnings | Highlighted warnings and failure indicators | | **Additional Information** | Command | The exact command executed | | | Historical Comparison | Performance comparison with previous runs of this task | ## Next Steps How to get even more out of Tracer's capabilities: Learn how to debug failed tasks # Cost Center Mapping Source: https://opensre.com/docs/use-cases/cost-center-mapping Attribute cloud costs to projects, teams, and research groups ## Cost Center Mapping Cost center mapping in Tracer gives organizations full transparency into where their compute budget is going. Instead of a single, opaque cloud bill, Tracer maps every cost to the right project, team, grant, or principal investigator automatically. ## How It Works Tracer enables precise cost attribution by mapping compute spending to cost centers, projects, teams, grants, or any custom organizational dimension. * Accurate per-project and per-team cost attribution for finance, ops, and R\&D * Improved budget forecasting and accountability for research and compute spend * Greater cost efficiency through clear visibility into resource usage * Easier optimization and fair cost allocation across teams and projects Each pipeline run is tagged with relevant metadata, enabling real-time and historical reporting across multiple dimensions. Finance teams, researchers, and administrators can easily see who is spending what, compare groups, and manage budgets proactively. With built-in tagging, attribution, and budget tracking, Tracer turns cloud billing data into clear, auditable cost insights. This eliminates guesswork and enables fair, data-driven cost allocation across your organization. # Cost Optimization Source: https://opensre.com/docs/use-cases/cost-optimization Tools for understanding and reducing compute costs Tracer provides cost-related metrics and execution data to help teams understand how compute resources are used and where waste occurs. These features do not modify workloads. They make it easier to make informed decisions about scaling, allocation, and configuration. ## Cost Analysis Cost Analysis shows compute spend at the cluster, namespace, and workload levels. Tracer combines cloud billing information with execution signals such as CPU usage, memory usage, and process activity. Cost Overview dashboard showing pipeline costs broken down by user, department, environment, and pipeline type This allows you to: * Review cost patterns over time * Identify workloads that consistently use more or fewer resources than expected * Compare cost across pipeline runs * Pinpoint costs by container, tool, job, and run * Detect over-allocated or under-allocated configurations * Attribute costs by pipeline type, user, environment, or cost department Cost breakdown by tool showing resource usage across different computational tools Tracer displays these metrics using simple graphs that relate cost to observed execution behavior. This helps determine whether a workload is constrained, over-provisioned, or operating as expected. You can use historical data to compare configuration changes or evaluate different instance types before adjusting compute resources. For more information on how costs are attributed, see [Cost Center Mapping](/docs/use-cases/cost-center-mapping). ## Resource Efficiency Detection Tracer evaluates runtime behavior to highlight potential inefficiencies. Examples include: * Instances that use only a fraction of their allocated CPU or memory * Workloads with low execution activity relative to provisioned capacity * Containers that remain active after their tasks complete * Nodes that stay online without active workloads These findings are based on measured execution signals, not predictions or estimates. They can help identify where right-sizing or consolidation may reduce cost. ## No Automatic Optimization Tracer does not apply automatic scaling decisions, shut down resources, or change workload configurations. All adjustments remain under your control. The platform provides execution and cost information that can be used to guide manual optimization efforts. You can also explore related capabilities in Tracer/tune and Tracer/sweep: Tracer/tune
Optimize specific pipelines and workloads Tracer/sweep
Systemwide cloud waste detection
# Data Lake Integration Source: https://opensre.com/docs/use-cases/datalake Export pipeline telemetry and metrics to your data lake for custom analysis ### Data Lake Query all your pipeline runs to analyze sample processing turnaround times, costs per pipeline execution, error correlations and performance trends across projects. ### How It Works We ingest all your pipeline telemetry, metrics, and logs into a multi-tenant data lake. You can then use the Tracer web interface to query this data to analyze performance, identify bottlenecks, and optimize your pipelines. Identify underperforming steps, track which projects need attention, and monitor how efficiency changes over time. Tracer allows you to export all pipeline telemetry, metrics, and logs to your data lake for custom analysis and long-term retention. You can calculate cost per sample, measure run durations, and compare throughput across pipelines to identify optimization opportunities. Tracer also makes it easier to spot regressions, monitor efficiency trends, and inspect error types so teams can locate bottlenecks, prioritize fixes, and improve their workflows using real execution data. ## Key Analytics ### Pipeline Success Rate Analysis Track overall pipeline health with metrics such as success rate, total executions, and average turnaround times. Pipeline Success Rate Analysis ### Most Expensive Pipelines Identify your most expensive pipelines for any time window and surface the average cost per run to understand spend drivers. Most Expensive Pipelines ### Pipeline Resource Utilization Spot pipelines that consistently underuse allocated compute, helping you pinpoint wasted capacity and optimize resource allocation. Pipelines Underutilising CPU ### Longest Tool Execution Surface the tools or steps that contribute the most to overall runtime, including the exact commands, so you know where to focus optimization efforts. Longest Tool Execution # Accessing Logs Source: https://opensre.com/docs/use-cases/logging Comprehensive logging and observability for scientific pipelines ## Access the logs by going into your run details Tracer automatically captures logs from every tool, task, and environment, so you can debug issues, review performance, or audit results without any manual setup. ## How Logs Work: Logs are collected through the Tracer Agent and sent to Tracer's backend log capture service. The Tracer Agent reads logs from the local file system and transfers them to our internal services which process and index them for later analysis. # Metrics Source: https://opensre.com/docs/use-cases/metrics Track and analyze comprehensive pipeline metrics with Tracer ## Overview Tracer automatically captures detailed metrics for every process in your pipeline, providing deep visibility into resource usage, performance, and execution patterns without requiring any code changes. ## Key Metrics Tracked ### Resource Utilization * CPU Usage: Per-process utilization, saturation levels, and hotspots * Memory Usage: Allocation, peak consumption, and memory efficiency * I/O Ops: Disk reads/writes, throughput, and bottleneck detection * Network Activity: Bandwidth usage, transfer volume, and congestion * GPU Utilization: Compute load and VRAM usage (when applicable) ### Performance Metrics * Execution Time: Wall-clock, CPU, and wait times per task * Throughput: Data processed per unit time (e.g., samples/hour) * Latency: Startup delays, scheduling lag, and queueing overhead * Parallelization: Concurrency levels and contention across workers ### Cost Metrics * Compute Costs: Per-task and per-pipeline cost attribution * Resource Costs: Storage, network, and transfer charges * Efficiency: Cost per sample, cost per GB processed * Waste Detection: Idle resources, over-provisioning, and inefficiencies ## Real-Time Monitoring Inspect live metrics as your pipeline runs, with immediate visibility into performance, utilization, and potential bottlenecks. ## Historical Analysis * Analyze past runs to uncover long-term patterns and optimization opportunities: * Performance Trends: How execution times shift across versions * Utilization Patterns: Persistent over/under-use of compute or memory * Cost Evolution: Impact of configuration changes on total spend * Regression Detection: Automatic alerts for performance degradation # Right Sizing Source: https://opensre.com/docs/use-cases/right-sizing Optimize resource allocation based on actual usage patterns Right-Sizing in Tracer optimizes scientific compute by using actual CPU and RAM usage from previous runs rather than static rules or guesswork. After each workflow execution, Tracer analyzes per-step utilization and compares it with current cloud instance pricing to recommend the most cost-efficient instance for the next run. Rightsizing Tracer uses statistical models to predict RAM requirements and account for variability, minimizing memory-related failures that AWS’s native tools cannot anticipate. The system detects idle CPU, excess memory headroom, and under-provisioned configurations, then maps each workflow to an instance type that fits both performance and cost. As more runs complete, recommendations become increasingly accurate, lowering compute spend, reducing over-provisioning, and preventing failures caused by mis-sized infrastructure. # Vercel Source: https://opensre.com/docs/vercel Connect Vercel so OpenSRE can check deployment status and runtime logs during investigations OpenSRE queries Vercel to retrieve deployment status and runtime logs — helping identify whether a recent deployment caused an incident or surface errors from serverless function executions. ## Prerequisites * Vercel account with project access * Vercel API token ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup ``` Select **Vercel** when prompted and provide your API token. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} VERCEL_API_TOKEN=your-vercel-api-token VERCEL_TEAM_ID=team_xxxxxxxxx # optional, for team accounts ``` | Variable | Default | Description | | ------------------ | ------- | -------------------------------------- | | `VERCEL_API_TOKEN` | — | **Required.** Vercel API token | | `VERCEL_TEAM_ID` | — | Team ID for team/organization accounts | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "vercel-prod", "service": "vercel", "status": "active", "credentials": { "api_token": "your-vercel-api-token", "team_id": "team_xxxxxxxxx" } } ] } ``` ## Creating a Vercel API token 1. In Vercel, go to **Account Settings** → **Tokens** 2. Click **Create Token** 3. Give it a name (e.g., `opensre`) and set the scope to **Full Account** or a specific team 4. Copy the token For team accounts, find your Team ID in **Team Settings** → **General** → **Team ID**. ## Verify ```bash theme={null} opensre integrations verify vercel ``` Expected output: ``` Service: vercel Status: passed Detail: Connected to Vercel API and listed 5 project(s) ``` ## Troubleshooting | Symptom | Fix | | ---------------------- | ------------------------------------------------------------- | | **401 Unauthorized** | Check that the API token is valid and not expired | | **403 Forbidden** | The token may not have access to the team — check token scope | | **Projects not found** | Ensure `VERCEL_TEAM_ID` is set if using a team account | ## Security best practices * Create a **dedicated read-only token** for OpenSRE — do not reuse deployment tokens. * Store the token in `.env`, not in source code. * Rotate tokens periodically. # VictoriaLogs Source: https://opensre.com/docs/victoria_logs Connect VictoriaLogs so OpenSRE can pull structured log evidence during investigations OpenSRE queries [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) using LogsQL to retrieve structured log evidence — correlating recent application errors, request anomalies, and stream-level events with the alert under investigation. ## Prerequisites * A reachable VictoriaLogs instance (e.g. `http://vmlogs:9428`) * LogsQL knowledge for any custom queries you want OpenSRE to run ## Setup ### Option 1: Environment variables Add to your `.env`: ```bash theme={null} VICTORIA_LOGS_URL=http://vmlogs.monitoring.svc:9428 # Optional — only set this if your deployment is multi-tenant. # Single-tenant clusters should leave it unset; OpenSRE will not send # the AccountID header at all. # VICTORIA_LOGS_TENANT_ID=acme ``` | Variable | Default | Description | | ------------------------- | --------- | -------------------------------------------------------------------- | | `VICTORIA_LOGS_URL` | — | **Required.** Base URL of your VictoriaLogs instance | | `VICTORIA_LOGS_TENANT_ID` | *(unset)* | Optional. Sent as the `AccountID` header on multi-tenant deployments | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "victoria-logs-prod", "service": "victoria_logs", "status": "active", "credentials": { "base_url": "http://vmlogs.monitoring.svc:9428" } } ] } ``` ## Verify ```bash theme={null} opensre integrations verify victoria_logs ``` Expected output: ``` Service: victoria_logs Status: passed Detail: Connected to VictoriaLogs at http://vmlogs.monitoring.svc:9428. ``` ## How it works in investigations When VictoriaLogs is configured, the `victoria_logs_query` tool becomes available to the investigation agent. It runs LogsQL queries against `/select/logsql/query` and returns structured rows — defaulting to a wildcard match over the past hour, but the agent narrows the query based on alert context (service, level, trace ID, time window). Typical agent uses: * Pull recent error-level logs for the affected service to surface stack traces * Filter by request ID or trace ID to follow a single failing transaction across services * Compare log volume in the incident window against a known-good baseline to spot regressions ## Troubleshooting | Symptom | Fix | | --------------------------------- | --------------------------------------------------------------------------------------------------------------- | | **Status: missing** | Set `VICTORIA_LOGS_URL` or add a `victoria_logs` entry to your integrations store | | **Connection refused** | Verify the URL is reachable from this host; check firewall rules | | **404 on `/select/logsql/query`** | Confirm you are pointing at VictoriaLogs (not VictoriaMetrics for time-series); they expose different endpoints | | **Empty rows** | Normal when no logs match — widen the time range with a longer `start` (e.g. `-24h`) or relax the LogsQL filter | # Work Management Source: https://opensre.com/docs/work-management Track human tasks, choose what to do next, and schedule reminders or check-ins. ## Commands at a glance | Command | What it does | | -------------------------- | ---------------------------------------------- | | `/work` | List open work items | | `/work add ` | Add a durable task | | `/work done <id-or-title>` | Mark one or more tasks completed | | `/work next` | Rank open tasks by priority, due date, and age | | `/work path` | Print the local work-item store path | | `opensre work ...` | Script the same workflow from the terminal | Use `/tasks` for OpenSRE runtime jobs. Use `/work` for human todos, hackathon work, reminders, and follow-ups. ## Natural language You can ask normally: ```text theme={null} Add task "finish Slack reminder smoke test" for the hackathon, priority high. ``` ```text theme={null} Give me an overview of all tasks that still need to be completed. ``` ```text theme={null} What should I focus on next? ``` ```text theme={null} Mark the docs and CLI tasks as completed. ``` The agent stores these as work items and uses the same store when ranking or listing them later. ## Reminders and check-ins One-off reminders: ```bash theme={null} opensre work add "ping owners about demo blockers" \ --project hackathon --priority high \ --remind-at 2026-07-30T09:30 \ --provider slack --chat-id C0123ABCD ``` Recurring check-ins: ```bash theme={null} opensre work schedule-checkin \ --cron "0 9 * * 1-5" \ --tz Europe/London \ --target slack:C0123ABCD \ --target telegram:-100123456 \ --project hackathon ``` Slack and Telegram gateway turns can default reminders and recurring check-ins to the current chat. Terminal commands require either `--provider` and `--chat-id`, or one or more `--target` flags. Use repeated `--target provider:chat_id` flags to send the same reminder or check-in to more than one service. Slack webhook delivery may use `--target slack` without a channel id; Telegram, Discord, and Rocket.Chat need explicit chat/channel ids. ## Storage Work items are stored locally as JSON under `~/.opensre/work_items/` by default. Set `OPENSRE_WORK_ITEMS_DIR` to choose another folder. Existing entries from the old Slack captured-task JSONL store are imported once into the new work-item store. Recurring check-ins are stored in OpenSRE's scheduler store and run whenever the gateway scheduler or `opensre cron start` is active. # X (Twitter) MCP Source: https://opensre.com/docs/x-mcp Connect X's official MCP server so OpenSRE can search, read, and post on X during investigations OpenSRE connects to X's official [Model Context Protocol (MCP)](https://github.com/xdevplatform/xmcp) server, exposing X's API — tweet search, timelines, user profiles, likes, retweets, bookmarks, and posting — as tools the agent can call while investigating an incident. Unlike PostHog or Sentry's always-on hosted MCP servers, XMCP is designed to run **locally**: you clone the [xmcp repo](https://github.com/xdevplatform/xmcp), supply your own X API credentials, and run the server yourself. OpenSRE connects to that local endpoint (or a tunneled URL if you need remote access, e.g. via ngrok). ## Tools | Tool | What it does | | -------------- | --------------------------------------------------------------------------------- | | `list_x_tools` | List the tools the connected X MCP server exposes (compact, filterable) | | `call_x_tool` | Call a named X MCP tool (e.g. search tweets, inspect a timeline, look up a tweet) | The agent typically calls `list_x_tools` first to discover what is available, then `call_x_tool` with the chosen tool name and arguments. Pass `name_filter` (space- or comma-separated terms, e.g. `"search tweet"`) to narrow a large tool list, and `include_schema=true` on a narrowed list to fetch the full input schema for the tool you intend to call. ## Prerequisites * A running instance of [xmcp](https://github.com/xdevplatform/xmcp) with your own X API credentials configured (`X_BEARER_TOKEN`, and OAuth1/OAuth2 credentials if you need write access such as posting) * Network access from OpenSRE to the xmcp server's URL (local by default; tunnel it if OpenSRE runs elsewhere) <Info> XMCP authenticates to the X API using its own environment (`X_BEARER_TOKEN`, OAuth credentials) at startup. OpenSRE does not need to know your X API credentials for `streamable-http`/`sse` connections to an already-running server — it only needs the server's URL. An optional auth token can be set if your endpoint sits behind an authenticating tunnel or proxy. </Info> ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup x_mcp opensre integrations verify x_mcp ``` You'll be prompted for the xmcp server URL (defaults to `http://127.0.0.1:8000/mcp`) and, optionally, an auth token if the endpoint is tunneled behind an authenticating proxy. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} X_MCP_MODE=streamable-http X_MCP_URL=http://127.0.0.1:8000/mcp X_MCP_AUTH_TOKEN= # optional, only for a tunneled/proxied endpoint ``` | Variable | Default | Description | | ------------------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `X_MCP_URL` | `http://127.0.0.1:8000/mcp` | X MCP server URL (local or tunneled) | | `X_MCP_MODE` | `streamable-http` | Transport: `streamable-http`, `sse`, or `stdio` | | `X_MCP_AUTH_TOKEN` | — | Optional bearer token, only needed if the endpoint requires client auth | | `X_MCP_COMMAND` | — | Command to launch the xmcp server directly (`stdio` mode only) | | `X_MCP_ARGS` | — | Arguments for the local xmcp command (`stdio` mode only) | | `X_BEARER_TOKEN` | — | X API bearer token forwarded to the server when OpenSRE launches it (`stdio` mode only); not sent over the MCP transport | To have OpenSRE launch the local xmcp server itself instead of connecting to one you already started, use `stdio` mode: ```bash theme={null} X_MCP_MODE=stdio X_MCP_COMMAND=python X_MCP_ARGS=server.py X_BEARER_TOKEN=your_x_api_bearer_token ``` ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "x-mcp-local", "service": "x_mcp", "status": "active", "credentials": { "url": "http://127.0.0.1:8000/mcp", "mode": "streamable-http" } } ] } ``` ## Verify ```bash theme={null} opensre integrations verify x_mcp ``` A successful check connects to the MCP server and reports how many tools it discovered. If it fails, confirm the xmcp server is running and reachable at the configured URL, and that its own X API credentials (`X_BEARER_TOKEN`) are valid.