# ARCHITECTURE Source: https://opensre.com/docs/ARCHITECTURE # OpenSRE architecture How the OpenSRE codebase is structured: the eight first-party packages, what each is responsible for, and which may depend on which. These dependency rules are CI-enforced (`make check-imports`), so they are real invariants rather than aspirations. ## The layer stack The packages sit in five tiers. **Higher tiers may import lower tiers; a lower tier may never import a higher one.** Packages on the same tier are peers — the last column says whether peers may import each other. | Tier | Packages | May import | Must never import | Peer rule | | - | - | - | - | - | | 1 (top) | `surfaces`, `gateway` | `bootstrap`, `tools`, `integrations`, `core`, `infrastructure`, `config` | — | Independent peers; must not import each other, except three tracked CLI/REPL gateway-command edges listed in `.importlinter.strict`. | | 2 | `bootstrap` | `tools`, `integrations`, `core`, `infrastructure`, `config` | `surfaces`, `gateway` | Composition root. The only package that may import `tools` **and** `integrations` together, because wiring needs both. | | 3 | `tools` | `core`, `infrastructure`, `config` | `surfaces`, `gateway`, `bootstrap` | Peer of `integrations`: must not import it. Existing edges are listed as debt in `.importlinter.strict`. | | 3 | `integrations` | `core`, `infrastructure`, `config` | `tools`, `surfaces`, `gateway`, `bootstrap` | Peer of `tools`: must not import it. | | 4 | `core`, `infrastructure` | `config` | `surfaces`, `gateway`, `bootstrap`, `tools`, `integrations` | Siblings: **may** cross-import each other. | | 5 (bottom) | `config` | — (nothing first-party) | everything above | Independent — imports no other first-party package. | The shortcut: **dependencies point downward only.** A surface can reach all the way down; `config` can reach nothing. The single deliberate exception is `core ⟷ infrastructure`, a mutually-dependent pair by design (see below). ```mermaid theme={null} flowchart TD subgraph T1["Tier 1 — hosts"] SURFACES[surfaces] GATEWAY[gateway] end subgraph T2["Tier 2 — composition root"] BOOT[bootstrap] end subgraph T3["Tier 3 — capability"] TOOLS[tools] INTEGRATIONS[integrations] end subgraph T4["Tier 4 — runtime + infrastructure"] CORE[core] INFRA[infrastructure] end subgraph T5["Tier 5 — config"] CONFIG[config] end SURFACES --> BOOT GATEWAY --> BOOT SURFACES --> TOOLS SURFACES --> INTEGRATIONS GATEWAY --> TOOLS GATEWAY --> INTEGRATIONS BOOT --> TOOLS BOOT --> INTEGRATIONS TOOLS --> CORE TOOLS --> INFRA INTEGRATIONS --> CORE INTEGRATIONS --> INFRA CORE <--> INFRA CORE --> CONFIG INFRA --> CONFIG ``` The arrows show edges between **adjacent** tiers to keep the diagram readable. The actual rule is broader: a tier may import **any** tier below it, not only the one directly beneath — so a surface may import `config` directly, and a tool may import `infrastructure`. Refer to the "May import" column above for the complete set of allowed edges. ## The layers in detail ### Tier 1 — `surfaces` and `gateway` The entry points a human or an external system talks to. Nothing first-party may import from here, so a surface can be added or removed without touching the layers below it. * **`surfaces/`** — one folder per UI/client: `surfaces/cli` (the stateless `opensre ` runner), `surfaces/interactive_shell` (the stateful REPL), and `surfaces/shared` for code two or more surfaces use. A surface owns its own I/O, prompts, and presentation, and composes lower layers to do the actual work. The two terminal surfaces are peers and do not import each other (`tests/shared/test_surface_border.py` pins both directions at zero); what both need lives in `surfaces/shared` — `terminal/` (output tracking, tables, prompts, banner, health and feedback rendering), `llm_setup/`, `error_handling/` — or a lower layer. `surfaces/entrypoint.py` is the `opensre` console script: it hands the CLI a `CliHost` (how to open the shell, how to run the gateway attached), so composition happens in one place above both. Slack is not a surface: its inbound transport lives in `gateway/transports/slack`, outbound delivery in `integrations/slack`. * **`gateway/`** — the standalone messaging gateway for inbound chat platforms (`gateway/transports/{telegram,slack,discord,buzz}`), plus the FastAPI `gateway/web/` surface (health, alert intake) and the shared per-turn machinery in `gateway/core/{session,storage,middleware}` (`middleware/` holds the inbound-decision, identity, and approval steps every transport composes). A peer of `surfaces`, not a child. ### Tier 2 — `bootstrap` The **composition root** for process and harness wiring. Hosts (`surfaces`, `gateway`) cannot import each other, and `tools` / `integrations` are peers that must not import each other — yet registering harness adapters needs both capability packages. `bootstrap/` is the only package allowed to import `tools` **and** `integrations` together. Every entrypoint boots through `configure_process()` from `bootstrap/process.py`: a profile (CLI, gateway, web, scheduler worker, embedded) selects which boot steps run, and the step order is fixed by this package — profiles cannot invent a different sequence. Adapter and scheduler-runner registration lives in `bootstrap/adapters.py` and nowhere else; surfaces and gateway used to keep peer copies — do not reintroduce them. Host-owned concerns stay out of this package: CLI/gateway logging, Rich product ports, CLI’s update-tolerant Sentry init. ### Tier 3 — `tools` and `integrations` The capability layer — "do a thing against the outside world" — split by responsibility: * **`integrations/`** — the boundary for **user config and external clients**: per-vendor config normalization, verification (`verifier.py`), API clients (`client.py`), the store/catalog that resolves credentials, and integration-local helpers. One folder per vendor (`integrations/datadog`, `integrations/grafana`, `integrations/github`, …) plus cross-cutting pieces like `integrations/llm_cli`. * **`tools/`** — the **vendor-neutral agent-callable** boundary: the registry, interactive-shell tool framework, `tools/system/` for capabilities with no vendor in their domain purpose, and the narrow `tools/cross_vendor/` bucket for capabilities that genuinely span vendors. A tool tied to one vendor belongs under `integrations//tools/`, beside that vendor's client and configuration. See [tool-placement-policy.md](tool-placement-policy.md). `tools` and `integrations` are peers, so neither may import the other in new code. The exceptions in `.importlinter.strict` are tracked migration debt, not precedent: they cover the existing cross-vendor tools and a few incidental credential/client edges. Single-vendor tools avoid the peer edge entirely by living inside their owning `integrations//` package. `bootstrap` discovers and registers both tool families into the `core.tool` registry. Do **not** reintroduce top-level `vendors/` or `services/` packages. ### Tier 4 — `core` and `infrastructure` The shared runtime and cross-cutting services the capability layer is built on. * **`core/`** — the provider-agnostic agent runtime: the think → call tools → observe loop (`core.agent.Agent`), agent state (`core/state`) and context-budget enforcement (`core/context_budget.py`), tool contracts, schema, registry ports, execution, and error reporting (`core/tool`), tool authoring helpers (`core/tool_framework`), shared LLM clients (`core/llm`), agent-harness session handling (`core/agent_harness`), and pure domain rules (`core/domain`). * **`infrastructure/`** — cross-cutting services with no agent logic of their own: guardrails, masking, sandbox, analytics, auth, notifications, observability, scheduler, deployment, and the shared **turn host** (`turn_host`) — the single `TurnRunner` the interactive shell and every gateway transport run turns through, so the two entry paths cannot drift. Deploy-time assets live under `infrastructure/deployment/` (EC2/packaging Python tooling plus the `cloudflare_install_proxy` edge worker); these are not imported by the app. These two are the one bidirectional pair by design: `core` reaches `infrastructure` for guardrails, masking, observability, and evidence/log compaction, while `infrastructure` reaches back into `core` for the shared state and session types (`core.state`, `core.agent_harness.session`). Splitting them into separate tiers would forbid that edge, so they share a tier as siblings. ### Tier 5 — `config` The floor: shared constants, prompts, and UI theme. Everything above may read from `config`, but `config` imports no other first-party package — keeping it a leaf means constants can be imported anywhere without dragging runtime along. ## Cross-layer flows Two worked examples showing runtime control flow and how results return to the host. Dashed registry-dispatch arrows are callbacks installed by `bootstrap`; they do not represent static imports from `core` into capability packages. ### A chat turn from the interactive shell ```mermaid theme={null} flowchart LR A["surfaces/interactive_shell\n user question"] --> B["core/agent_harness\n session + turn orchestration"] B --> C["core/agent\n ReAct loop, context budget, LLM"] C --> D["core/tool\n runtime registry"] D -. "dispatch registered callable" .-> E["tools\n vendor-neutral tools"] D -. "dispatch registered callable" .-> F["integrations/vendor/tools\n vendor-owned tools + clients"] C --> G["infrastructure\n guardrails, masking, observability"] ``` 1. The shell (or a gateway transport) hands the message to the shared agent harness in `core/agent_harness` — the surface never runs agent logic itself. 2. The harness runs the ReAct agent (`core/agent`): think → call tools → observe, under the context budget. 3. `bootstrap` has already registered vendor-neutral tools from `tools` and vendor-owned tools from `integrations//tools` behind the registry's `core` contract. Dispatch through that registry is a runtime callback, not a forbidden `core` import of either capability package. 4. Vendor-owned tools use the client and resolved credentials in their own integration package; `infrastructure` supplies guardrails, masking, and observability around calls. 5. The answer flows back up to the surface, which owns how it is presented or delivered. ### An inbound gateway message ```mermaid theme={null} flowchart LR A["gateway/transports\n inbound chat message"] --> B["gateway/core session + storage\n resolve conversation state"] B --> C["core registry\n dispatch the requested capability"] C --> D["infrastructure\n notifications, observability"] ``` `gateway` receives a message, resolves session state from its own storage, then uses the same boot-registered tier-3 capabilities as a surface — without ever importing `surfaces`, since the two are independent tier-1 peers. ## Related docs * [`AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/AGENTS.md) — repo map and per-area "files to touch" guides. # DEVELOPMENT Source: https://opensre.com/docs/DEVELOPMENT # Development guide Contributor-focused workflows: local setup details stay in [SETUP.md](https://github.com/Tracer-Cloud/opensre/blob/main/SETUP.md) at the repo root (Windows, troubleshooting, MCP). ## Clone and install ```bash theme={null} git clone https://github.com/Tracer-Cloud/opensre.git cd opensre make install ``` [`make install`](https://github.com/Tracer-Cloud/opensre/blob/main/Makefile) runs `uv sync --frozen --extra dev` and the analytics install helper. Use **`uv run opensre …`** from the repo root so you always hit this checkout’s `.venv`, not another `opensre` on your `PATH`. ```bash theme={null} opensre onboard uv run opensre # open the interactive shell ``` ## Local checks and PR validation Follow [CI.md](https://github.com/Tracer-Cloud/opensre/blob/main/CI.md) for local validation, required PR checks, and review follow-through. ## Interactive shell action policy Action-planner behavior, postprocessing transforms, compatibility seams, and the rule-extension checklist are documented in [`docs/interactive-shell-action-policy.md`](https://github.com/Tracer-Cloud/opensre/blob/main/docs/interactive-shell-action-policy.md). ## Package architecture The eight first-party packages, the five-tier layering (which package may import which), the folder diagram, per-layer responsibilities, and cross-layer flows are documented in [`docs/ARCHITECTURE.md`](ARCHITECTURE.md). ## Tool registry — surface-scoped, lazy loading Loading every vendor tool at startup was slow. A static index (`tools/registry_index.py`) reads tool metadata by scanning the source, without importing executors, so a turn loads only the tools it needs. * `get_registered_tools(surface)` imports only that surface's tool modules. * `get_tool_descriptors(surface)` returns metadata with no executor import. * `load_tool(descriptor)` imports the executor, only when a tool runs. Adding a vendor tool is a `@tool`/`BaseTool` module; the index finds it and no other vendor is imported. `tests/tools/test_registry_index.py` checks the index matches the imported registry exactly, so they cannot drift. ## VS Code dev container The dev container is defined under [`.devcontainer/`](https://github.com/Tracer-Cloud/opensre/tree/main/.devcontainer). It builds from [`.devcontainer/Dockerfile`](https://github.com/Tracer-Cloud/opensre/blob/main/.devcontainer/Dockerfile) (Python **3.13**), then **`postCreateCommand`** creates `.venv-devcontainer` and runs **`pip install -e '.[dev]'`** (not `uv`). Docker Desktop, OrbStack, Colima, or another compatible runtime must be available on the host. ## Deployment Full deployment instructions, prerequisites, and environment variable reference: **[DEPLOYMENT.md](https://github.com/Tracer-Cloud/opensre/blob/main/DEPLOYMENT.md)** Quick reference: | Path | Commands | | - | - | | Gateway (AMI + systemd — gateway only) | `make build-gateway-image` → `make deploy-gateway` / `make destroy-gateway` | | Hosted (Railway / ECS / Vercel) | Deploy with repo `Dockerfile`; set `LLM_PROVIDER` + API key | ### Hosted runtime (Railway / ECS / Vercel) 1. Deploy the repository `Dockerfile` and select the runtime with `MODE` (`web`, `gateway`, or `scheduler`). The container installs the current main-channel build on start. Restart it, or run `opensre update -y` inside it, to pick up a newer build. Rebuild the image when the toolchain (Git, GitHub CLI, Node, Codex) changes. 2. Set `LLM_PROVIDER` and the matching API key (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` — see [`.env.example`](https://github.com/Tracer-Cloud/opensre/blob/main/.env.example)). 3. Set `DATABASE_URL` when gateway records must be shared through Postgres (required for Slack Events API unless the single-replica local-dedup escape hatch is enabled). 4. Use a shared `OPENSRE_HOME` mount when multiple processes need the same scheduler tasks. If `OPENSRE_CONTEXT_ROOT` is set, make that root durable and shared for organization-bound sessions too. 5. Add integration and storage env vars your deployment needs. Minimal LLM env: ```bash theme={null} export LLM_PROVIDER=anthropic export ANTHROPIC_API_KEY=... ``` For Railway: add Postgres and set `DATABASE_URL` only when the gateway needs shared records; Redis is not part of the gateway persistence path. Set `OPENSRE_DEPLOYMENT_METHOD=railway` for telemetry labeling. ## Telemetry and privacy `opensre` ships with two telemetry stacks, both opt-out: * **Product analytics** — lifecycle and usage events sent through `app.opensre.com` (commands used, success/failure, rough runtime, CLI/Python/OS/arch, and limited command metadata). * **Sentry** — crashes and errors (stack traces, environment, release). Events are tagged with `entrypoint`, `opensre.runtime`, and `deployment_method`. Sensitive headers, paths, and secret-shaped keys are scrubbed before send. Product events also carry `execution_environment` (`local`, `ci`, `container`, or `ci_container`), `is_ci`, `is_container`, and `container_runtime`. Use these first-party fields to exclude automated environments from product funnels. A random install ID is stored under `~/.opensre/anonymous_id`. Telemetry is off in GitHub Actions and pytest. ### Kill-switch matrix | Env var | Product analytics | Sentry | | - | - | - | | `OPENSRE_NO_TELEMETRY=1` | disabled | disabled | | `DO_NOT_TRACK=1` | disabled | disabled | | `OPENSRE_ANALYTICS_DISABLED=1` | disabled | unaffected | | `OPENSRE_SENTRY_DISABLED=1` | unaffected | disabled | | `OPENSRE_SENTRY_LOGGING_DISABLED=1` | unaffected | disables `logger.error`/`logger.exception` forwarding to Sentry; `capture_exception` unaffected | Full opt-out: ```bash theme={null} export OPENSRE_NO_TELEMETRY=1 ``` ### Sentry DSN Self-hosted users can set `SENTRY_DSN` to their project; unset uses the bundled default. `SENTRY_DSN=` (empty) drops events in `before_send`. ### Deployment tagging Set `OPENSRE_DEPLOYMENT_METHOD` to `railway`, `ec2`, `vercel`, or `local` (default `local`) to label Sentry events. ### Local product event log By default, outbound product event properties are also appended to `~/.opensre/analytics_events.txt` (rotates at 1000 lines). Disable: ```bash theme={null} export OPENSRE_ANALYTICS_LOG_EVENTS=0 ``` Metadata events do not collect alert contents, file contents, hostnames, credentials, or raw CLI arguments. The separately documented `$ai_generation` event contains redacted prompt and response text. # NAMING Source: https://opensre.com/docs/NAMING # Naming conventions for `core/` A small, enforceable vocabulary so file and type names say what they are. The goal is that a reader can tell a data type from a process, a mutable state from a frozen view, and a package's purpose from its name alone. ## Glossary (one meaning per term) | Term | Means | Example | | - | - | - | | **State** | Mutable agent/session facts that evolve during a run | `AgentState`, harness port `SessionState` | | **Storage** / **Repo** | Durable persistence backends (not the in-memory turn port) | `SessionStore`, `SessionRepo`, `JsonlSessionStore` | | **Snapshot** | A frozen view captured at a boundary (turn start, run start) | `TurnSnapshot` | | **RunInput** / **RunResult** | The input to and output from one `Agent.run()` boundary | `AgentRunInput`, `AgentRunResult` | | **Resources** | Handles passed into tool executors for one call | `ToolCallResources` | | **Budget** | An LLM token/window policy — not application state | `enforce_token_budget` | | **Host** | The callback contract an algorithm drives (a `Protocol`) | `LoopHost` | ## Module naming: `{domain}_{role}.py` Name a file for the concept it holds, not with a generic bucket word. ``` core/agent/ agent.py # the Agent facade react_loop.py # ReactLoop + run_react_loop (the algorithm) loop_host.py # LoopHost (the callback contract) run_io.py # AgentRunInput, AgentRunResult (the run boundary's I/O) mixins.py # the reusable *Mixin behaviors provider_hooks.py # ProviderHookDelegate ``` ## Type naming * **Mixins** carry a `Mixin` suffix — they cannot stand alone (they assume fields/methods the host provides). `EventEmitterMixin`, `ToolFilterMixin`, `SteeringMixin`. * **Protocols** are named by their role, not with a `Protocol` suffix — matches the stdlib (`Iterable`, `SupportsRead`) and `agent_harness/ports.py` (`OutputSink`, `SessionState`). `LoopHost`, not `LoopHostProtocol`. * **Do not prefix a type with its own package name.** Inside `core/agent/`, a class is `EventEmitterMixin`, not `AgentEventEmitter` — the namespace already says "agent." ## Anti-patterns (do not add in new code) * `context.py` at `core/` or `core/agent/` root — "context" is overloaded across the repo. Name the concept (`run_io.py`, `turn_snapshot.py`). * `models.py` when the file holds only run I/O — too vague. Say what the models are (`run_io.py`). * `*Context` without a domain prefix when another `*Context` already exists. * A package whose only child is a single sub-package — collapse the wrapper. * Calling the mutable harness session port `SessionStore` — that word means durable persistence. The port is `SessionState` (`InMemorySessionState` for headless); JSONL durability stays `SessionStore` / `SessionRepo`. * Module-level mutable globals for “current session” — pass `SessionState` explicitly (see local notes: no-globals design). ## Imports Use fully qualified paths in code; keep short mental labels for docs. | Mental label | Import | | - | - | | ReAct run I/O | `from core.agent.run_io import AgentRunInput, AgentRunResult` | | ReAct loop | `from core.agent.react_loop import run_react_loop` | | Loop callback contract | `from core.agent.loop_host import LoopHost` | | The agent primitive | `from core.agent import Agent` | | Harness turn snapshot | `from core.agent_harness.turns.turn_snapshot import TurnSnapshot` | Re-export from a package `__init__.py` only for its single canonical symbol (`Agent`), not everything — avoid `from core.agent import *`-style ambiguity. # Adding tools and integrations Source: https://opensre.com/docs/adding-tools-and-integrations # Adding Tools & Integrations — Definition of Done Use this checklist whenever you add or materially change: * a tool — under `integrations//tools/` for a single-vendor tool, or `tools/system/` / `tools/cross_vendor/` for a cross-cutting one (see [tool-placement-policy.md](tool-placement-policy.md)) * an integration under `integrations//` — its config, client, verifier, and tools This is the detailed definition of done; use it with [AGENTS.md](https://github.com/Tracer-Cloud/opensre/blob/main/AGENTS.md) and [CI.md](https://github.com/Tracer-Cloud/opensre/blob/main/CI.md). ## 1. Tool checklist ### Files usually involved * `integrations//tools/_tool/__init__.py` — the tool package (most common path: the tool belongs to a vendor integration) * `tools/system//` or `tools/cross_vendor//` — only when the tool is not vendor-specific (e.g. `tools/system/sre_guidance_tool/`) * `integrations//client.py` — reuse a dedicated integration API client instead of inlining requests * `core/tool_framework/utils/` — shared helper code reused across vendors * `docs/.mdx` — user-facing usage, parameters, examples * `tests/tools/test_.py` — behavior and regression coverage Tools are registry-discovered from **both** `tools/` and `integrations//tools/`, so placement is about ownership, not discovery — see [tool-placement-policy.md](tool-placement-policy.md). Wherever a tool lives, it calls integration-local clients/helpers rather than inlining transport, and never lives in a top-level `vendors/` or `services/` package. Tool packages must be substantive production modules — no empty or discovery-only `__init__.py`, no thin wrapper that only satisfies registry import. Any tool with validation, credential/parameter resolution, transport/client calls, output normalization, or error handling should split those concerns into focused sibling files (`tool.py`, `models.py`, `validation.py`, `delivery.py`/`client.py`, `results.py`), leaving `__init__.py` as a small registry entrypoint that imports the public tool object. ### Contract and implementation * [ ] Pick the simplest shape that fits (`@tool(...)` for lightweight tools, a richer class only when needed) * [ ] `__init__.py` is a small registry entrypoint; non-trivial tools use sibling modules for implementation concerns * [ ] Metadata is complete and accurate: `name`, `description`, `source`, `surfaces`, `requires`, and any `use_cases` / `outputs` / `retrieval_controls` * [ ] `input_schema` matches the actual runtime arguments and required fields * [ ] `is_available` returns `True` only when the tool can genuinely run * [ ] `extract_params` maps resolved integration state into tool args correctly * [ ] Validation, credential/parameter resolution, transport/client calls, and result formatting are separated so each can be tested independently * [ ] Reusable transport or integration-specific parsing lives in `integrations//` or `core/tool_framework/utils/`, not copied into the tool body * [ ] Failure responses have a stable, agent-friendly shape; expected external failures (missing config, auth, rate limit, upstream 4xx/5xx) return structured errors rather than raising — unexpected exceptions use the global `BaseTool` wrapper intentionally or are migrated with telemetry coverage * [ ] Output is normalized enough for the agent/LLM to consume reliably * [ ] Secrets never leak through `extract_params`, return values, logs, or traceable tool-call kwargs; secret/PII output is run through `infrastructure/safety/masking/` before return * [ ] External side effects declare `side_effect_level`, `requires_approval`, and `approval_reason` where appropriate * [ ] A pure read may set `concurrency_safe=True` so it runs in parallel with other reads from the same model response: it must declare `side_effect_level` `NONE` or `READ_ONLY`, keep the default action role, need no approval or runtime context, and share no unlocked client or cache between calls * [ ] To appear in both chat and action turns, set `surfaces=(ToolSurface.CHAT, ToolSurface.ACTION)` ### Live payload parsing If the tool parses API, MCP, log, or webhook payloads: * [ ] Validate against the real or documented upstream response shape, not only idealized mocks * [ ] Handle alternate field names used in live payloads * [ ] Handle missing or partial fields without returning unusable output * [ ] Preserve important context when truncating, tailing, paginating, or flattening data * [ ] Upstream 429 / 5xx responses return a clear, agent-friendly error rather than raising * [ ] Add at least one regression test using a realistic fixture payload Common failure modes to consider: grouped + ungrouped log content; nested/foldered resources; paginated responses; `hasMore` / cursor mismatches; content-vs-pointer shapes (`logs_content` vs `logs_url`-style payloads). ### Skill guidance (optional) A tool can carry workflow guidance the model reads on every call by shipping a `SKILL.md`. The guidance is **appended to the tool's `description`** under a `Workflow guidance:` heading — it is permanent schema text, not a side channel. All guidance targeting one tool is combined and truncated at **2400 characters** (`tools/registry_skill_guidance.py`), so budget it like description text: the longer the guidance, the more of every request it consumes. Skill guidance and a harness playbook are **independent and composable** — a tool may have both. Skill guidance rewrites one tool's description with call rules (parameters, refusals, what the tool owns); a harness playbook under `core/agent_harness/prompts/skills/` describes the multi-step workflow around it. The GitHub tools (`github_cli`, `ci_fix`, `security_fix`) each ship a `SKILL.md`; `ci_fix` and `security_fix` also have a workflow card. Give the two different `name`s (`operating-github-ci-fixer` vs `repair-github-ci`) and keep tool-level facts in the tool card only. Placement rules: `core/agent_harness/prompts/skills/AGENTS.md`. **When to add (two independent axes — evaluate both):** | Axis | Add when | Skip when | | - | - | - | | **Tool `SKILL.md`** (this section) | ≥2 of: misuse is expensive; not obvious from the tool `description`; reused across many turns | One clear tool; thin/rare vendor; tip that fits in `description` | | **Harness playbook** (`core/agent_harness/prompts/skills/*/SKILL.md` + `skill_view`) | Multi-step WHEN / DO NOT, sibling carve-outs, same-turn vs next-turn, or a report template | Single-tool tip with no cross-turn flow | **Neither** is the default for most of \~70 `tools/` packages. Missing `SKILL.md` is usually correct. **Never** add one-per-vendor stubs, or one skill per observability vendor when the failure mode is shared (query hygiene is one class, not Datadog + Grafana + CloudWatch copies). Harness authoring template: `core/agent_harness/prompts/skills/_template/SKILL_TEMPLATE.md`. **File.** A `SKILL.md` with YAML frontmatter and a markdown body: ```yaml theme={null} --- name: operating-github-cli # required — gerund-first kebab-case, ≤ 64 chars description: > # required — ≤ 1024 chars; the model reads this to decide relevance One or two sentences: when to reach for these tools. tools: # required — the registered tool name(s) this guidance applies to - github_cli disable-model-invocation: false # optional — set true to suppress attachment entirely --- # Body — markdown workflow guidance shown after the tool's own description. ``` **Register it, or it is silently ignored.** The loader reads only files it is told about: * [ ] **Explicit:** add the file's path to `_skill_guidance_files()` in `tools/registry_skill_guidance.py`. A `SKILL.md` that exists on disk but is absent from that tuple is **never loaded** — unlisted and missing files are skipped with no diagnostic, the same trap as forgetting a `docs.json` entry. * [ ] Or place it under `tools/system/python_execution_tool/skills/*/SKILL.md`, which is discovered automatically. **Check the registry-load logs** — the loader warns and skips rather than failing the build: * `unknown_tool` — a name under `tools:` matches no registered tool; that target is dropped. * `invalid_metadata` — missing `name`/`description`/`tools`, an over-long `name`/`description`, or a `name` that is not lowercase kebab-case; the whole skill is skipped. * `parse_failed` — malformed frontmatter YAML. ## 2. Integration checklist ### Files usually involved * `integrations//__init__.py` — package facade: a docstring, plus re-exports of the public API when callers need them (see the `__init__.py` rule in [AGENTS.md](https://github.com/Tracer-Cloud/opensre/blob/main/AGENTS.md)) * `integrations//config.py` — config model, `classify()`, validators, selectors, normalization helpers * `integrations//client.py` — a dedicated API client, when the integration makes direct remote calls * `integrations//verifier.py` — local verification logic * `integrations//tools/_tool/` — the vendor's agent-callable tools (see §1) * `integrations/catalog.py` — resolve the integration into the shared runtime config * `integrations/verify.py` — wire the local verification path * `docs/.mdx` — user-facing setup, usage, verification * `tests/integrations/test_.py`, plus `tests/tools/` or `tests/e2e/` where tools or scenarios exercise it `integrations//` owns everything about one vendor — config, resolution, clients, verifiers, helpers, **and its tools**. Only vendor-less (`tools/system/`) and cross-vendor (`tools/cross_vendor/`) tools live under top-level `tools/`. ### Examples from the repo * Datadog: `integrations/datadog/` (with `integrations/datadog/tools/`), `integrations/catalog.py`, tests under `tests/integrations/datadog/` and `tests/tools/test_datadog_*.py`. * Grafana: `integrations/grafana/` (with `integrations/grafana/tools/`), `integrations/catalog.py`, `surfaces/cli/wizard/local_grafana_stack/`, tests under `tests/integrations/grafana/` and `tests/tools/test_grafana_*.py`. * Bitbucket: `integrations/bitbucket/` shows the facade layout — a one-line `__init__.py` beside `config.py`, `client.py`, `verifier.py`, and `tools/`. ### Core completeness * [ ] Config, normalization, validators, and `classify()` are in place under `integrations//config.py`, leaving `__init__.py` a facade * [ ] Catalog resolution / env loading is wired correctly * [ ] Verification path is wired in `integrations/verify.py` and adapters/registry as needed * [ ] Integration-local client added under `integrations//client.py` (only if it makes direct remote calls) * [ ] Tool layer is wired and stable * [ ] CLI setup flow is updated if the integration is user-configurable locally * [ ] `opensre integrations setup ` parity is added, or intentionally documented as out of scope * [ ] New required env vars / credentials are added to `.env.example` (never `.env`) * [ ] Sensitive credentials follow the [Credential resolution](#credential-resolution) contract below * [ ] `make verify-integrations` passes ### Credential resolution Secret env names and non-secret config follow different write/read paths. Keep this contract when adding or changing an integration. | Surface | Write (wizard / setup) | Read (runtime) | | - | - | - | | Integration store (`~/.opensre/integrations.json`) | Always on setup | First (preferred) | | Local credentials file (`~/.opensre/credentials.json`) | Secrets via `sync_env_secret` | Via `resolve_env_credential` when env is unset | | `.env` / process env | Public config and secrets (`sync_env_values` / `sync_env_secret`) | Plain `os.getenv` for that tier; `resolve_env_credential` reads env first | **Hard rules for new code** * Never use bare `os.getenv` for a secret env name (`*_TOKEN`, `*_KEY`, `*_PASSWORD`, `*_SECRET`, connection strings, and similar). Use `resolve_env_credential` from `config.llm_credentials` (env first, then the credentials file). * Webhook / `*_URL` values are **never** written to the credentials file (wizard routes them to store/`.env`, not `sync_env_secret`). Read with store → plain `os.getenv` only. Webhook URLs often **embed** a secret token — treat them like passwords for logging/masking. * Leave `load_env_integration_services` plain-env-only (startup-safe; no credentials-file read at boot). * Store still wins in `resolve_effective` / merge — env/credentials-file is the fallback tier only. * Tools receive credentials through `extract_params` (resolved integration state), never their own env reads. At execution, keys listed in the tool's `injected_params` override model-supplied values, so the verified source wins even when the model passes a token. GitHub tools require an attested app grant and preserve explicitly empty tokens without environment fallback. * Set `OPENSRE_DISABLE_KEYRING=1` to skip local-file reads/writes (env and store still work). Canonical helpers: `resolve_env_credential` (env → credentials file), `sync_env_secret` / `save_credential` (secret writes to the credentials file and `.env`), `sync_env_values` (`.env` keys, including secrets). ## 3. Discovery and edge cases For tools that list, search, or inspect resources: * [ ] Folder/nested resource layouts are considered where the upstream supports them * [ ] Large result sets are capped or paginated intentionally * [ ] Partial fetches are surfaced clearly (`truncated`, `fetch_error`, etc.) * [ ] Time/order-sensitive results preserve causal ordering where it matters ## 4. Docs and tests ### Docs * [ ] Ship or update a `docs/` page/section in the same PR (new tool, CLI command, agent behavior, or integration; and whenever a tool's API/schema or an integration's setup changes) * [ ] Any new `docs/` page is registered in `docs/docs.json` (without the `.mdx` suffix) so Mintlify navigation shows it ### Tests * [ ] Unit tests for config/normalization * [ ] Tool contract tests, or equivalent schema/metadata coverage * [ ] A registry/discovery test proves the tool is visible on the expected surface(s) * [ ] Runtime behavior tests for success and failure paths * [ ] At least one realistic fixture for live-payload parsing when external payloads are involved * [ ] A test proves the agent can discover or invoke the tool through the normal runtime path * [ ] `tests/integrations/` updated when integration wiring changes Green tests are not enough if they only cover idealized mocks. ### Final gate (new integrations) Everything above is complete, **and**: * [ ] Screenshot or demo GIF showing the integration working end-to-end * [ ] E2E test added * [ ] CI checks pass (see [CI.md](https://github.com/Tracer-Cloud/opensre/blob/main/CI.md)) ## 5. Reviewer focus Before opening or approving the PR, confirm the items most often missed are handled **explicitly**: tool placement (§1), live-payload robustness (§1), onboarding/setup/docs parity (§2 and §4), pagination/truncation/partial-response behavior (§3), and tests that cover realistic payloads and usefulness to the agent — not only happy-path mocks (§4). Follow [CI.md](https://github.com/Tracer-Cloud/opensre/blob/main/CI.md) for the mandatory pre-push commands. # Agents Source: https://opensre.com/docs/agents Set up and maintain the OpenSRE CLI (install, setup, verify, first run) # OpenSRE CLI Setup Set up or maintain `opensre` with minimal friction. If you are running this prompt, follow the steps below and set up OpenSRE for the user so they can ask about their infrastructure from the interactive shell. Do not clone the OpenSRE repo unless the user asked to contribute. The published CLI is enough. ## Step 1: Install or upgrade the CLI Check whether the CLI exists: ```bash theme={null} command -v opensre opensre --version ``` If `opensre` is missing, install it. On macOS or Linux: ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc ``` On Windows, in PowerShell: ```powershell theme={null} & ([scriptblock]::Create((irm https://install.opensre.com/install.ps1))) -dc ``` Windows details: [Windows](https://www.opensre.com/docs/environments/windows-local). No sudo on macOS/Linux in the usual case. If the installer cannot use a writable directory already on `PATH`, it puts the binary in `~/.local/bin` and prints the command to add that directory. Apply that PATH update (or open a new terminal), then re-check `command -v opensre`. When `opensre` is already present, upgrade it: ```bash theme={null} opensre update ``` Confirm it runs: ```bash theme={null} opensre --help ``` ## Step 2: Get started First launch is interactive and needs a TTY. Do not try to fake the sign-in gate. Run `opensre` and prompt the user when it asks for input: ```bash theme={null} opensre ``` The browser opens the OpenSRE sign-up page. The user can create an account or sign in using email or another enabled provider. This activates the hosted model; do not ask the user for a separate LLM key during first run. When sign-in finishes, it opens the interactive shell. Add tools later with `opensre integrations setup ` when the agent should query them. To connect one tool later: ```bash theme={null} opensre integrations setup ``` Replace `` with a slug such as `datadog`, `grafana`, or `slack`. If sign-in cannot validate the webapp account, run `opensre account status`. The interactive shell intentionally stays closed while the session is expired, revoked, incomplete, or unreachable. ## Step 3: Verify ```bash theme={null} opensre integrations verify ``` One tool only: ```bash theme={null} opensre integrations verify datadog ``` If verify fails, check the printed error. Common causes: missing or expired credential, wrong URL, or a skipped setup step. Re-run `opensre integrations setup ` for that tool. ## Step 4: Suggest a first run They are already in the interactive shell after Step 2. Ask the user to describe an incident or ask a question in plain language, or use `/help`. To start it again later: ```bash theme={null} opensre ``` ## Gotchas * **`opensre: command not found`** — new terminal, or add the bin directory the installer printed (often `~/.local/bin` on macOS/Linux). * **Sign-in blocks or looks hung** — it is waiting for webapp browser authentication. Show the user the URL and prompt; do not kill it. * **The shell keeps returning to sign-in** — run `opensre account status`; the account must be active before the shell or hosted model starts. * **Only connected tools are queried** — the agent cannot pull Datadog data if Datadog was never set up. Run `opensre integrations verify` before a production run. Human docs: [https://opensre.com/docs/install](https://opensre.com/docs/install) and [https://opensre.com/docs/quickstart](https://opensre.com/docs/quickstart) # Keep CI green with an agent Source: https://opensre.com/docs/cicd/ci-repair-agent A recurring reliability report, a bounded repair loop for one pull request, or the managed service that repairs CI for your organization. Three ways to let OpenSRE keep working on your CI after the first report. All of them start from the demo picker (`opensre`, or `/demo` in the shell) or from a plain request. ## The reliability agent Demo option **B**, "Set up an agent that improves CI/CD reliability over time", turns the [CI reliability analysis](/docs/cicd/cicd-analytics-demo) into a recurring check for one repository. ```text theme={null} set up an agent that improves CI/CD reliability for Tracer-Cloud/opensre ``` * It analyzes the repository now, then schedules the report for weekdays at 08:00 local time. Later runs cover the last 7 days. * Reports land in the shell inbox: `/loops messages`. The agent never posts to Slack or another chat channel. * Manage it with `/loops list`, `/loops run `, `/loops stop ` and `/loops delete `. * The loop fires while the shell is open. `/loops service install` keeps it running after you close the terminal (launchd on macOS, systemd on Linux; Windows uses `opensre cron start --service` from Task Scheduler). ## The repair loop A repair loop watches one pull request, edits, tests and pushes fixes for its failing GitHub Actions checks, and stops on its own: | Rule | Value | | - | - | | Trigger | Every 30 seconds while active | | Attempts | At most three failed repair attempts | | Time budget | Ten minutes from scheduling | | Target | An open pull request whose branch lives in the same repository | Ask for it with the repository and the PR number: ```text theme={null} schedule a CI repair loop for Tracer-Cloud/opensre pull request 6054 ``` OpenSRE asks for approval before the loop starts, because it will push to the PR branch. A pull request that is closed, or opened from a fork, is refused before anything is scheduled, with the reason and what to choose instead. While it runs and after it finishes: ```text theme={null} /loops show the report: outcome, attempts, evidence links /loops stop stop an active repair what did the most recent CI repair run do ``` A finished run keeps its report and attempt records and removes its checkout. Asking for the same PR again while a loop is active returns that loop with its original deadline. > [!NOTE] > > Want to see the loop end to end without touching a real PR? Ask for the demo repair: OpenSRE creates a small private repository named `opensre-ci-repair-demo-` plus four letters or digits, opens a pull request with a failing check, and repairs it. The repository and its pull request remain afterwards. It needs a GitHub token that may create a private repository and a signed-in coding agent. ## The managed service Demo option **C**, "Run CI/CD repairs remotely", runs the same repair loop on your organization's hosted gateway instead of your machine, so it keeps running after the shell closes and uses the organization's GitHub credential rather than yours. What you need: * An organization on [app.opensre.com](https://app.opensre.com) with the managed gateway provisioned from [Agent Backend settings](https://app.opensre.com/settings/agent-backend). * The organization's GitHub token connected on the web app's Integrations page. The gateway picks up a changed token within a minute. Then, in the shell: ```text theme={null} Check whether my organization's hosted gateway is running. run the CI repair loop for Tracer-Cloud/opensre pull request 6054 on my org's hosted gateway what did the most recent CI repair run on the gateway do ``` The gateway asks for approval through the shell before a loop starts, and answers questions about its loops while a repair is running. See [Deployment](/docs/guides/deployment#opensre-cloud-managed-gateway) for starting and stopping the gateway. ## Related * [Fix GitHub CI](/docs/cicd/github-ci-fix) — a one-off repair from the shell, no loop * [CI/CD analytics demo](/docs/cicd/cicd-analytics-demo) — the report the reliability agent repeats * [Shell commands](/docs/getting-started/interactive-shell-commands) — everything `/loops` can do # CI/CD analytics demo Source: https://opensre.com/docs/cicd/cicd-analytics-demo See how reliable your CI is and how much developer time it costs, using real repositories on your machine. ## Overview On every interactive launch OpenSRE asks which demo to run. The recommended option analyzes a real repository from your machine and reports CI/CD reliability KPIs, including the developer time lost to unreliable CI. That analysis is read-only and takes a couple of minutes. The other choices set up a recurring CI/CD agent, show the upcoming managed service, or connect Slack. Press Escape to skip, or use the custom-answer row to type a different request. Run it again at any time with `/demo`, or by asking: ```text theme={null} run the CI/CD analytics demo analyze Tracer-Cloud/opensre CI/CD performance how much developer time does flaky CI cost us ``` ## Prerequisites * `git` on `PATH` * A GitHub connection in the OpenSRE app that can read Actions history * `opensre account login` on local machines to load app connections ## What happens The agent guides you through the selected demo in the shell. 1. **Scan.** It scans your home directory for git checkouts and paints a bar chart of the last 30 days of commits, your own contributions first. On macOS it leaves out Desktop, Documents and Downloads, which would open a permission prompt. To include one, start `opensre` from inside that folder. Outside this demo you can also ask for a folder directly, for example `scan ~/Documents for git repositories`. 2. **Pick a repository.** A menu lists your checkouts that have GitHub Actions workflows, then the open-source example, then a row to type your own. ![The repository scan: checkouts found and commit activity over the last 30 days](https://mintcdn.com/tracer/pusIxCBwQ1b8M9V-/images/onboarding-ci-analysis-scan.png?fit=max\&auto=format\&n=pusIxCBwQ1b8M9V-\&q=85\&s=aca72d88d9de331e140610873d9ccb36) 3. **Analyze.** It reads the repository's Actions history for the last 30 days and paints the report: one sentence on what waiting on CI cost, the key results, then a comparison table against langchain-ai/langchain and anomalyco/opencode. The benchmark columns ship with OpenSRE and name the day they were measured. Every analysis reads GitHub; nothing is answered from a cache. 4. **Next step.** A menu offers the CI reliability agent (it schedules the check for the repository you just analyzed, no second repository question), the Slack hand-off, or exit. ![The CI/CD reliability report for Tracer-Cloud/opensre with the comparison table](https://mintcdn.com/tracer/pusIxCBwQ1b8M9V-/images/onboarding-ci-analysis-report.png?fit=max\&auto=format\&n=pusIxCBwQ1b8M9V-\&q=85\&s=9cd0e8970caeefc6f9900ba86f71b648) Typing a request such as `analyze owner/repo CI/CD performance for the last 30 days` skips the scan and repository picker. ## The KPIs | KPI | Meaning | | - | - | | GitHub Actions executions | Completed runs on the default branch and on pull requests in the window | | PR-triggered failed workflows and failure rate | Pull request runs that failed, including runs that only passed after a re-run | | CI reliability failures | Failures where the identical commit passed later, by re-run or a fresh run. CI, not the code, was at fault | | Source-code failures | Failures that passed only after the branch moved to a new commit | | Developer downtime from unreliable CI | The working-hours part of the wait below, attributed to each PR's author, with the heaviest-hit developers in hours per week | | Wall-clock wait on merged PRs | Per merged PR: how much later its commits went green than they would have had CI run normally. For each commit with a CI reliability failure, the expected green time is the first queued run plus the slowest workflow's normal duration; the actual green time is when its last workflow passed | | Normal duration | Per workflow, the median duration of first-attempt passing runs | | Default-branch red time | Hours during which the latest commit on the default branch had a failing push-triggered check, with mean time to recovery; the red time is attributed per workflow so one failing scan is not mistaken for a broken build | Only PRs that were merged count toward blocked time, so the number reflects delays that sat on the path to a real merge. Workflows that fail and recover in parallel on the same commit count once, a commit's wait ends when the next commit of the same PR is pushed or the PR merges, and a commit whose workflows never all passed is left out. A later commit that triggered no workflow is invisible; the merge time still bounds the wait. When GitHub omits a PR number, the run belongs to the first later merge of that head repository and branch. When it attaches several, the run belongs to the merged PR whose lifetime contains the run. Developer downtime is the part of each wait that falls inside working hours: Monday to Friday, 09:00 to 18:00 in this machine's timezone, printed with the figure. A re-run that happened over a weekend adds wall-clock wait but no downtime. Each blocked PR is attributed to its author, so the report can say how many hours a week the heaviest-hit developers lost. The report shows the calculation, not only its result. It states the formula (expected green = first run queued + normal duration; blocked = actually green minus expected green, in working hours), then a table of the most blocked PRs with their author, queue time, normal duration, expected and actual green times, and the working-hours and wall-clock wait, then the sum across all blocked PRs, the division by the developers affected, the per-week figure, and the heaviest-hit developers. The wall-clock total across PRs not merged yet is shown when it differs. ## The CI reliability agent Pick "Set up an agent that improves CI/CD reliability over time" from the startup menu to analyze the repository now, then schedule that report as a recurring check. The demo scans your machine, asks which repository to watch, reads GitHub Actions (a token is required), and creates the loop (weekdays at 08:00 local time by default). Choosing this option after an analytics report schedules that repository without asking you to pick it again. Every later run analyzes the last 7 days; use `/loops run ` to run it now. * Reports land in this shell's inbox: `/loops messages`. The loop never posts to Slack or any chat channel, even when those integrations are configured. * Manage it with `/loops list`, `/loops stop `, and `/loops delete `. * Each run is deterministic: no model turn sits between the schedule and the report, and the raw figures are saved as JSON next to the report path named at the end of every delivery, under `~/.opensre/ci_reliability_reports/`. * The loop fires while the shell is open. To keep it running when the shell is closed, run `/loops service install`; it installs a per-user service (launchd on macOS, systemd on Linux) that starts at login. `/loops service remove` deletes it. Windows is not supported yet; run `opensre cron start --service` from Task Scheduler instead. * Running the demo again for the same repository reuses the existing loop. ## Limits * Windows up to 90 days. Each scope reads at most 2,000 runs; the report says so when a cap was hit. * Merged PR detection covers the 500 most recently updated closed PRs. * Scheduled and manually dispatched workflows count toward executions but not toward default-branch red time. ## Troubleshooting | Symptom | Fix | | - | - | | **GitHub connection missing or rejected** | Connect or reconnect GitHub in the OpenSRE app, then select **I’ve connected GitHub — continue** | | **No repositories offered** | None of the checkouts under your home directory has a GitHub origin and a `.github/workflows` folder; pick the example repository | | **The picker did not appear** | It needs an interactive TTY. Run `uv run opensre` in a terminal, or type `/demo` | # Fix GitHub CI Source: https://opensre.com/docs/cicd/github-ci-fix Use OpenSRE to inspect failing GitHub Actions checks on a pull request or branch, apply a fix, and push a repair branch. ## Overview OpenSRE can inspect failing GitHub Actions checks on a pull request or named branch, apply a code fix, commit it, push it, and wait for the new checks. It can also fix failing CI on a branch with no PR — say "fix the CI on main" and OpenSRE inspects the failing workflow runs on that branch's head commit, applies a fix in a linked worktree, and pushes a fresh repair branch. Say "fix the CI on main and push straight to main" to push the fix onto the branch itself instead. > [!WARNING] > > This is mutating: OpenSRE checks out the branch, edits files, commits, and pushes. At the default autonomy level (`/auto high`) those steps run without prompts; set `/auto med` to be asked before each mutating step. See [Trust mode and `/auto`](/docs/getting-started/interactive-shell-commands#trust-mode-auto-and-confirmations). ## Prerequisites * Git installed on the machine running OpenSRE * `gh` on `PATH` for checks and logs * GitHub connected in the OpenSRE app with repository write access and workflow/check read access * Nothing extra for the fix itself: OpenSRE uses an installed coding-agent CLI (Pi, Claude Code, Codex, or Cursor) when one is logged in, and otherwise its built-in coding agent on the LLM you already configured ## Setup There is no dedicated `integrations setup` target for this tool. Connect GitHub in the OpenSRE app with repository write and workflow/check read access. A named repository or PR URL uses an isolated checkout; an explicitly selected local checkout uses a linked worktree so your branch and uncommitted changes remain untouched. For requests about your current checkout, you can set the coding workspace: ```bash theme={null} export CODING_WORKSPACE=/path/to/repo # optional, for current-checkout requests # optional: # export CODING_AGENT=auto # pi | claude-code | codex | cursor | opensre # export CODING_MODEL=... ``` With `CODING_AGENT=auto` (the default) OpenSRE picks the first ready CLI and falls back to its built-in agent (`opensre`). The built-in agent reads, searches, and edits files and runs commands (tests, linters, builds) only inside the repair checkout, with `CI=true` and without your OpenSRE or provider secrets in its environment, and never commits or pushes itself. `CODING_MODEL` applies to CLIs only; the built-in agent uses your configured LLM. ## Credentials | Need | Access | | - | - | | Read failing checks / logs | Workflow and check read via token + `gh` | | Push code fix | Contents write on the same repository | | Apply the fix | A logged-in coding agent CLI when auto-detected, else the built-in agent on your configured LLM | ## Tools | Tool | What it does | | - | - | | `fix_github_pr_ci` | Inspect failing Actions checks, apply a fix, commit, and push to the existing PR head branch or a fresh branch repair worktree | Related skill: `repair-github-ci`. Run `/demo` in the interactive shell for the guided CI/CD onboarding menu; its recommended first option is CI/CD analytics. ### Use it In the interactive shell: ```text theme={null} fix CI on https://github.com/Tracer-Cloud/opensre/pull/4597 and push fix failing checks on Tracer-Cloud/opensre#4597 fix CI on main in Tracer-Cloud/opensre and push the current PR CI is failing, fix and push fix the CI on main fix the CI on main in Tracer-Cloud/opensre-webapp and push straight to main ``` ### What happens 1. OpenSRE reads PR metadata or the latest failing Actions run on the target branch. 2. It fetches compact log excerpts for failing Actions jobs. 3. It creates an isolated checkout for a named repository, or verifies that your explicitly selected checkout matches and uses a linked worktree for branch repairs. 4. It refuses fork PRs because it only pushes to branches in the same repository. 5. After approval, it checks out the PR head branch or creates a fresh repair branch for a branch target such as `main`; a selected local checkout uses a temporary linked worktree. 6. If the PR branch is behind its base branch (whether or not GitHub already reports a conflict), it merges the base branch into the PR branch first, so the fix is made on top of the current base and a failure already fixed there needs no further change. Conflicts go to the coding agent, lockfiles are regenerated rather than hand-merged, and the merge is committed only when no conflict remains. 7. If files changed, it commits only the fix-run changes and pushes the PR branch or fresh repair branch. 8. It verifies the exact pushed commit, allows one minute for its checks to register, waits for applicable workflows, and reports whether they passed, failed, were superseded by another push, or did not finish within 15 minutes. If GitHub reports the pushed commit as conflicted (the base moved during the repair), it merges the base branch into the pushed branch once, pushes again, and verifies that commit instead. If no checks are failing, OpenSRE says so and does not push. The one exception is a PR that conflicts with its base branch: GitHub never starts checks for it, so OpenSRE treats bringing the base branch in as the fix. It does **not** open a new PR. Merged or closed PRs are refused — if the failure already landed on the base branch, ask for the branch itself ("fix the CI on main"). In branch mode OpenSRE reads the failing workflow runs on the branch's head commit, creates a fresh `opensre/ci-fix-*` repair branch in the isolated checkout (or a linked worktree when you selected a local checkout), pushes that branch, and waits for the pushed commit's own workflow runs. It does **not** push directly to `main` unless you ask for that. When you explicitly ask to push straight to the branch ("push straight to main", "commit directly to main"), OpenSRE makes the fix in a fresh clone of that branch, commits it there, pushes it without force, and verifies that commit's workflow runs. If `main` moved during the repair, nothing is pushed. If a branch protection rule requires a pull request, GitHub refuses the push and OpenSRE reports that; ask for a repair branch instead. ## Verify There is no dedicated `integrations verify` target. Confirm readiness by: 1. An authorized GitHub connection in the OpenSRE app 2. `gh` available on `PATH` 3. If supplying a checkout explicitly, its `origin` matches the PR repository 4. An LLM is configured for OpenSRE (the built-in agent uses it when no coding agent CLI is ready) If a worker stops after pushing, retry the repair to resume verification of the recorded commit. A concurrent update to the PR stops verification of the older commit. ## Troubleshooting | Symptom | Fix | | - | - | | **Fork PR refused** | Only same-repository PR head branches are supported | | **No push when checks are green** | Expected — OpenSRE does not push when nothing is failing | | **"blocked on N file(s) a person must decide"** | The coding agent could not resolve those merge conflicts. The merge was aborted; resolve the listed files by hand, push, and run the fix again. If the message starts with "Pushed a CI fix", the fix commit is already on the branch and only the base merge is missing. When the agent named a choice only a person can make, OpenSRE also comments it on the PR and skips that PR until new commits land on it | | **Approval prompts** | Appear at `/auto med` or lower; the default `/auto high` runs the repair without them | | **gh / checks unavailable** | Install GitHub CLI and ensure the token can read Actions checks | ## Security * Asks before mutating the working tree or pushing when the autonomy level is `/auto med` or lower. * Pushes only to the existing PR head branch in the same repository. * For branch targets, creates and pushes a fresh repair branch unless you explicitly ask to push straight to the branch; a direct push is never forced. * Store tokens in `.env` or your secret manager — not in source control. # Scan CI health across repositories Source: https://opensre.com/docs/cicd/github-ci-health-scan Find every open pull request and branch with failing CI across all your GitHub organizations and repositories in one call. ## Overview Ask OpenSRE "how many PRs and branches have failing CI across my organizations?" and it scans every repository your token can see, in parallel, and answers with the exact list: which PRs, which branches, which checks. The scan is read-only. It never re-runs, retries, or fixes anything; hand a result to [Fix GitHub CI](/docs/cicd/github-ci-fix) for that. ## Prerequisites * GitHub connected in the OpenSRE app via the [GitHub integration](/docs/integrations/code/github) * Token scopes: `repo` (private repositories and their checks) and `read:org` (to list organizations) A token without `read:org` still scans your own account and any owner you name explicitly. ## What it scans | Scope | Default | Option | | - | - | - | | Owners | Your account plus every organization you belong to | `owners: ["Tracer-Cloud", "davincios"]` | | Repositories | Not archived, pushed to in the last 365 days | `since_days: 0` scans everything; `visibility: private` narrows | | Heads per repository | Default branch head plus the 100 most recently updated open PRs | `include_all_branches: true` adds every branch head | A check counts as failing when its conclusion is `FAILURE`, `TIMED_OUT`, `ACTION_REQUIRED`, or `STARTUP_FAILURE`, or a commit status (Vercel, Aikido, and similar) is `FAILURE` or `ERROR`. Cancelled checks are listed separately and not counted; a cancelled run is usually superseded, not broken. Stale branches are the reason `include_all_branches` is off by default. On a real organization almost every failing branch was either a years-old ref or a duplicate of a failing PR; the PR list is the signal. ## Example turn ```text theme={null} you> how many PRs and branches currently have failing CI across all my orgs? opensre> Scanned 152 repositories under davincios, FlymbleTeam, netrunnerhq, Tracer-Cloud in 7.0s. 8 of 201 open PRs are failing CI and 2 default branches are red: PRs - Tracer-Cloud/opensre #6148 perf/windows-onedir-lifecycle-5935 (fork, draft) — windows test - Tracer-Cloud/tracer-client #583 dependabot/cargo/main/jsonwebtoken-10.1.0 — Build and Lint, Build and Test, ... - ... Default branches - Tracer-Cloud/nextflow-test-pipelines main — build-and-push, build-and-push-minimal - Tracer-Cloud/tracer-agent-archive main — Analyze (python), test-kubernetes, ... ``` ## Result fields | Field | Meaning | | - | - | | `failing_prs` | Open PRs whose head commit has a failing check: repo, number, branch, sha, fork/draft flags, check names and links | | `failing_default_branches` | Repositories whose default branch head is red | | `failing_branches` | Other branch heads with failures; only filled with `include_all_branches` | | `counts` | Totals of the three lists | | `repos_in_scope`, `repos_scanned` | Attempted versus successfully read; the difference is in `errors` | | `errors` | Repositories that could not be read, with the reason | | `coverage_notices` | Every cap that applied: skipped stale repositories, owners that could not be listed, repositories with more than 100 open PRs, heads with more than 1000 checks | | `elapsed_seconds` | Wall-clock time of the scan | | `rate_limit_cost`, `rate_limit_remaining` | GraphQL points spent and left this hour | ## Speed and rate limits Repositories are read ten per GraphQL request, eight requests in flight at once. 150 repositories with 200 open PRs finish in about 7 seconds; with every branch head included, about 10 seconds. `concurrency` (1 to 16) tunes the fan-out. A full scan spends a few hundred of GitHub's 5000 hourly GraphQL points. When the budget is gone GitHub answers every request with a rate-limit error and the tool reports that once instead of listing 150 unreadable repositories. Wait for the hour to reset or pass `owners` to narrow the scan. ## Gotchas * A PR from a fork shows as a failing PR but never as a failing branch; the branch lives in the fork. * A Vercel or Aikido "failure" is often a blocked preview deploy or a security gate rather than a broken build. The check name tells you which. * Only the 100 most recently updated open PRs per repository are scanned. A repository past that cap gets a coverage notice. * A misspelled or inaccessible owner in `owners` is reported in `coverage_notices`; the other owners are still scanned. # CI/CD with OpenSRE Source: https://opensre.com/docs/cicd/overview Measure how reliable your CI is, fix failing checks, and keep pull requests green with a scheduled agent. OpenSRE reads your GitHub Actions history, tells you what unreliable CI costs your team, fixes failing checks on a pull request, and can keep doing that on a schedule. Everything runs against the repositories you already have, on your machine or on your organization's hosted gateway. ## Pick a starting point > **[Measure CI reliability](/docs/cicd/cicd-analytics-demo)** > > A 30-day report for one repository: failure rate, CI-caused versus code-caused failures, developer hours lost, red time on the default branch. The recommended first demo. > **[Fix failing CI on a PR](/docs/cicd/github-ci-fix)** > > Read the failing checks, apply a fix with a coding agent, push to the PR branch, and wait for the new checks. > **[Keep CI green with an agent](/docs/cicd/ci-repair-agent)** > > A recurring reliability report, a bounded repair loop for one PR, or the managed service that runs repairs for your organization. > **[Find every red PR](/docs/cicd/github-ci-health-scan)** > > One read-only scan across all repositories your token can see: which PRs and branches are failing, and on which checks. ## Before you begin **1. Install and sign in** Follow [Install](/docs/install). Signing in activates the hosted model; no LLM key is needed. **2. Connect GitHub** The demo offers this when GitHub is not connected yet. To do it yourself: ```bash theme={null} opensre integrations setup github ``` Choose **Authorize in browser**, or paste a token that can read Actions history. Fixing and pushing needs Contents and Pull requests write access on the repository. See [GitHub](/docs/integrations/code/github). **3. Pick the demo, or ask directly** Start `opensre` and choose **A** in the demo picker, or type any of the prompts below. ## Your first CI/CD tasks Type these in the shell, with your own repository: ```text theme={null} analyze Tracer-Cloud/opensre CI/CD performance for the last 30 days how much developer time does flaky CI cost us how many PRs and branches currently have failing CI across all my orgs? fix failing checks on Tracer-Cloud/opensre#4597 set up an agent that improves CI/CD reliability for Tracer-Cloud/opensre ``` > [!TIP] > > Analysis and the health scan are read-only. Fixing and repair loops push commits; they refuse pull requests opened from forks and never push directly to a protected branch such as `main`. ## What each capability does | Capability | Reads | Writes | Runs where | | - | - | - | - | | [CI reliability analysis](/docs/cicd/cicd-analytics-demo) | 30 days of Actions runs for one repository | A report in the shell | Your machine | | [Health scan](/docs/cicd/github-ci-health-scan) | Open PRs and branch heads across your organizations | Nothing | Your machine | | [Fix a PR](/docs/cicd/github-ci-fix) | Failing checks and logs of one PR or branch | A commit on the PR branch, or a fresh repair branch | Your machine | | [Reliability agent](/docs/cicd/ci-repair-agent#the-reliability-agent) | The last 7 days, every weekday morning | A report in the shell inbox | Your machine, as a scheduled loop | | [Repair loop](/docs/cicd/ci-repair-agent#the-repair-loop) | One PR's failing checks | Up to three repair pushes within ten minutes | Your machine, or the hosted gateway | ## From a script or a CI job `opensre ask` runs one turn and exits. The read-only tasks, analysis and the health scan, run headless as they are: ```bash theme={null} opensre --json ask "how many PRs currently have failing CI in Tracer-Cloud?" ``` Fixing a PR or scheduling an agent uses tools that need approval, and a headless run has nobody to approve. Authorize the tool the task uses, for that run only: ```bash theme={null} # fix a pull request opensre ask --allowed-tool fix_github_pr_ci "fix failing checks on Tracer-Cloud/opensre#4597" # schedule the reliability agent opensre ask --allowed-tool schedule_ci_reliability_loop "set up an agent that improves CI/CD reliability for Tracer-Cloud/opensre" # schedule a bounded repair loop for one pull request opensre ask --allowed-tool schedule_ci_repair_loop "schedule a CI repair loop for Tracer-Cloud/opensre pull request 6054" ``` Each `--allowed-tool` covers one tool; a task that needs two takes the flag twice. A machine that is not signed in needs `LLM_PROVIDER` and a provider key plus a GitHub token in the environment. See [Headless CLI](/docs/guides/headless-cli) for `--allowed-tool`, `--json` output and exit codes. ## Where the results go * Analysis reports print in the shell; the scheduled agent also saves the raw figures under `~/.opensre/ci_reliability_reports/`. * A repair pushes to the pull request's own branch and reports the commit and the check results. `/loops show ` prints a finished repair's report with its evidence links. * The banner's **CI/CD fixes** counter shows how many distinct repairs pushed checks that passed. # Daily Update — 2026-04-02 Source: https://opensre.com/docs/daily-updates/2026-04-02 OpenSRE engineering daily update for 2026-04-02 (Europe/London) Thanks to everyone who contributed today: Ceren Camkiran, Ebrahim Sameh, edgarmb14, Kalio, Luke Gimza, Tan Wee Joe, Vaibhav Upreti, and vincenthus 🙏🚀 ## Main updates shipped * Multi-provider LLM support -- added OpenRouter, Gemini, and NVIDIA NIM providers with automatic detection via `LLM_PROVIDER` env var, alongside fixes to make chat nodes (`general_node`, `chat_agent_node`) fully provider-aware for OpenAI and Anthropic with provider-keyed caches and lazy imports. * Pydantic config rollout -- replaced ad-hoc config handling with a Pydantic-based strict config across the app, touching CLI, integrations, state, and tooling layers. * Simulation engine overhaul -- large rework of the synthetic test engine (+1470/-735) covering root-cause diagnosis nodes, the mock Grafana backend, and RDS Postgres scenario fixtures including new replication-lag and connection-exhaustion scenarios. * Expanded RDS Postgres metric fixtures -- split the `000-healthy` CloudWatch metrics pack into per-metric JSON files (CPUUtilization, ReplicaLag, WriteIOPS, etc.) and updated the scenario loader and fixture generator accordingly. * Grafana local demo moved to wizard CLI -- Grafana stack seeding and live-demo entrypoints now live under `app/cli/wizard/`, Make targets updated, and the obsolete bundled local RCA demo path removed. * CLI improvements -- better error message for unknown test IDs (hints `opensre tests list`), simplified JSON output handling in `args.py`, and a readability refactor in `investigate.py` with lint fixes. * Tooling hygiene -- added `.editorconfig` (4-space Python, 100-char line length, matching `ruff.toml`), added `.ruff_cache/` to `.gitignore`, removed unused `mypy.ini` kubernetes stubs, and disabled PostHog analytics firing in CI environments. * Release workflow & CI -- added `release.yml` GitHub Actions workflow and improved the wizard onboarding flow to avoid re-prompting for provider selection. ## Source pull requests * [#239](https://github.com/Tracer-Cloud/opensre/pull/239) improved simulation engine (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/nodes/chat.py`, `app/nodes/root_cause_diagnosis/claim_validator.py`, `app/nodes/root_cause_diagnosis/evidence_checker.py`, `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tools/DataDogLogsTool/__init__.py`, `tools/base.py`, `mypy.ini`, `tests/synthetic/mock_grafana_backend/backend.py`, `tests/synthetic/mock_grafana_backend/formatters.py`, `tests/synthetic/mock_grafana_backend/selective_backend.py`, and 86 more) * [#240](https://github.com/Tracer-Cloud/opensre/pull/240) improved gating (author: vincenthus; contributors: vincenthus; labels: *none*; files: `tools/TracerFailedRunTool/__init__.py`, `tools/TracerRunTool/__init__.py`, `tools/TracerTasksTool/__init__.py`) * [#242](https://github.com/Tracer-Cloud/opensre/pull/242) chore: release (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `.github/workflows/release.yml`) * [#245](https://github.com/Tracer-Cloud/opensre/pull/245) push metrics outpack with expanded data for 000 (author: vincenthus; contributors: vincenthus; labels: *none*; files: `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_BinLogDiskUsage.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_CPUUtilization.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_CommitLatency.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_CommitThroughput.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_DatabaseConnections.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_DiskQueueDepth.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_FreeStorageSpace.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_FreeableMemory.json`, `tests/synthetic/rds_postgres/000-healthy/aws_cloudwatch_metrics_MaximumUsedTransactionIDs.json`, and 14 more) * [#247](https://github.com/Tracer-Cloud/opensre/pull/247) Improve CLI error message for unknown test id (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/cli/__main__.py`) * [#249](https://github.com/Tracer-Cloud/opensre/pull/249) simplified JSON output handling in CLI helper (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/cli/args.py`) * [#253](https://github.com/Tracer-Cloud/opensre/pull/253) remove unused sys import to pass lint (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/cli/args.py`) * [#248](https://github.com/Tracer-Cloud/opensre/pull/248) Small refactor to improve readability and reduce duplication. No beha… (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/cli/investigate.py`) * [#252](https://github.com/Tracer-Cloud/opensre/pull/252) resolve lint issue (remove whitespace) (author: Ceren Camkiran; contributors: Ceren Camkiran and Tan Wee Joe; labels: *none*; files: `app/cli/investigate.py`) * [#233](https://github.com/Tracer-Cloud/opensre/pull/233) feature: Support OpenRouter, Gemini, and NVIDIA LLMs (author: Tan Wee Joe; contributors: Tan Wee Joe; labels: `enhancement`; files: `.env.example`, `SETUP.md`, `app/cli/wizard/config.py`, `app/cli/wizard/flow_test.py`, `app/cli/wizard/validation.py`, `config/config.py`, `app/integrations/clients/llm_client.py`) * [#241](https://github.com/Tracer-Cloud/opensre/pull/241) fix: use OpenAI in chat when LLM\_PROVIDER=openai (author: Kalio; contributors: Kalio; labels: *none*; files: `app/integrations/clients/aws_sdk_client.py`, `app/nodes/chat.py`, `app/nodes/publish_findings/formatters/evidence.py`, `app/nodes/publish_findings/report_context.py`, `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tools/utils/data_validation.py`, `pyproject.toml`, `tests/nodes/test_chat.py`) * [#246](https://github.com/Tracer-Cloud/opensre/pull/246) Removed outbound\_telemetry directory after moving files to utils (author: edgarmb14; contributors: edgarmb14; labels: *none*; files: `app/outbound_telemetry/__init__.py`, `utils/__init__.py`, `utils/config.py`, `utils/config_test.py`) * [#255](https://github.com/Tracer-Cloud/opensre/pull/255) Fixes #214: Make chat provider-aware for OpenAI and Anthropic (author: Luke Gimza; contributors: Luke Gimza and vincenthus; labels: *none*; files: `app/nodes/chat.py`, `utils/cfg_helpers.py`, `utils/config_test.py`, `pyproject.toml`, `tests/conftest.py`, `tests/e2e/grafana_validation/env_requirements.py`, `tests/e2e/grafana_validation/grafana_validation.py`, `tests/e2e/grafana_validation/test_grafana_cloud_push.py`, `tests/e2e/upstream_prefect_ecs_fargate/infrastructure_sdk/deploy.py`, `tests/nodes/test_chat.py`, and 4 more) * [#276](https://github.com/Tracer-Cloud/opensre/pull/276) implemented pydantic config (author: vincenthus; contributors: vincenthus; labels: *none*; files: `.github/workflows/ci.yml`, `.github/workflows/issue-comment-to-slack.yml`, `Makefile`, `app/cli/investigate.py`, `app/cli/investigate_test.py`, `app/cli/wizard/integration_health.py`, `config/config.py`, `app/config_test.py`, `app/integrations/clients/datadog/client.py`, `app/integrations/clients/grafana/config.py`, and 15 more) * [#278](https://github.com/Tracer-Cloud/opensre/pull/278) fix: clean up chat model caches (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/nodes/chat.py`, `app/sentry.py`, `tests/nodes/test_chat.py`, `tests/nodes/test_chat_provider_awareness.py`) * [#279](https://github.com/Tracer-Cloud/opensre/pull/279) Refactor repeated string-list validation in synthetic schemas (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `tests/synthetic/schemas.py`) * [#280](https://github.com/Tracer-Cloud/opensre/pull/280) fix: disable analytics in CI pipelines (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/analytics/provider.py`, `app/analytics/provider_test.py`) * [#281](https://github.com/Tracer-Cloud/opensre/pull/281) docs: refresh OpenSRE positioning in README (author: vincenthus; contributors: vincenthus; labels: *none*; files: `README.md`) * [#282](https://github.com/Tracer-Cloud/opensre/pull/282) Add .editorconfig and clean up tooling config (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `.editorconfig`, `.gitignore`, `mypy.ini`) * [#251](https://github.com/Tracer-Cloud/opensre/pull/251) Bug: Move local Grafana demo to wizard CLI and update makefile (author: Tan Wee Joe; contributors: Tan Wee Joe and Vaibhav Upreti; labels: `bug`; files: `Makefile`, `SETUP.md`, `app/cli/tests/discover.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/grafana_seed.py`, `docs/local-grafana-live.md`, `docs/local-rca-demo.md`) * [#283](https://github.com/Tracer-Cloud/opensre/pull/283) chore: improve ci remove space select logic (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `app/cli/wizard/flow.py`, `app/cli/wizard/flow_test.py`, `app/cli/wizard/prompts.py`, `app/cli/wizard/prompts_test.py`) * [#284](https://github.com/Tracer-Cloud/opensre/pull/284) chore: do not ask for provider again in opensre onboard (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `app/cli/wizard/flow.py`, `app/cli/wizard/flow_test.py`) ## Generation metadata * Generator version: `opensre 0.1.0` * Fallback summary used: `no` * UTC window: `2026-04-01T23:00:00+00:00` to `2026-04-02T23:00:00+00:00` # Daily Update — 2026-04-03 Source: https://opensre.com/docs/daily-updates/2026-04-03 OpenSRE engineering daily update for 2026-04-03 (Europe/London) Thanks to everyone who contributed today: Andrew Van Dyke, Ceren Camkiran, Ebrahim Sameh, Shoaib Ansari, shoaib050326, Vaibhav Upreti, venturevd, vincenthus, and Yeoreum Song 🙏🚀 ## Main updates shipped * New `opensre update` CLI command -- hits the GitHub releases API, compares versions, and runs a pip upgrade or prints a re-install command for frozen binaries; supports `--check` and `--yes` flags. * New `opensre health` CLI command -- reuses the existing integration verification flow to print a compact local health summary including environment and integration store path. * MCP server added -- a minimal `opensre_mcp.py` exposes a single `run_rca` tool wrapping the existing investigation workflow, enabling use from MCP clients such as Copilot and Claude Desktop, with tests covering happy path and malformed input. * Google Docs integration shipped -- adds a `GoogleDocsIntegrationConfig`, a full client (create, insert, share, retrieve, validate), and a `GoogleDocsCreateReportTool` that generates structured postmortem reports with Executive Summary, Root Cause, Timeline, and Remediation sections. * Tool-decorating refactor merged -- large rework (\~+4927/-2606) touching integrations clients (Coralogix, Honeycomb), wizard flow, integration health, alert templates, and node processing. * Mintlify docs moved into repo -- `docs-mintlify/` directory added (\~12 k lines) including comparisons, quickstart pages, and CI spellcheck configuration; Vale vocabulary wired up via `.vale.ini` to eliminate 320 false-positive spellcheck findings. * CodeQL quality alerts resolved -- consolidated duplicate `import`/`import from` statements in `jwt_auth.py`, `flow.py`, and a test file, and removed unused JS variables in `radar-chart.js` and `toc-actions.js`. ## Source pull requests * [#285](https://github.com/Tracer-Cloud/opensre/pull/285) Add opensre update command (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `.editorconfig`, `.gitignore`, `README.md`, `app/analytics/cli.py`, `app/analytics/events.py`, `app/cli/__main__.py`, `app/cli/update.py`, `app/cli/update_test.py`, `mypy.ini`) * [#292](https://github.com/Tracer-Cloud/opensre/pull/292) Feature/tool decorating (author: vincenthus; contributors: vincenthus; labels: *none*; files: `.env.example`, `README.md`, `app/cli/__main__.py`, `app/cli/alert_templates.py`, `app/cli/args.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/flow_test.py`, `app/cli/wizard/integration_health.py`, `app/cli/wizard/integration_health_test.py`, `app/integrations/cli.py`, and 79 more) * [#296](https://github.com/Tracer-Cloud/opensre/pull/296) Add opensre health command (author: Yeoreum Song; contributors: Yeoreum Song; labels: *none*; files: `app/cli/__main__.py`, `app/cli/health_test.py`) * [#294](https://github.com/Tracer-Cloud/opensre/pull/294) Create opensre\_mcp.py (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/integrations/opensre_mcp.py`) * [#295](https://github.com/Tracer-Cloud/opensre/pull/295) Create opensre\_mcp\_test.py (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/integrations/opensre_mcp_test.py`) * [#293](https://github.com/Tracer-Cloud/opensre/pull/293) feat: Add Google Docs integration for shareable incident postmortems (author: shoaib050326; contributors: Shoaib Ansari, shoaib050326, and vincenthus; labels: *none*; files: `README.md`, `SETUP.md`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/clients/google_docs/__init__.py`, `app/integrations/clients/google_docs/client.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/state.py`, `tools/GoogleDocsCreateReportTool/__init__.py`, and 2 more) * [#307](https://github.com/Tracer-Cloud/opensre/pull/307) feat: move mintlify docs (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `docs-mintlify/.cspell.json`, `docs-mintlify/.gitignore`, `docs-mintlify/.tool-versions`, `docs-mintlify/DataLake.mp4`, `docs-mintlify/LICENSE`, `docs-mintlify/README.md`, `docs-mintlify/Sitemap Tracer.xml`, `docs-mintlify/comparisons/Tracer_vs_AWS_CloudWatch.mdx`, `docs-mintlify/comparisons/Tracer_vs_Airflow.mdx`, `docs-mintlify/comparisons/Tracer_vs_Dagster.mdx`, and 324 more) * [#308](https://github.com/Tracer-Cloud/opensre/pull/308) fix: resolve #291 (author: venturevd; contributors: Andrew Van Dyke and venturevd; labels: *none*; files: `.gitignore`) * [#326](https://github.com/Tracer-Cloud/opensre/pull/326) fix: resolve CodeQL quality alerts (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/auth/jwt_auth.py`, `app/cli/wizard/flow.py`, `docs-mintlify/.vale.ini`, `docs-mintlify/.vale/styles/Vocab/Tracer/accept.txt`, `docs-mintlify/.vale/styles/Vocab/Tracer/reject.txt`, `docs-mintlify/public/radar-chart.js`, `docs-mintlify/toc-actions.js`, and a remote client test) * [#335](https://github.com/Tracer-Cloud/opensre/pull/335) Fix/vale spellcheck vocabulary (author: vincenthus; contributors: vincenthus; labels: *none*; files: `docs-mintlify/.vale.ini`, `docs-mintlify/styles/config/vocabularies/Mintlify/accept.txt`, `docs-mintlify/styles/config/vocabularies/Mintlify/reject.txt`) * [#337](https://github.com/Tracer-Cloud/opensre/pull/337) fix: add .vale.ini to activate vocabulary for Mintlify spellcheck (author: vincenthus; contributors: vincenthus; labels: *none*; files: `docs-mintlify/.vale.ini`) * [#338](https://github.com/Tracer-Cloud/opensre/pull/338) fix: add .vale.ini to activate vocabulary for Mintlify spellcheck (author: vincenthus; contributors: vincenthus; labels: *none*; files: `docs-mintlify/.vale.ini`, `docs-mintlify/quickstart-1.mdx`, `docs-mintlify/quickstart.mdx`, `docs-mintlify/styles/config/vocabularies/Mintlify/accept.txt`) ## Generation metadata * Generator version: `opensre 0.1.0` * Fallback summary used: `no` * UTC window: `2026-04-02T23:00:00+00:00` to `2026-04-03T23:00:00+00:00` # Daily Update — 2026-04-04 Source: https://opensre.com/docs/daily-updates/2026-04-04 OpenSRE engineering daily update for 2026-04-04 (Europe/London) Thanks to everyone who contributed today: Ceren Camkiran, Shoaib Ansari, shoaib050326, Shriyash soni, and vincenthus 🙏🚀 ## Main updates shipped * Daily update workflow simplified -- Slack delivery removed so the workflow now only generates and commits markdown archives under `docs/daily-updates`, with telemetry debug redaction broadened to catch sensitive key substrings like `oauth_refresh_token`. * Production Dockerfile shipped -- new repo-root `Dockerfile` uses Python 3.11 slim, installs deps from `pyproject.toml`, exposes the runtime port, runs as non-root, and adds a `HEALTHCHECK` against the `/ok` endpoint; comprehensive structural tests added in `app/dockerfile_test.py`. * MCP server wired up -- `opensre-mcp` CLI entrypoint added to `pyproject.toml` and setup/OpenClaw configuration documented in `docs/SETUP.md`. * Contributors workflow hardened -- retry logic and explicit HTTP/network error handling added to API calls, broad exception swallowing removed from first-commit lookup, and sort order aligned to earliest-first as documented (fixes #212). * `TracerClientBase._get` typing tightened -- input changed to `Mapping[str, Any]` and return typed as `JSONDict`; runtime behaviour unchanged and focused unit tests added for URL construction, param passing, and empty-param defaults. * Docs links fixed -- all `opensre.com` references replaced with `tracer.mintlify.app` across README, CONTRIBUTING, SECURITY, and Mintlify config files to unblock access while the custom domain misconfiguration is resolved upstream. * Test quality improved -- `MemoryKeyring._entries` moved from class-level to instance attribute to prevent state bleed between tests, and `client_test.py` assertions strengthened with identity checks and `raise_for_status` verification. ## Source pull requests * [#339](https://github.com/Tracer-Cloud/opensre/pull/339) fix: remove daily update Slack delivery and broaden debug redaction (author: vincenthus; contributors: vincenthus; labels: *none*; files: `.github/workflows/daily-update.yml`, `app/analytics/provider.py`, `app/analytics/provider_test.py`, `app/integrations/daily_update.py`, `app/integrations/daily_update_test.py`, `docs/daily-updates/2026-04-02.md`) * [#305](https://github.com/Tracer-Cloud/opensre/pull/305) Add MCP CLI entrypoint for opensre\_mcp (author: Ceren Camkiran; contributors: Ceren Camkiran and vincenthus; labels: *none*; files: `pyproject.toml`) * [#299](https://github.com/Tracer-Cloud/opensre/pull/299) Add production Dockerfile at repo root for container deployment (author: shoaib050326; contributors: Shoaib Ansari and shoaib050326; labels: *none*; files: `Dockerfile`, `app/dockerfile_test.py`) * [#341](https://github.com/Tracer-Cloud/opensre/pull/341) Fix spelling errors in daily updates and strengthen test assertions (author: Copilot; contributors: no human contributors recorded in merged PRs today; labels: *none*; files: `docs/daily-updates/2026-04-02.md`, `docs/daily-updates/2026-04-03.md`, `tests/shared/keyring_backend.py`, and a remote client test) * [#306](https://github.com/Tracer-Cloud/opensre/pull/306) Add MCP server usage and OpenClaw configuration docs (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `docs/SETUP.md`) * [#310](https://github.com/Tracer-Cloud/opensre/pull/310) fix(ci): harden contributors updater and align earliest-first ordering (author: Shriyash soni; contributors: Shriyash soni; labels: *none*; files: `.github/workflows/contributors.yml`) * [#312](https://github.com/Tracer-Cloud/opensre/pull/312) refactor(types): tighten TracerClientBase \_get typing (author: Shriyash soni; contributors: Shriyash soni; labels: *none*; files: `app/integrations/clients/tracer_client/tracer_client_base.py`, `app/integrations/clients/tracer_client/tracer_client_base_test.py`) * [#346](https://github.com/Tracer-Cloud/opensre/pull/346) fix(docs): point docs links to working Mintlify URL (author: vincenthus; contributors: vincenthus; labels: *none*; files: `CONTRIBUTING.md`, `README.md`, `SECURITY.md`, `docs-mintlify/Sitemap Tracer.xml`, `docs-mintlify/docs.json`, `docs-mintlify/technology/product-benefits.mdx`, `docs-mintlify/toc-actions.js`) ## Generation metadata * Generator version: `opensre 0.1.0` * Fallback summary used: `no` * UTC window: `2026-04-03T23:00:00+00:00` to `2026-04-04T23:00:00+00:00` # Daily Update — 2026-04-05 Source: https://opensre.com/docs/daily-updates/2026-04-05 OpenSRE engineering daily update for 2026-04-05 (Europe/London) Thanks to everyone who contributed today: Abhinnavverma, Aniruddha Khandare, Ankit Juneja, Ankit Juneja, Devesh, Ebrahim Sameh, James, Jayant Singh Bisht, qorex, Rohit Rajan, Rohit Rajan, Shoaib Ansari, shoaib050326, vincenthus, Vysakh Ramakrishnan, and Yash Kapure 🙏🚀 ## Main updates shipped * feat(tools): reduce tool creation to a single file (#275). * Add guidelines for creating single-file tools. * Issue/140 elasticsearch integration. * feat: add OpsGenie integration for alert intake and investigation. * feat: Add Vercel integration for deployment monitoring. * feat: add Jira integration for incident ticket management. * \[FEATURE] Add MongoDB integration for database RCA #323. * docs: consolidate MongoDB integration from opensre docs-mintlify. * 27 additional merged pull requests shipped today. ## Source pull requests * [#349](https://github.com/Tracer-Cloud/opensre/pull/349) feat(tools): reduce tool creation to a single file (#275) (author: shoaib050326; contributors: Shoaib Ansari and shoaib050326; labels: *none*; files: `tools/registry.py`, `tools/simple_tools.py`, `tools/simple_tools_test.py`) * [#354](https://github.com/Tracer-Cloud/opensre/pull/354) Add guidelines for creating single-file tools (author: Devesh; contributors: Devesh; labels: *none*; files: `CONTRIBUTING.md`, `tools/registry_test.py`) * [#343](https://github.com/Tracer-Cloud/opensre/pull/343) Issue/140 elasticsearch integration (author: Rohit Rajan; contributors: Rohit Rajan and Rohit Rajan; labels: *none*; files: `app/integrations/clients/elasticsearch/__init__.py`, `app/integrations/clients/elasticsearch/client.py`, `app/state.py`, `tools/ElasticsearchLogsTool/__init__.py`, `tools/ElasticsearchLogsTool/_client.py`, `tests/test_elasticsearch_client.py`) * [#353](https://github.com/Tracer-Cloud/opensre/pull/353) feat: add OpsGenie integration for alert intake and investigation (author: Ebrahim Sameh; contributors: Ebrahim Sameh and vincenthus; labels: *none*; files: `app/cli/__main__.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/clients/opsgenie/__init__.py`, `app/integrations/clients/opsgenie/client.py`, `app/integrations/clients/opsgenie/client_test.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/plan_actions/detect_sources_opsgenie_test.py`, and 9 more) * [#351](https://github.com/Tracer-Cloud/opensre/pull/351) feat: Add Vercel integration for deployment monitoring (author: Ebrahim Sameh; contributors: Ebrahim Sameh and vincenthus; labels: *none*; files: `app/cli/__main__.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/cli/wizard/integration_health_test.py`, `app/integrations/clients/vercel/__init__.py`, `app/integrations/clients/vercel/client.py`, `app/integrations/clients/vercel/client_test.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/integrations/verify_test.py`, and 10 more) * [#298](https://github.com/Tracer-Cloud/opensre/pull/298) feat: add Jira integration for incident ticket management (author: Jayant Singh Bisht; contributors: Jayant Singh Bisht and vincenthus; labels: *none*; files: `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/clients/jira/__init__.py`, `app/integrations/clients/jira/client.py`, `app/integrations/models.py`, `tests/cli_smoke_test.py`, `tests/integrations/test_jira_client.py`) * [#348](https://github.com/Tracer-Cloud/opensre/pull/348) \[FEATURE] Add MongoDB integration for database RCA #323 (author: Yash Kapure; contributors: vincenthus and Yash Kapure; labels: *none*; files: `.env.example`, `app/cli/__main__.py`, `app/integrations/cli.py`, `app/integrations/models.py`, `app/integrations/mongodb.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/resolve_integrations/node.py`, `app/state.py`, `tools/MongoDBCollectionStatsTool/__init__.py`, and 10 more) * [#366](https://github.com/Tracer-Cloud/opensre/pull/366) docs: consolidate MongoDB integration from opensre docs-mintlify (author: vincenthus; contributors: vincenthus; labels: *none*; files: `docs/docs.json`, `docs/mongodb.mdx`, `docs/technology/how-it-works.mdx`) * [#368](https://github.com/Tracer-Cloud/opensre/pull/368) Remove stray docs-mintlify directory (author: vincenthus; contributors: vincenthus; labels: *none*; files: `docs-mintlify/integrations/mongodb.mdx`) * [#340](https://github.com/Tracer-Cloud/opensre/pull/340) added the health check fastapi endpoint for hosted deployments (author: James; contributors: James; labels: *none*; files: `config/webapp.py`, runtime config, `pyproject.toml`, `tests/test_webapp.py`) * [#369](https://github.com/Tracer-Cloud/opensre/pull/369) refactor: extract pipeline/ and state/ packages from flat app root (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/graph_pipeline.py`, `app/pipeline/__init__.py`, `app/pipeline/graph.py`, `app/pipeline/routing.py`, `app/pipeline/runners.py`, `app/routing.py`, `app/runners.py`, `app/state.py`, `app/state/__init__.py`, `app/state/agent_state.py`, and 2 more) * [#370](https://github.com/Tracer-Cloud/opensre/pull/370) refactor: complete semantic restructure — services, types, constants, entrypoints (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/__init__.py`, `app/cli/wizard/integration_health.py`, `app/constants/__init__.py`, `app/constants/prompts.py`, `app/entrypoints/__init__.py`, `app/entrypoints/cli.py`, `app/entrypoints/mcp.py`, `app/entrypoints/mcp_test.py`, `app/entrypoints/sdk.py`, `app/integrations/clients/__init__.py`, and 123 more) * [#371](https://github.com/Tracer-Cloud/opensre/pull/371) refactor: complete semantic restructure — services, types, constants,… (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/__init__.py`, `app/cli/wizard/integration_health.py`, `app/constants/__init__.py`, `app/constants/prompts.py`, `app/entrypoints/__init__.py`, `app/entrypoints/cli.py`, `app/entrypoints/mcp.py`, `app/entrypoints/mcp_test.py`, `app/entrypoints/sdk.py`, `app/integrations/clients/__init__.py`, and 123 more) * [#372](https://github.com/Tracer-Cloud/opensre/pull/372) feat: add Bedrock Agent deployment tests (author: vincenthus; contributors: vincenthus; labels: *none*; files: `Makefile`, `README.md`, `pytest.ini`, `tests/deployment/__init__.py`, `tests/deployment/bedrock/__init__.py`, `tests/deployment/bedrock/conftest.py`, `tests/deployment/bedrock/infrastructure_sdk/__init__.py`, `tests/deployment/bedrock/infrastructure_sdk/agent.py`, `tests/deployment/bedrock/infrastructure_sdk/deploy.py`, `tests/deployment/bedrock/infrastructure_sdk/destroy.py`, and 2 more) * [#373](https://github.com/Tracer-Cloud/opensre/pull/373) refactor: remove all backward-compatibility shims (author: vincenthus; contributors: vincenthus; labels: *none*; files: `Makefile`, `app/cli/investigate.py`, `app/graph_pipeline.py`, `app/integrations/clients/__init__.py`, `app/routing.py`, `app/runners.py`, `pytest.ini`, `tests/benchmarks/toolcall_model_benchmark/pipeline_benchmark.py`, `tests/e2e/rca/run_rca_test.py`, `tests/synthetic/rds_postgres/run_suite.py`) * [#374](https://github.com/Tracer-Cloud/opensre/pull/374) fix: handle IAM cleanup errors explicitly in Bedrock tests (author: vincenthus; contributors: vincenthus; labels: *none*; files: `tests/deployment/bedrock/infrastructure_sdk/iam.py`) * [#376](https://github.com/Tracer-Cloud/opensre/pull/376) feat: add hosted, Vercel, and EC2 deployment tests (author: vincenthus; contributors: vincenthus; labels: *none*; files: `Makefile`, `README.md`, `tests/deployment/ec2/__init__.py`, `tests/deployment/ec2/conftest.py`, `tests/deployment/ec2/infrastructure_sdk/__init__.py`, `tests/deployment/ec2/infrastructure_sdk/deploy.py`, `tests/deployment/ec2/infrastructure_sdk/destroy.py`, `tests/deployment/ec2/infrastructure_sdk/instance.py`, `tests/deployment/ec2/test_agent_e2e.py`, and 14 more) * [#342](https://github.com/Tracer-Cloud/opensre/pull/342) feat : Created v0.1 script for benchmark generation (author: Abhinnavverma; contributors: Abhinnavverma and vincenthus; labels: *none*; files: `Makefile`, `README.md`, `docs/benchmarks/README.md`, `tests/benchmarks/toolcall_model_benchmark/__init__.py`, `tests/benchmarks/toolcall_model_benchmark/benchmark_generator.py`, `tests/benchmarks/toolcall_model_benchmark/pricing.py`) * [#380](https://github.com/Tracer-Cloud/opensre/pull/380) fix: add health command to CLI help and landing page (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/cli/__main__.py`) * [#381](https://github.com/Tracer-Cloud/opensre/pull/381) fix: remove unused EvidenceSource re-export from app.state (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/state/__init__.py`, `app/state/types.py`) * [#356](https://github.com/Tracer-Cloud/opensre/pull/356) feat: add opensre onboard local\_llm — zero-config Ollama setup (author: Ankit Juneja; contributors: Ankit Juneja, Ankit Juneja, and vincenthus; labels: *none*; files: `app/cli/__main__.py`, `app/cli/local_llm/__init__.py`, `app/cli/local_llm/command.py`, `app/cli/local_llm/hardware.py`, `app/cli/local_llm/local_llm_test.py`, `app/cli/local_llm/ollama.py`, `app/cli/wizard/config.py`, `app/cli/wizard/validation.py`, `config/config.py`, `services/llm_client.py`) * [#384](https://github.com/Tracer-Cloud/opensre/pull/384) fix: use install script instead of pip for `opensre update` (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/cli/update.py`, `tests/cli/test_update.py`) * [#386](https://github.com/Tracer-Cloud/opensre/pull/386) feat: interactive input picker for `opensre investigate` (author: vincenthus; contributors: Ankit Juneja and vincenthus; labels: *none*; files: `app/cli/investigate_input.py`, `app/cli/local_llm/command.py`, `app/cli/payload.py`, `app/cli/wizard/validation.py`) * [#385](https://github.com/Tracer-Cloud/opensre/pull/385) Add opensre version subcommand (author: qorex; contributors: qorex and vincenthus; labels: *none*; files: `app/cli/__main__.py`, `tests/cli/test_version_cmd.py`) * [#367](https://github.com/Tracer-Cloud/opensre/pull/367) feat: Add Prefect integration for workflow and worker investigation (author: Aniruddha Khandare; contributors: Aniruddha Khandare; labels: *none*; files: `app/integrations/clients/prefect/__init__.py`, `app/integrations/clients/prefect/client.py`, `app/integrations/models.py`, `app/state.py`, `tools/PrefectFlowRunsTool/__init__.py`, `tools/PrefectFlowRunsTool/tool_test.py`, `tools/PrefectWorkerHealthTool/__init__.py`, `tools/PrefectWorkerHealthTool/tool_test.py`) * [#358](https://github.com/Tracer-Cloud/opensre/pull/358) Update installation process for virtual environment and pip (author: Devesh; contributors: Devesh; labels: *none*; files: `Makefile`) * [#355](https://github.com/Tracer-Cloud/opensre/pull/355) Implement health check endpoint with response model and tests (author: Devesh; contributors: Devesh and vincenthus; labels: *none*; files: `config/webapp.py`, `pyproject.toml`, `tests/test_webapp.py`) * [#391](https://github.com/Tracer-Cloud/opensre/pull/391) fix: remove unused Prefect client global (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/integrations/clients/prefect/client.py`) * [#393](https://github.com/Tracer-Cloud/opensre/pull/393) fix: collapse dual orchestrator into single execution path (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/nodes/resolve_integrations/node.py`, `app/pipeline/runners.py`, `tests/e2e/rca/run_rca_test.py`) * [#394](https://github.com/Tracer-Cloud/opensre/pull/394) fix: code quality cleanup (dotenv, routing logging, state drift, speculative models) (author: vincenthus; contributors: vincenthus; labels: *none*; files: `config/config.py`, `app/pipeline/routing.py`, `app/state/agent_state.py`, `pyproject.toml`, `tests/app/test_agent_state_sync.py`) * [#389](https://github.com/Tracer-Cloud/opensre/pull/389) Enhance UI for terminal command interface (author: Devesh; contributors: Devesh; labels: *none*; files: `app/cli/__main__.py`, `app/cli/health_view.py`, `tests/cli/test_health.py`, `tests/cli_smoke_test.py`) * [#395](https://github.com/Tracer-Cloud/opensre/pull/395) fix: stop persisting JWT token in runtime state (author: Vysakh Ramakrishnan; contributors: Vysakh Ramakrishnan; labels: *none*; files: `app/nodes/auth.py`, `app/nodes/resolve_integrations/node.py`, `app/state/agent_state.py`, `tests/test_auth.py`) * [#397](https://github.com/Tracer-Cloud/opensre/pull/397) test: full unit test coverage for all investigation tools (author: vincenthus; contributors: vincenthus; labels: *none*; files: `tests/tools/conftest.py`, `tests/tools/test_aws_operation_tool.py`, `tests/tools/test_cloudwatch_batch_metrics_tool.py`, `tests/tools/test_cloudwatch_logs_tool.py`, `tests/tools/test_coralogix_logs_tool.py`, `tests/tools/test_datadog_context_tool.py`, `tests/tools/test_datadog_events_tool.py`, `tests/tools/test_datadog_logs_tool.py`, `tests/tools/test_datadog_metrics_tool.py`, `tests/tools/test_datadog_monitors_tool.py`, and 52 more) * [#396](https://github.com/Tracer-Cloud/opensre/pull/396) feat: auto-append benchmark results to README after generation (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `.github/workflows/benchmark-readme.yml`, `Makefile`, `README.md`, `docs/benchmarks/README.md`, `tests/benchmarks/toolcall_model_benchmark/__init__.py`, `tests/benchmarks/toolcall_model_benchmark/benchmark_generator.py`, `tests/benchmarks/toolcall_model_benchmark/readme_updater.py`, `tests/benchmarks/toolcall_model_benchmark/test_benchmark_generator.py`, `tests/benchmarks/toolcall_model_benchmark/test_readme_updater.py`) * [#399](https://github.com/Tracer-Cloud/opensre/pull/399) fix: resolve all 6 open CodeQL security/quality alerts (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/pipeline/graph.py`, `tests/app/test_agent_state_sync.py`, `tests/tools/test_datadog_context_tool.py`, `tests/tools/test_tracer_failed_run_tool.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `yes` * UTC window: `2026-04-04T23:00:00+00:00` to `2026-04-05T23:00:00+00:00` # Daily Update — 2026-04-06 Source: https://opensre.com/docs/daily-updates/2026-04-06 OpenSRE engineering daily update for 2026-04-06 (Europe/London) Thanks to everyone who contributed yesterday: abhishek-marathe04, Ankit Juneja, Ankit Juneja, Matt Van Horn, Raman, Vaibhav Upreti, and vincenthus 🙏🚀 ## Main updates shipped (April 6, 2026) * feat: real-time streaming UI, interactive deploy, and multi-remote support — vincenthus * feat: Add GitLab integration with investigation tools and optional MR commenting.(Fixes Issue #318) — abhishek-marathe04 * feat: add remote agent connection (health + trigger) — vincenthus * feat: add Bitbucket integration for dev-tools investigation — Matt Van Horn * feat: add Kafka integration for streaming and consumer-lag RCA — Matt Van Horn * feat: add ClickHouse integration for OLAP database RCA — Matt Van Horn * feat: add SSE streaming to remote investigate endpoint — vincenthus * chore: refactor cli — Vaibhav Upreti * perf(ci): speed up CI pipeline with uv, pytest-xdist, and job consolidation — vincenthus * fix: resolve critical ollama tag matching issue causing redundant downloads — Ankit Juneja * Feat/daily update 20 highlights — vincenthus * fix: remove typo in contributors workflow API URL (fixes issue #212) — Raman ## Source pull requests * [#398](https://github.com/Tracer-Cloud/opensre/pull/398) feat: add remote agent connection (health + trigger) (author: vincenthus; contributors: vincenthus; labels: *none*; files: `Makefile`, `app/cli/__main__.py`, `app/cli/wizard/store.py`, `config/config.py`, `app/graph_pipeline.py`, `deployment/remote/__init__.py`, `deployment/remote/client.py`, `deployment/remote/renderer.py`, `deployment/remote/server.py`, `deployment/remote/stream.py`, and 29 more) * [#401](https://github.com/Tracer-Cloud/opensre/pull/401) chore: refactor cli (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `app/cli/__main__.py`, `app/cli/args.py`, `app/cli/commands/__init__.py`, `app/cli/commands/deploy.py`, `app/cli/commands/general.py`, `app/cli/commands/integrations.py`, `app/cli/commands/onboard.py`, `app/cli/commands/remote.py`, `app/cli/commands/tests.py`, `app/cli/constants.py`, and 2 more) * [#400](https://github.com/Tracer-Cloud/opensre/pull/400) feat: add SSE streaming to remote investigate endpoint (author: vincenthus; contributors: Matt Van Horn and vincenthus; labels: *none*; files: `.env.example`, `app/integrations/clickhouse.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `tools/ClickHouseQueryActivityTool/__init__.py`, `tools/ClickHouseSystemHealthTool/__init__.py`, `app/types/evidence.py`, `pyproject.toml`, `tests/integrations/test_clickhouse.py`) * [#405](https://github.com/Tracer-Cloud/opensre/pull/405) fix: remove typo in contributors workflow API URL (fixes issue #212) (author: Raman; contributors: Raman; labels: *none*; files: `.github/workflows/contributors.yml`) * [#403](https://github.com/Tracer-Cloud/opensre/pull/403) feat: add Bitbucket integration for dev-tools investigation (author: Matt Van Horn; contributors: Matt Van Horn and vincenthus; labels: *none*; files: `.env.example`, `app/integrations/bitbucket.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `tools/BitbucketCommitsTool/__init__.py`, `tools/BitbucketFileContentsTool/__init__.py`, `tools/BitbucketSearchCodeTool/__init__.py`, `app/types/evidence.py`, `tests/integrations/test_bitbucket.py`) * [#409](https://github.com/Tracer-Cloud/opensre/pull/409) Feat/daily update 20 highlights (author: vincenthus; contributors: Matt Van Horn and vincenthus; labels: *none*; files: `.github/workflows/daily-update.yml`, `.github/workflows/security-autofix.yml`, `app/integrations/clickhouse.py`, `app/integrations/daily_update.py`, `app/integrations/verify.py`, `deployment/remote/server.py`, `tests/integrations/test_daily_update.py`) * [#402](https://github.com/Tracer-Cloud/opensre/pull/402) feat: add ClickHouse integration for OLAP database RCA (author: Matt Van Horn; contributors: Matt Van Horn and vincenthus; labels: *none*; files: `deployment/remote/server.py`) * [#410](https://github.com/Tracer-Cloud/opensre/pull/410) perf(ci): speed up CI pipeline with uv, pytest-xdist, and job consolidation (author: vincenthus; contributors: vincenthus; labels: *none*; files: `.github/workflows/ci.yml`, `Makefile`, `pyproject.toml`, `pytest.ini`) * [#404](https://github.com/Tracer-Cloud/opensre/pull/404) feat: add Kafka integration for streaming and consumer-lag RCA (author: Matt Van Horn; contributors: Matt Van Horn and vincenthus; labels: *none*; files: `.env.example`, `app/integrations/kafka.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `tools/KafkaConsumerGroupTool/__init__.py`, `tools/KafkaTopicHealthTool/__init__.py`, `app/types/evidence.py`, `mypy.ini`, `pyproject.toml`, `tests/integrations/test_kafka.py`) * [#407](https://github.com/Tracer-Cloud/opensre/pull/407) feat: Add GitLab integration with investigation tools and optional MR commenting.(Fixes Issue #318) (author: abhishek-marathe04; contributors: abhishek-marathe04; labels: *none*; files: `.env.example`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/gitlab.py`, `app/integrations/gitlab_test.py`, `app/integrations/models.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/plan_actions/detect_sources_test.py`, `app/nodes/publish_findings/node.py`, `app/nodes/publish_findings/node_test.py`, and 11 more) * [#412](https://github.com/Tracer-Cloud/opensre/pull/412) feat: real-time streaming UI, interactive deploy, and multi-remote support (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/cli/__main__.py`, `app/cli/commands/__init__.py`, `app/cli/commands/deploy.py`, `app/cli/commands/doctor.py`, `app/cli/commands/general.py`, `app/cli/commands/remote.py`, `app/cli/commands/tests.py`, `app/cli/context.py`, `app/cli/errors.py`, `app/cli/exit_codes.py`, and 28 more) * [#408](https://github.com/Tracer-Cloud/opensre/pull/408) fix: resolve critical ollama tag matching issue causing redundant downloads (author: Ankit Juneja; contributors: Ankit Juneja and Ankit Juneja; labels: *none*; files: `app/cli/local_llm/command.py`, `app/cli/local_llm/local_llm_test.py`, `app/cli/local_llm/ollama.py`, `app/cli/wizard/validation.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-05T23:00:00+00:00` to `2026-04-06T23:00:00+00:00` # Daily Update — 2026-04-07 Source: https://opensre.com/docs/daily-updates/2026-04-07 OpenSRE engineering daily update for 2026-04-07 (Europe/London) Thanks to everyone who contributed yesterday: Shoaib, Shoaib Ansari, and vincenthus 🙏🚀 ## Main updates shipped (April 7, 2026) * feat: Add Railway support to opensre deploy (#271) — Shoaib * fix(ci): daily-update scheduled runs actually generate docs — vincenthus ## Source pull requests * [#300](https://github.com/Tracer-Cloud/opensre/pull/300) feat: Add Railway support to opensre deploy (#271) (author: Shoaib; contributors: Shoaib and Shoaib Ansari; labels: *none*; files: `.env.example`, `Dockerfile`, `app/analytics/cli.py`, `app/analytics/events.py`, `app/cli/commands/deploy.py`, `app/cli/deploy.py`, `app/cli/deploy_test.py`, `app/dockerfile_test.py`, `infra/docker-compose.database.yml`, `tests/test_dockerfile.py`) * [#499](https://github.com/Tracer-Cloud/opensre/pull/499) fix(ci): daily-update scheduled runs actually generate docs (author: vincenthus; contributors: vincenthus; labels: *none*; files: `.github/workflows/daily-update.yml`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-06T23:00:00+00:00` to `2026-04-07T23:00:00+00:00` # Daily Update — 2026-04-08 Source: https://opensre.com/docs/daily-updates/2026-04-08 OpenSRE engineering daily update for 2026-04-08 (Europe/London) Thanks to everyone who contributed yesterday: no human contributors recorded in merged PRs today 🙏🚀 ## Main updates shipped (April 8, 2026) * No pull requests were merged into `main` today. ## Source pull requests * No pull requests were merged during this London calendar day. ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-07T23:00:00+00:00` to `2026-04-08T23:00:00+00:00` # Daily Update — 2026-04-09 Source: https://opensre.com/docs/daily-updates/2026-04-09 OpenSRE engineering daily update for 2026-04-09 (Europe/London) Thanks to everyone who contributed yesterday: Tan Wee Joe 🙏🚀 ## Main updates shipped (April 9, 2026) * 491 separate out local vs enterprise opensre integrations in docs (#491) — Tan Wee Joe ## Source pull requests * [#516](https://github.com/Tracer-Cloud/opensre/pull/516) 491 separate out local vs enterprise opensre integrations in docs (author: Tan Wee Joe; contributors: Tan Wee Joe; labels: *none*; files: `docs/aws.mdx`, `docs/bitbucket.mdx`, `docs/clickhouse.mdx`, `docs/coralogix.mdx`, `docs/docs.json`, `docs/github.mdx`, `docs/gitlab.mdx`, `docs/google-docs.mdx`, `docs/honeycomb.mdx`, `docs/integrations-overview.mdx`, and 6 more) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-08T23:00:00+00:00` to `2026-04-09T23:00:00+00:00` # Daily Update — 2026-04-10 Source: https://opensre.com/docs/daily-updates/2026-04-10 OpenSRE engineering daily update for 2026-04-10 (Europe/London) Thanks to everyone who contributed yesterday: Ankit Juneja, Ankit Juneja, Devesh, and mayankbharati-ops 🙏🚀 ## Main updates shipped (April 10, 2026) * feat: add PostgreSQL integration (#322) — Ankit Juneja * Add CLI commands for post-deploy operations on hosted services — Devesh * refactor(#181): remove 6 dead functions across 3 files — mayankbharati-ops ## Source pull requests * [#418](https://github.com/Tracer-Cloud/opensre/pull/418) Add CLI commands for post-deploy operations on hosted services (author: Devesh; contributors: Devesh; labels: *none*; files: `README.md`, `app/cli/commands/remote.py`, `app/cli/layout.py`, `app/cli/wizard/store.py`, `deployment/remote/ops.py`, `deployment/remote/server.py`, `tests/cli/test_remote.py`, `tests/cli/wizard/test_store.py`, `tests/cli_smoke_test.py`, `tests/remote/test_ops.py`) * [#522](https://github.com/Tracer-Cloud/opensre/pull/522) refactor(#181): remove 6 dead functions across 3 files (author: mayankbharati-ops; contributors: mayankbharati-ops; labels: *none*; files: `app/cli/commands/remote.py`, `app/nodes/publish_findings/formatters/report.py`, `services/vercel/client.py`) * [#515](https://github.com/Tracer-Cloud/opensre/pull/515) feat: add PostgreSQL integration (author: Ankit Juneja; contributors: Ankit Juneja and Ankit Juneja; labels: *none*; files: `app/integrations/cli.py`, `app/integrations/models.py`, `app/integrations/postgresql.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/resolve_integrations/node.py`, `tools/PostgreSQLCurrentQueriesTool/__init__.py`, `tools/PostgreSQLReplicationStatusTool/__init__.py`, `tools/PostgreSQLServerStatusTool/__init__.py`, `tools/PostgreSQLSlowQueriesTool/__init__.py`, and 13 more) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-09T23:00:00+00:00` to `2026-04-10T23:00:00+00:00` # Daily Update — 2026-04-11 Source: https://opensre.com/docs/daily-updates/2026-04-11 OpenSRE engineering daily update for 2026-04-11 (Europe/London) Thanks to everyone who contributed yesterday: no human contributors recorded in merged PRs today 🙏🚀 ## Main updates shipped (April 11, 2026) * No pull requests were merged into `main` today. ## Source pull requests * No pull requests were merged during this London calendar day. ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `yes` * UTC window: `2026-04-10T23:00:00+00:00` to `2026-04-11T23:00:00+00:00` # Daily Update — 2026-04-12 Source: https://opensre.com/docs/daily-updates/2026-04-12 OpenSRE engineering daily update for 2026-04-12 (Europe/London) Thanks to everyone who contributed yesterday: no human contributors recorded in merged PRs today 🙏🚀 ## Main updates shipped (April 12, 2026) * No pull requests were merged into `main` today. ## Source pull requests * No pull requests were merged during this London calendar day. ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `yes` * UTC window: `2026-04-11T23:00:00+00:00` to `2026-04-12T23:00:00+00:00` # Daily Update — 2026-04-13 Source: https://opensre.com/docs/daily-updates/2026-04-13 OpenSRE engineering daily update for 2026-04-13 (Europe/London) Thanks to everyone who contributed yesterday: Abhishek Marathe, Devesh, Harsha Pasham, Sundaram Kumar Jha, Vaibhav Upreti, and Yash Kumar Saini 🙏🚀 ## Main updates shipped (April 13, 2026) * feat: add MariaDB integration for database RCA — Yash Kumar Saini * feat: Add discord integration. (Fixes #359) — Abhishek Marathe * P2 Run one-off diagnostic Python safely during investigations — Harsha Pasham * fix: harden installer scripts and validate release assets — Sundaram Kumar Jha * ci: add Windows ARM64 binary to release pipeline — Sundaram Kumar Jha * refactor: convert MariaDB env config to dataclass — Devesh * docs: add MariaDB integration page — Yash Kumar Saini ## Source pull requests * [#492](https://github.com/Tracer-Cloud/opensre/pull/492) feat: add MariaDB integration for database RCA (author: Yash Kumar Saini; contributors: Yash Kumar Saini; labels: *none*; files: `.env.example`, `app/auth/auth.py`, `app/cli/commands/tests.py`, `app/cli/constants.py`, `app/cli/wizard/env_sync.py`, `app/integrations/cli.py`, `app/integrations/mariadb.py`, `app/integrations/models.py`, `app/integrations/store.py`, `app/integrations/verify.py`, and 19 more) * [#485](https://github.com/Tracer-Cloud/opensre/pull/485) P2 Run one-off diagnostic Python safely during investigations (author: Harsha Pasham; contributors: Harsha Pasham and Vaibhav Upreti; labels: *none*; files: `app/nodes/investigate/processing/post_process.py`, `app/sandbox/__init__.py`, `app/sandbox/runner.py`, `tools/run_diagnostic_code.py`, `tests/sandbox/__init__.py`, `tests/sandbox/test_runner.py`, `tests/tools/test_run_diagnostic_code_tool.py`) * [#510](https://github.com/Tracer-Cloud/opensre/pull/510) feat: Add discord integration. (Fixes #359) (author: Abhishek Marathe; contributors: Abhishek Marathe and Vaibhav Upreti; labels: *none*; files: `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/cli.py`, `app/integrations/gitlab.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/publish_findings/node.py`, `app/nodes/resolve_integrations/node.py`, `deployment/remote/server.py`, and 14 more) * [#531](https://github.com/Tracer-Cloud/opensre/pull/531) chore(deps-dev): update confluent-kafka requirement from >=2.3.0 to >=2.14.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#540](https://github.com/Tracer-Cloud/opensre/pull/540) chore(deps): update opentelemetry-exporter-otlp-proto-http requirement from >=1.20.0 to >=1.41.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#539](https://github.com/Tracer-Cloud/opensre/pull/539) chore(deps): update python-dotenv requirement from >=1.0.0 to >=1.2.2 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#543](https://github.com/Tracer-Cloud/opensre/pull/543) chore(deps-dev): update pytest-asyncio requirement from >=0.23.0 to >=1.3.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#537](https://github.com/Tracer-Cloud/opensre/pull/537) chore(deps-dev): update clickhouse-connect requirement from >=0.8.0 to >=0.15.1 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#546](https://github.com/Tracer-Cloud/opensre/pull/546) docs: add MariaDB integration page (author: Yash Kumar Saini; contributors: Yash Kumar Saini; labels: *none*; files: `docs/docs.json`, `docs/integrations-overview.mdx`, `docs/mariadb.mdx`) * [#532](https://github.com/Tracer-Cloud/opensre/pull/532) chore(deps): update mcp requirement from >=1.20.0 to >=1.27.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#533](https://github.com/Tracer-Cloud/opensre/pull/533) chore(deps-dev): update pytest-cov requirement from >=4.0.0 to >=7.1.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#536](https://github.com/Tracer-Cloud/opensre/pull/536) chore(deps): update hosted tracing requirement (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#544](https://github.com/Tracer-Cloud/opensre/pull/544) chore(deps): update rich requirement from >=13.0.0 to >=15.0.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#541](https://github.com/Tracer-Cloud/opensre/pull/541) chore(deps): update boto3 requirement from >=1.34.0 to >=1.42.88 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#534](https://github.com/Tracer-Cloud/opensre/pull/534) chore(deps): update google-api-python-client requirement from \<3.0.0,>=2.0.0 to >=2.194.0,\<3.0.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#542](https://github.com/Tracer-Cloud/opensre/pull/542) chore(deps): update legacy graph runtime requirement (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#538](https://github.com/Tracer-Cloud/opensre/pull/538) chore(deps): update questionary requirement from >=2.1.0 to >=2.1.1 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#535](https://github.com/Tracer-Cloud/opensre/pull/535) chore(deps): update opentelemetry-api requirement from >=1.20.0 to >=1.41.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python`; files: `pyproject.toml`) * [#547](https://github.com/Tracer-Cloud/opensre/pull/547) refactor: convert MariaDB env config to dataclass (author: Devesh; contributors: Devesh; labels: *none*; files: `app/integrations/verify.py`) * [#552](https://github.com/Tracer-Cloud/opensre/pull/552) fix: harden installer scripts and validate release assets (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `install.ps1`, `install.sh`) * [#554](https://github.com/Tracer-Cloud/opensre/pull/554) ci: add Windows ARM64 binary to release pipeline (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `.github/workflows/release.yml`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-12T23:00:00+00:00` to `2026-04-13T23:00:00+00:00` # Daily Update — 2026-04-14 Source: https://opensre.com/docs/daily-updates/2026-04-14 OpenSRE engineering daily update for 2026-04-14 (Europe/London) Thanks to everyone who contributed yesterday: Ankit Juneja, Ankit Juneja, Ceren Camkiran, Devesh, Gautam Jain, Menma, Micheal Angelo, Mudit Tyagi, Oluwasegun Haziz, Sundaram Kumar Jha, Vaibhav Upreti, and vincenthus 🙏🚀 ## Main updates shipped (April 14, 2026) * feat: openclaw integration v1 — Vaibhav Upreti * feat(mysql): add MySQL database integration (#390) — Ankit Juneja * feat: Implement source provenance tracking and rendering — Devesh * fix: resolve remaining CodeQL code-quality alerts (#568) — Sundaram Kumar Jha * feat: add trello integration with e2e tests — Ceren Camkiran * fix: harden release versioning and installer fallback behavior — Sundaram Kumar Jha * fix(remote): fail closed when OPENSRE\_API\_KEY is unset — Sundaram Kumar Jha * feat: add e2e remote Vercel connection test after deploy (#390) — Oluwasegun Haziz * Add PostHog config validation and E2E test — Ceren Camkiran * fix: duplicate trello field in EffectiveIntegrations (fixes #562) — Micheal Angelo * fix: add trello to EffectiveIntegrations — Ceren Camkiran * Create trello.mdx — Ceren Camkiran * docs: clarify two contribution paths in CONTRIBUTING.md — vincenthus * doc-fix: Moved Remote Hosted Ops Section — Menma * doc-update: update repo name in setup.md — Mudit Tyagi * doc-update: update hyperlinks in security.md — Mudit Tyagi * Fix: remove empty benchmark placeholder section in readme — Gautam Jain ## Source pull requests * [#556](https://github.com/Tracer-Cloud/opensre/pull/556) feat: add e2e remote Vercel connection test after deploy (#390) (author: Oluwasegun Haziz; contributors: Oluwasegun Haziz; labels: *none*; files: `tests/deployment/vercel/infrastructure_sdk/client.py`, `tests/deployment/vercel/test_remote_connection_e2e.py`) * [#550](https://github.com/Tracer-Cloud/opensre/pull/550) Add PostHog config validation and E2E test (author: Ceren Camkiran; contributors: Ceren Camkiran and Vaibhav Upreti; labels: *none*; files: `app/integrations/models.py`, `app/integrations/posthog.py`, `tests/e2e/posthog/__init__.py`, `tests/e2e/posthog/test_orchestrator.py`, `tests/integrations/test_config_validation.py`) * [#558](https://github.com/Tracer-Cloud/opensre/pull/558) fix: harden release versioning and installer fallback behavior (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `.gitattributes`, `.github/workflows/release.yml`, `deployment/remote/system_metrics.py`, `install.ps1`, `install.sh`, `packaging/sync_release_version.py`) * [#557](https://github.com/Tracer-Cloud/opensre/pull/557) feat(mysql): add MySQL database integration (author: Ankit Juneja; contributors: Ankit Juneja and Ankit Juneja; labels: *none*; files: `app/cli/constants.py`, `app/integrations/cli.py`, `app/integrations/models.py`, `app/integrations/mysql.py`, `app/integrations/postgresql.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/resolve_integrations/node.py`, `tools/MySQLCurrentProcessesTool/__init__.py`, `tools/MySQLReplicationStatusTool/__init__.py`, and 17 more) * [#525](https://github.com/Tracer-Cloud/opensre/pull/525) feat: add trello integration with e2e tests (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/integrations/trello.py`, `tests/e2e/trello/__init__.py`, `tests/e2e/trello/test_orchestrator.py`, `tests/integrations/test_trello.py`) * [#526](https://github.com/Tracer-Cloud/opensre/pull/526) fix: add trello to EffectiveIntegrations (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/integrations/models.py`) * [#549](https://github.com/Tracer-Cloud/opensre/pull/549) Create trello.mdx (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `docs/integrations/trello.mdx`) * [#560](https://github.com/Tracer-Cloud/opensre/pull/560) fix(remote): fail closed when OPENSRE\_API\_KEY is unset (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `deployment/remote/server.py`, `tests/deployment/remote/test_server.py`, `tests/remote/test_discord_interactions.py`) * [#563](https://github.com/Tracer-Cloud/opensre/pull/563) fix: duplicate trello field in EffectiveIntegrations (fixes #562) (author: Micheal Angelo; contributors: Micheal Angelo; labels: *none*; files: `app/integrations/models.py`) * [#565](https://github.com/Tracer-Cloud/opensre/pull/565) doc-fix: Moved Remote Hosted Ops Section (author: Menma; contributors: Menma; labels: *none*; files: `DEPLOYEMENT.md`, `README.md`) * [#519](https://github.com/Tracer-Cloud/opensre/pull/519) feat: Implement source provenance tracking and rendering (author: Devesh; contributors: Devesh; labels: *none*; files: `app/cli/wizard/env_sync.py`, `app/integrations/postgresql.py`, `app/nodes/publish_findings/formatters/evidence.py`, `app/nodes/publish_findings/formatters/report.py`, `app/nodes/publish_findings/report_context.py`, `deployment/remote/server.py`, `docs/daily-updates/2026-04-08.mdx`, `tests/nodes/publish_findings/test_report_provenance.py`) * [#566](https://github.com/Tracer-Cloud/opensre/pull/566) Fix: remove empty benchmark placeholder section in readme (author: Gautam Jain; contributors: Gautam Jain; labels: *none*; files: `README.md`) * [#571](https://github.com/Tracer-Cloud/opensre/pull/571) doc-update: update repo name in setup.md (author: Mudit Tyagi; contributors: Mudit Tyagi; labels: *none*; files: `SETUP.md`) * [#572](https://github.com/Tracer-Cloud/opensre/pull/572) doc-update: update hyperlinks in security.md (author: Mudit Tyagi; contributors: Mudit Tyagi; labels: *none*; files: `SECURITY.md`) * [#573](https://github.com/Tracer-Cloud/opensre/pull/573) docs: clarify two contribution paths in CONTRIBUTING.md (author: vincenthus; contributors: vincenthus; labels: *none*; files: `CONTRIBUTING.md`) * [#578](https://github.com/Tracer-Cloud/opensre/pull/578) fix: resolve remaining CodeQL code-quality alerts (#568) (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `app/cli/layout.py`, `app/cli/local_llm/hardware.py`, `app/cli/local_llm/ollama.py`, `app/cli/prompt_support.py`, `app/cli/tests/discover.py`, `app/llm_credentials.py`, `app/nodes/plan_actions/detect_sources.py`, `deployment/remote/client.py`, `deployment/remote/system_metrics.py`, `services/eks/eks_k8s_client.py`, and 22 more) * [#586](https://github.com/Tracer-Cloud/opensre/pull/586) feat: openclaw integration v1 (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `SETUP.md`, `app/cli/constants.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/models.py`, `app/integrations/openclaw.py`, `app/integrations/verify.py`, `app/nodes/investigate/models.py`, `app/nodes/investigate/node.py`, `app/nodes/investigate/processing/post_process.py`, and 17 more) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-13T23:00:00+00:00` to `2026-04-14T23:00:00+00:00` # Daily Update — 2026-04-15 Source: https://opensre.com/docs/daily-updates/2026-04-15 OpenSRE engineering daily update for 2026-04-15 (Europe/London) Thanks to everyone who contributed yesterday: Devesh, Sundaram Kumar Jha, and vincenthus 🙏🚀 ## Main updates shipped (April 15, 2026) * refactor(integrations): consolidate integration resolution and verification around a shared catalog (#none) — Sundaram Kumar Jha * feat: add Windows CI coverage and fix Windows test portability — Sundaram Kumar Jha * fix: refactor daily update generation and improve overview documentation — Devesh * fix: resolve CodeQL unreachable-code warnings (#396, #397) — vincenthus * fix: resolve runtime config parameter type warning — vincenthus ## Source pull requests * [#588](https://github.com/Tracer-Cloud/opensre/pull/588) fix: resolve CodeQL unreachable-code warnings (#396, #397) (author: vincenthus; contributors: vincenthus; labels: *none*; files: `tests/services/opsgenie/test_client.py`, `tests/services/vercel/test_client.py`) * [#589](https://github.com/Tracer-Cloud/opensre/pull/589) fix: resolve runtime config parameter type warning (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/nodes/resolve_integrations/node.py`) * [#590](https://github.com/Tracer-Cloud/opensre/pull/590) feat: add Windows CI coverage and fix Windows test portability (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `.github/workflows/ci.yml`, `app/cli/wizard/prompts.py`, `pyproject.toml`, `tests/deployment/remote/test_system_metrics.py`, `tests/cli/test_prompt_support.py`, `tests/cli/wizard/test_prompts.py`, `tests/cli_smoke_test.py`, `tests/integrations/test_store.py`) * [#592](https://github.com/Tracer-Cloud/opensre/pull/592) fix: refactor daily update generation and improve overview documentation (author: Devesh; contributors: Devesh; labels: *none*; files: `app/integrations/daily_update.py`, `docs/daily-updates/overview.mdx`, `docs/docs.json`) * [#594](https://github.com/Tracer-Cloud/opensre/pull/594) refactor(integrations): consolidate integration resolution and verification around a shared catalog (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `app/cli/wizard/flow.py`, `app/integrations/catalog.py`, `app/integrations/store.py`, `app/integrations/verify.py`, `app/nodes/resolve_integrations/node.py`, `tests/integrations/test_verify.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-14T23:00:00+00:00` to `2026-04-15T23:00:00+00:00` # Daily Update — 2026-04-16 Source: https://opensre.com/docs/daily-updates/2026-04-16 OpenSRE engineering daily update for 2026-04-16 (Europe/London) Thanks to everyone who contributed yesterday: Ceren Camkiran, Ebrahim Sameh, Malik Amir Hamza, and Octopus 🙏🚀 ## Main updates shipped (April 16, 2026) * feat: add Jira investigation tools for incident ticket management (#287) — Malik Amir Hamza * fix: plumb EKS tool output into state evidence via post\_process mappers — Ebrahim Sameh * feat: add MiniMax provider support — Octopus * synthetic: strengthen validation for 005-failover — Ceren Camkiran ## Source pull requests * [#584](https://github.com/Tracer-Cloud/opensre/pull/584) fix: plumb EKS tool output into state evidence via post\_process mappers (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `app/nodes/investigate/processing/post_process.py`, `tests/nodes/investigate/__init__.py`, `tests/nodes/investigate/test_post_process.py`) * [#621](https://github.com/Tracer-Cloud/opensre/pull/621) synthetic: strengthen validation for 005-failover (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `tests/synthetic/rds_postgres/005-failover/QA_VALIDATION.md`, `tests/synthetic/rds_postgres/005-failover/answer.yml`) * [#619](https://github.com/Tracer-Cloud/opensre/pull/619) feat: add Jira investigation tools for incident ticket management (#287) (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `.env.example`, `app/integrations/catalog.py`, `app/nodes/plan_actions/detect_sources.py`, `services/jira/__init__.py`, `services/jira/client.py`, `tools/JiraAddCommentTool/__init__.py`, `tools/JiraCreateIssueTool/__init__.py`, `tools/JiraIssueDetailTool/__init__.py`, `tools/JiraSearchIssuesTool/__init__.py`, `app/types/evidence.py`, and 5 more) * [#620](https://github.com/Tracer-Cloud/opensre/pull/620) feat: add MiniMax provider support (author: Octopus; contributors: octo-patch and Octopus; labels: *none*; files: `config/config.py`, `services/llm_client.py`, `tests/services/test_llm_client.py`, `tests/test_config.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-15T23:00:00+00:00` to `2026-04-16T23:00:00+00:00` # Daily Update — 2026-04-17 Source: https://opensre.com/docs/daily-updates/2026-04-17 OpenSRE engineering daily update for 2026-04-17 (Europe/London) Thanks to everyone who contributed yesterday: ANIRUDDHA ADAK, Ankit Juneja, Ankit Juneja, Ceren Camkiran, Chris Chen, Malik Amir Hamza, Matt Van Horn, Oluwasegun Haziz, vincenthus, and Yassir Maknaoui 🙏🚀 ## Main updates shipped (April 17, 2026) * feat: Add Azure SQL Database integration for managed database RCA — Oluwasegun Haziz * feat: add remote runtime investigation with Slack thread context (#301) — Malik Amir Hamza * feat: mask sensitive infrastructure identifiers before model calls (#…) — Malik Amir Hamza * fix(synthetic-qa): Identify healthy alerts correctly (#596) — ANIRUDDHA ADAK * Fix: Include eks\_\* keys in \_INVESTIGATED\_EVIDENCE\_KEYS for is\_clearly\_healthy (Fixes #582) — ANIRUDDHA ADAK * fix: remove duplicate EKS mapper definitions and dict keys (CodeQL alerts) — vincenthus * Ask for Ollama host URL instead of API key during onboard — Matt Van Horn * fix(tests): prevent flaky test\_validation by preloading SDK error cla… — Malik Amir Hamza * fix: clarify Railway deployment prerequisites — Yassir Maknaoui * feat(readme): add Public Beta positioning and status section — Ceren Camkiran * fix(docs): replace invalid --service flag with positional argument in verify commands — Ankit Juneja * Fix: remove invalid --service flag from integrations verify docs — Chris Chen ## Source pull requests * [#622](https://github.com/Tracer-Cloud/opensre/pull/622) feat: Add Azure SQL Database integration for managed database RCA (author: Oluwasegun Haziz; contributors: Oluwasegun Haziz; labels: *none*; files: `app/integrations/azure_sql.py`, `app/integrations/catalog.py`, `app/integrations/cli.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `tools/AzureSQLCurrentQueriesTool/__init__.py`, `tools/AzureSQLResourceStatsTool/__init__.py`, `tools/AzureSQLServerStatusTool/__init__.py`, `tools/AzureSQLSlowQueriesTool/__init__.py`, `tools/AzureSQLWaitStatsTool/__init__.py`, and 9 more) * [#617](https://github.com/Tracer-Cloud/opensre/pull/617) Fix: Include eks\_\* keys in \_INVESTIGATED\_EVIDENCE\_KEYS for is\_clearly\_healthy (Fixes #582) (author: ANIRUDDHA ADAK; contributors: ANIRUDDHA ADAK; labels: *none*; files: `app/nodes/investigate/processing/post_process.py`, `app/nodes/root_cause_diagnosis/evidence_checker.py`, `pr_body.md`, `tests/nodes/investigate/test_post_process.py`, `tests/nodes/root_cause_diagnosis/test_evidence_checker.py`) * [#618](https://github.com/Tracer-Cloud/opensre/pull/618) fix(synthetic-qa): Identify healthy alerts correctly (#596) (author: ANIRUDDHA ADAK; contributors: ANIRUDDHA ADAK; labels: *none*; files: `app/nodes/extract_alert/extract.py`, `app/nodes/investigate/processing/post_process.py`, `app/nodes/plan_actions/build_prompt.py`, `app/nodes/plan_actions/node.py`, `app/nodes/root_cause_diagnosis/evidence_checker.py`, `tests/nodes/investigate/test_post_process.py`, `tests/nodes/root_cause_diagnosis/test_evidence_checker.py`) * [#623](https://github.com/Tracer-Cloud/opensre/pull/623) fix: remove duplicate EKS mapper definitions and dict keys (CodeQL alerts) (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/nodes/investigate/processing/post_process.py`, `tests/nodes/root_cause_diagnosis/test_evidence_checker.py`) * [#632](https://github.com/Tracer-Cloud/opensre/pull/632) fix(tests): prevent flaky test\_validation by preloading SDK error cla… (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `tests/cli/wizard/test_validation.py`) * [#630](https://github.com/Tracer-Cloud/opensre/pull/630) feat: add remote runtime investigation with Slack thread context (#301) (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `.env.example`, `app/cli/commands/general.py`, `deployment/remote/ops.py`, `deployment/remote/runtime_alert.py`, `deployment/remote/slack_context.py`, `docs/docs.json`, `docs/remote-runtime-investigation.mdx`, `tests/cli/test_investigate_service_flag.py`, `tests/remote/test_ops_fetch_logs.py`, `tests/remote/test_runtime_alert.py`, and 1 more) * [#634](https://github.com/Tracer-Cloud/opensre/pull/634) feat: mask sensitive infrastructure identifiers before model calls (#… (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `.env.example`, `app/masking/__init__.py`, `app/masking/context.py`, `app/masking/detectors.py`, `app/masking/policy.py`, `app/nodes/investigate/node.py`, `app/nodes/publish_findings/node.py`, `app/nodes/root_cause_diagnosis/node.py`, `app/state/agent_state.py`, `app/state/factory.py`, and 9 more) * [#633](https://github.com/Tracer-Cloud/opensre/pull/633) feat(readme): add Public Beta positioning and status section (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `README.md`, `app/cli/__main__.py`) * [#636](https://github.com/Tracer-Cloud/opensre/pull/636) fix(docs): replace invalid --service flag with positional argument in verify commands (author: Ankit Juneja; contributors: Ankit Juneja and Ankit Juneja; labels: *none*; files: `docs/aws.mdx`, `docs/bitbucket.mdx`, `docs/clickhouse.mdx`, `docs/coralogix.mdx`, `docs/github.mdx`, `docs/google-docs.mdx`, `docs/honeycomb.mdx`, `docs/integrations-overview.mdx`, `docs/kafka.mdx`, `docs/mongodb.mdx`, and 5 more) * [#637](https://github.com/Tracer-Cloud/opensre/pull/637) Fix: remove invalid --service flag from integrations verify docs (author: Chris Chen; contributors: Chris Chen; labels: *none*; files: `docs/aws.mdx`, `docs/bitbucket.mdx`, `docs/clickhouse.mdx`, `docs/coralogix.mdx`, `docs/github.mdx`, `docs/google-docs.mdx`, `docs/honeycomb.mdx`, `docs/integrations-overview.mdx`, `docs/kafka.mdx`, `docs/mongodb.mdx`, and 5 more) * [#628](https://github.com/Tracer-Cloud/opensre/pull/628) Ask for Ollama host URL instead of API key during onboard (author: Matt Van Horn; contributors: Matt Van Horn; labels: *none*; files: `app/cli/wizard/config.py`, `app/cli/wizard/flow.py`) * [#624](https://github.com/Tracer-Cloud/opensre/pull/624) fix: clarify Railway deployment prerequisites (author: Yassir Maknaoui; contributors: Yassir Maknaoui; labels: *none*; files: `.env.example`, `README.md`, `app/cli/deploy.py`, `app/cli/deploy_test.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-16T23:00:00+00:00` to `2026-04-17T23:00:00+00:00` # Daily Update — 2026-04-18 Source: https://opensre.com/docs/daily-updates/2026-04-18 OpenSRE engineering daily update for 2026-04-18 (Europe/London) Thanks to everyone who contributed yesterday: ANIRUDDHA ADAK, Devesh, Ebrahim Sameh, kaushal-bakrania, Sharky, Vadym Petrychenko, and vincenthus 🙏🚀 ## Main updates shipped (April 18, 2026) * feat: first-class Alertmanager integration (#316) — kaushal-bakrania * feat: add Kubernetes synthetic RCA test harness — Ebrahim Sameh * fix: resolve 10 AI quality findings across 5 files — vincenthus * fix: wire grafana\_backend into GrafanaTracesTool for synthetic tests — vincenthus * feat: add "Run All" option to interactive test picker — vincenthus * Refactor code for improved readability and conditional structure — Devesh * fix: Database directives for RDS QA testing — ANIRUDDHA ADAK * fix: align integrations count to 60+ in README — Vadym Petrychenko * Improve onboarding & examples for opensre — Sharky ## Source pull requests * [#638](https://github.com/Tracer-Cloud/opensre/pull/638) Improve onboarding & examples for opensre (author: Sharky; contributors: Sharky and vincenthus; labels: *none*; files: `IMPROVEMENTS.md`) * [#641](https://github.com/Tracer-Cloud/opensre/pull/641) Refactor code for improved readability and conditional structure (author: Devesh; contributors: Devesh; labels: *none*; files: `app/nodes/investigate/node.py`, `services/llm_client.py`) * [#587](https://github.com/Tracer-Cloud/opensre/pull/587) feat: first-class Alertmanager integration (author: kaushal-bakrania; contributors: kaushal-bakrania and vincenthus; labels: *none*; files: `app/cli/constants.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/catalog.py`, `app/integrations/cli.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/nodes/extract_alert/extract.py`, `app/nodes/investigate/processing/post_process.py`, `app/nodes/plan_actions/detect_sources.py`, and 12 more) * [#567](https://github.com/Tracer-Cloud/opensre/pull/567) fix: resolve 10 AI quality findings across 5 files (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/integrations/models.py`, `tests/integrations/test_trello.py`, `tests/remote/test_discord_interactions.py`, `tests/tools/test_mysql_current_processes_tool.py`, `tests/tools/test_mysql_slow_queries_tool.py`) * [#579](https://github.com/Tracer-Cloud/opensre/pull/579) fix: wire grafana\_backend into GrafanaTracesTool for synthetic tests (author: vincenthus; contributors: vincenthus; labels: *none*; files: `tools/GrafanaTracesTool/__init__.py`, `tests/synthetic/mock_grafana_backend/backend.py`, `tests/synthetic/mock_grafana_backend/formatters.py`, `tests/synthetic/mock_grafana_backend/selective_backend.py`, `tests/tools/test_grafana_traces_tool.py`) * [#648](https://github.com/Tracer-Cloud/opensre/pull/648) fix: align integrations count to 60+ in README (author: Vadym Petrychenko; contributors: Vadym Petrychenko; labels: *none*; files: `README.md`) * [#576](https://github.com/Tracer-Cloud/opensre/pull/576) feat: add "Run All" option to interactive test picker (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/cli/tests/__init__.py`, `app/cli/tests/interactive.py`, `app/cli/tests/runner.py`, `tests/cli/test_interactive.py`) * [#583](https://github.com/Tracer-Cloud/opensre/pull/583) feat: add Kubernetes synthetic RCA test harness (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `Makefile`, `app/nodes/plan_actions/detect_sources.py`, `tools/DataDogLogsTool/__init__.py`, `tools/DataDogMonitorsTool/__init__.py`, `tools/EKSEventsTool/__init__.py`, `tools/EKSListClustersTool/__init__.py`, `tools/EKSListDeploymentsTool/__init__.py`, `tools/EKSListPodsTool/__init__.py`, `tools/EKSNodeHealthTool/__init__.py`, `tools/EKSPodLogsTool/__init__.py`, and 22 more) * [#625](https://github.com/Tracer-Cloud/opensre/pull/625) fix: Database directives for RDS QA testing (author: ANIRUDDHA ADAK; contributors: ANIRUDDHA ADAK; labels: *none*; files: `app/nodes/root_cause_diagnosis/prompt_builder.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-17T23:00:00+00:00` to `2026-04-18T23:00:00+00:00` # Daily Update — 2026-04-19 Source: https://opensre.com/docs/daily-updates/2026-04-19 OpenSRE engineering daily update for 2026-04-19 (Europe/London) Thanks to everyone who contributed yesterday: ANIRUDDHA ADAK, Ceren Camkiran, chaosreload, Ebrahim Sameh, fuleinist, kaushal-bakrania, Malik Amir Hamza, mayankbharati-ops, paulovitorcl, Shoaib, Shoaib Ansari, vincenthus, weichao, Yash Kapure, and Yash Kumar Saini 🙏🚀 ## Main updates shipped (April 19, 2026) * feat: support multiple accounts, regions, and clusters per provider (#475) — Malik Amir Hamza * feat: RabbitMQ integration for queue and broker RCA (#324) — Yash Kumar Saini * feat: route Bitbucket and add bounded integration-wave slices (#484) — Shoaib * Bugfix: GitHub MPC Configuration #419 — Yash Kapure * fix: overlapping keyword redaction and hardcoded investigation loop limit — mayankbharati-ops * feat: structured retrieval intent in planning with tool controls contract (#480) — Shoaib Ansari * test(synthetic): add 8 Kubernetes RCA scenarios — Malik Amir Hamza * feat(#473): add routing metadata and explainable tool selection — mayankbharati-ops * feat(cli): add hosted deploy support to opensre deploy (#264) — Ceren Camkiran * fix: recognise EKS evidence keys in is\_clearly\_healthy short-circuit (#582) — Ebrahim Sameh * Fix MaskingContext.unmask() prefix collision (#639) — Chris Chen * chore: remove dead code identified by vulture analysis — vincenthus * ci: skip Windows runners on PR builds, run them only on push to main — vincenthus * fix: use typing\_extensions.TypedDict for Python 3.11 / Pydantic 2 compatibility — chaosreload * fix: hide unimplemented DataDogMetricsTool stub from planner (#669) — kaushal-bakrania * synthetic: enforce primary reliance on RDS events in 005-failover (#601) — Ceren Camkiran * fix: Database logic expansion for QA Edge Cases (Batch 3) — ANIRUDDHA ADAK * docs: remove Bash/PowerShell labels from command blocks (#681) — paulovitorcl ## Source pull requests * [#627](https://github.com/Tracer-Cloud/opensre/pull/627) fix: Database logic expansion for QA Edge Cases (Batch 3) (author: ANIRUDDHA ADAK; contributors: ANIRUDDHA ADAK; labels: *none*; files: `app/nodes/root_cause_diagnosis/prompt_builder.py`) * [#649](https://github.com/Tracer-Cloud/opensre/pull/649) synthetic: enforce primary reliance on RDS events in 005-failover (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `tests/synthetic/rds_postgres/005-failover/QA_VALIDATION.md`, `tests/synthetic/rds_postgres/005-failover/answer.yml`, `tests/synthetic/rds_postgres/run_suite.py`) * [#643](https://github.com/Tracer-Cloud/opensre/pull/643) feat: support multiple accounts, regions, and clusters per provider (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `.env.example`, `app/integrations/catalog.py`, `app/integrations/models.py`, `app/integrations/selectors.py`, `app/integrations/store.py`, `app/nodes/plan_actions/detect_sources.py`, `docs/docs.json`, `docs/multi-instance-integrations.mdx`, `tests/cli_smoke_test.py`, `tests/integrations/test_catalog_multi_instance.py`, and 5 more) * [#661](https://github.com/Tracer-Cloud/opensre/pull/661) test(synthetic): add 8 Kubernetes RCA scenarios (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `tests/synthetic/eks/001-oomkilled-crashloop/alert.json`, `tests/synthetic/eks/001-oomkilled-crashloop/answer.yml`, `tests/synthetic/eks/001-oomkilled-crashloop/eks_events.json`, `tests/synthetic/eks/001-oomkilled-crashloop/eks_pod_logs.json`, `tests/synthetic/eks/001-oomkilled-crashloop/eks_pods.json`, `tests/synthetic/eks/001-oomkilled-crashloop/scenario.yml`, `tests/synthetic/eks/002-image-pull-backoff/alert.json`, `tests/synthetic/eks/002-image-pull-backoff/answer.yml`, `tests/synthetic/eks/002-image-pull-backoff/eks_deployments.json`, `tests/synthetic/eks/002-image-pull-backoff/eks_events.json`, and 43 more) * [#520](https://github.com/Tracer-Cloud/opensre/pull/520) fix: overlapping keyword redaction and hardcoded investigation loop limit (author: mayankbharati-ops; contributors: mayankbharati-ops and vincenthus; labels: *none*; files: `app/guardrails/engine.py`, `app/investigation_constants.py`, `app/nodes/root_cause_diagnosis/node.py`, `app/pipeline/routing.py`, `tests/synthetic/rds_postgres/001-replication-lag/answer.yml`, `tests/synthetic/rds_postgres/002-connection-exhaustion/answer.yml`, `tests/synthetic/rds_postgres/003-storage-full/answer.yml`, `tests/synthetic/rds_postgres/004-cpu-saturation-bad-query/answer.yml`, `tests/synthetic/rds_postgres/005-failover/answer.yml`, `tests/synthetic/rds_postgres/006-replication-lag-cpu-redherring/answer.yml`, and 10 more) * [#585](https://github.com/Tracer-Cloud/opensre/pull/585) fix: recognise EKS evidence keys in is\_clearly\_healthy short-circuit (author: Ebrahim Sameh; contributors: Ebrahim Sameh and vincenthus; labels: *none*; files: `app/nodes/root_cause_diagnosis/evidence_checker.py`, `tests/nodes/root_cause_diagnosis/__init__.py`, `tests/nodes/root_cause_diagnosis/test_evidence_checker.py`) * [#645](https://github.com/Tracer-Cloud/opensre/pull/645) Fix MaskingContext.unmask() prefix collision (author: Chris Chen; contributors: Chris Chen and fuleinist; labels: *none*; files: `app/masking/context.py`) * [#501](https://github.com/Tracer-Cloud/opensre/pull/501) feat: route Bitbucket and add bounded integration-wave slices (author: Shoaib; contributors: Shoaib, Shoaib Ansari, and vincenthus; labels: *none*; files: `app/integrations/catalog.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/build_prompt.py`, `app/nodes/plan_actions/detect_sources.py`, `tools/AzureMonitorLogsTool/__init__.py`, `tools/BitbucketCommitsTool/__init__.py`, `tools/BitbucketFileContentsTool/__init__.py`, `tools/BitbucketSearchCodeTool/__init__.py`, `tools/OpenObserveLogsTool/__init__.py`, and 7 more) * [#517](https://github.com/Tracer-Cloud/opensre/pull/517) Bugfix: GitHub MPC Configuration #419 (author: Yash Kapure; contributors: vincenthus and Yash Kapure; labels: *none*; files: `.env.example`, `app/cli/wizard/env_sync.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/cli/wizard/prompts.py`, `app/integrations/cli.py`, `app/integrations/github_mcp.py`, `app/integrations/mcp_streamable_http_compat.py`, `app/nodes/publish_findings/renderers/terminal.py`, `deployment/remote/server.py`, and 11 more) * [#523](https://github.com/Tracer-Cloud/opensre/pull/523) feat(#473): add routing metadata and explainable tool selection (author: mayankbharati-ops; contributors: mayankbharati-ops and vincenthus; labels: *none*; files: `app/nodes/investigate/types.py`, `app/nodes/plan_actions/node.py`, `app/nodes/plan_actions/plan_actions.py`, `tools/CloudWatchBatchMetricsTool/__init__.py`, `tools/DataDogLogsTool/__init__.py`, `tools/SREGuidanceTool/__init__.py`, `tools/investigation_registry/__init__.py`, `tools/investigation_registry/prioritization.py`, `tools/registered_tool.py`, `tools/tool_decorator.py`, and 2 more) * [#615](https://github.com/Tracer-Cloud/opensre/pull/615) feat: RabbitMQ integration for queue and broker RCA (author: Yash Kumar Saini; contributors: vincenthus and Yash Kumar Saini; labels: *none*; files: `.env.example`, `Makefile`, `README.md`, `app/cli/constants.py`, `app/integrations/catalog.py`, `app/integrations/models.py`, `app/integrations/rabbitmq.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/detect_sources.py`, `tools/RabbitMQBrokerOverviewTool/__init__.py`, and 17 more) * [#664](https://github.com/Tracer-Cloud/opensre/pull/664) ci: skip Windows runners on PR builds, run them only on push to main (author: vincenthus; contributors: vincenthus; labels: *none*; files: `.github/workflows/ci.yml`) * [#593](https://github.com/Tracer-Cloud/opensre/pull/593) feat(cli): add hosted deploy support to opensre deploy (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/cli/commands/deploy.py`, hosted deploy helper, `tests/cli/test_deploy.py`, `tests/cli_smoke_test.py`) * [#497](https://github.com/Tracer-Cloud/opensre/pull/497) feat: structured retrieval intent in planning with tool controls contract (author: Shoaib; contributors: Shoaib, Shoaib Ansari, and vincenthus; labels: *none*; files: `app/nodes/plan_actions/node.py`, `app/state/agent_state.py`, `tools/base.py`, `tools/registered_tool.py`, `tools/tool_decorator.py`, `app/types/__init__.py`, `app/types/retrieval.py`, `tests/nodes/plan_actions/test_node_plan_actions.py`, `tests/tools/test_registry.py`, `tests/types/test_retrieval.py`, and 1 more) * [#665](https://github.com/Tracer-Cloud/opensre/pull/665) chore: remove dead code identified by vulture analysis (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/analytics/cli.py`, `app/cli/wizard/store.py`, `config/config.py`, `app/entrypoints/cli.py`, `app/integrations/azure_sql.py`, `app/integrations/mongodb.py`, `app/nodes/publish_findings/formatters/base.py`, `services/env.py`, `services/vercel/client.py`, `tools/registry.py`, and 4 more) * [#671](https://github.com/Tracer-Cloud/opensre/pull/671) fix: hide unimplemented DataDogMetricsTool stub from planner (author: kaushal-bakrania; contributors: kaushal-bakrania; labels: *none*; files: `tools/DataDogMetricsTool/__init__.py`, `tests/tools/test_datadog_metrics_tool.py`) * [#668](https://github.com/Tracer-Cloud/opensre/pull/668) fix: use typing\_extensions.TypedDict for Python 3.11 / Pydantic 2 compatibility (author: chaosreload; contributors: chaosreload and weichao; labels: *none*; files: `app/nodes/investigate/types.py`) * [#683](https://github.com/Tracer-Cloud/opensre/pull/683) docs: remove Bash/PowerShell labels from command blocks (author: paulovitorcl; contributors: paulovitorcl; labels: *none*; files: `docs/quickstart.mdx`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-18T23:00:00+00:00` to `2026-04-19T23:00:00+00:00` # Daily Update — 2026-04-20 Source: https://opensre.com/docs/daily-updates/2026-04-20 OpenSRE engineering daily update for 2026-04-20 (Europe/London) Thanks to everyone who contributed yesterday: Afif Ahmed, ANIRUDDHA ADAK, chaosreload, Ebrahim Sameh, Gaurav, gbsierra, kaushal-bakrania, Malik Amir Hamza, paulovitorcl, Rohit Rajan, Rohit Rajan, skredik, Tan Wee Joe, vincenthus, weichao, and Yash Kumar Saini 🙏🚀 ## Main updates shipped (April 20, 2026) * feat: Better Stack Telemetry (logs) integration (#692) — Afif Ahmed * fix(postgresql): wire is\_available + extract\_params into PostgreSQL @tool decorators (#703) — Ebrahim Sameh * fix(mysql): wire is\_available + extract\_params into MySQL @tool decorators (#704) — Ebrahim Sameh * fix(azure-sql): wire is\_available + extract\_params into Azure SQL @tool decorators (#706) — Ebrahim Sameh * feat: add GitDeployTimelineTool for incident-window deploy correlation (#708) — Malik Amir Hamza * feat(tests): Chaos Mesh lab via Make and tests/chaos\_engineering (#691) — Tan Wee Joe * feat: add VS Code devcontainer support (#682) — skredik * fix: improve RCA prompt for evidence-grounded diagnosis (Fixes #653) (#714) — Gaurav * fix: surface checkpoint and audit\_log keywords in synthetic test fixtures (#607, #610) — Rohit Rajan * test(synthetic): add 5 EKS noise scenarios (#709) — Malik Amir Hamza * fix: use typing\_extensions.TypedDict in remaining app modules (follow-up #668) (#693) — chaosreload * fix: restore RunnableConfig annotation fix regressed by #594 (#695) — Ebrahim Sameh * fix: consolidate pytest config into pytest.ini, stop silently ignoring pyproject (#700) — Ebrahim Sameh * fix(snowflake): rename tool param schema to db\_schema to silence Pydantic warning (#697) — Ebrahim Sameh * chore: consolidate ruff config into ruff.toml, stop silently ignoring pyproject (#701) — Ebrahim Sameh * docs: Better Stack Telemetry integration page (#715) — Afif Ahmed * fix: recognise grafana\_alert\_rules in check\_evidence\_availability (#684) — kaushal-bakrania * docs: sort integrations in alphabetical order in local and enterprise group (#680) — paulovitorcl * docs: scope sign-up button CSS to intended nav/sidebar areas (#679) — gbsierra ## Source pull requests * [#685](https://github.com/Tracer-Cloud/opensre/pull/685) fix: recognise grafana\_alert\_rules in check\_evidence\_availability (#684) (author: kaushal-bakrania; contributors: kaushal-bakrania; labels: *none*; files: `app/nodes/root_cause_diagnosis/evidence_checker.py`) * [#686](https://github.com/Tracer-Cloud/opensre/pull/686) docs: scope sign-up button CSS to intended nav/sidebar areas (author: gbsierra; contributors: gbsierra; labels: *none*; files: `docs/custom.css`) * [#688](https://github.com/Tracer-Cloud/opensre/pull/688) docs: sort integrations in alphabetical order in local and enterprise group (author: paulovitorcl; contributors: paulovitorcl; labels: *none*; files: `docs/docs.json`) * [#696](https://github.com/Tracer-Cloud/opensre/pull/696) fix: restore RunnableConfig annotation fix regressed by #594 (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `app/nodes/resolve_integrations/node.py`) * [#700](https://github.com/Tracer-Cloud/opensre/pull/700) fix: consolidate pytest config into pytest.ini, stop silently ignoring pyproject (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `pyproject.toml`, `pytest.ini`) * [#698](https://github.com/Tracer-Cloud/opensre/pull/698) fix(snowflake): rename tool param schema to db\_schema to silence Pydantic warning (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `tools/SnowflakeQueryHistoryTool/__init__.py`) * [#703](https://github.com/Tracer-Cloud/opensre/pull/703) fix(postgresql): wire is\_available + extract\_params into PostgreSQL @tool decorators (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `app/integrations/postgresql.py`, `tools/PostgreSQLCurrentQueriesTool/__init__.py`, `tools/PostgreSQLReplicationStatusTool/__init__.py`, `tools/PostgreSQLServerStatusTool/__init__.py`, `tools/PostgreSQLSlowQueriesTool/__init__.py`, `tools/PostgreSQLTableStatsTool/__init__.py`) * [#704](https://github.com/Tracer-Cloud/opensre/pull/704) fix(mysql): wire is\_available + extract\_params into MySQL @tool decorators (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `app/integrations/mysql.py`, `tools/MySQLCurrentProcessesTool/__init__.py`, `tools/MySQLReplicationStatusTool/__init__.py`, `tools/MySQLServerStatusTool/__init__.py`, `tools/MySQLSlowQueriesTool/__init__.py`, `tools/MySQLTableStatsTool/__init__.py`) * [#694](https://github.com/Tracer-Cloud/opensre/pull/694) fix: use typing\_extensions.TypedDict in remaining app modules (follow-up #668) (author: chaosreload; contributors: chaosreload and weichao; labels: *none*; files: `app/cli/deploy.py`, `app/cli/tests/discover.py`, `app/integrations/openclaw.py`, `app/nodes/publish_findings/report_context.py`, `app/state/agent_state.py`, `app/state/types.py`) * [#666](https://github.com/Tracer-Cloud/opensre/pull/666) test(synthetic): add 5 EKS noise scenarios (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `tests/synthetic/eks/009-noisy-healthy-restart-recovered/alert.json`, `tests/synthetic/eks/009-noisy-healthy-restart-recovered/answer.yml`, `tests/synthetic/eks/009-noisy-healthy-restart-recovered/datadog_logs.json`, `tests/synthetic/eks/009-noisy-healthy-restart-recovered/datadog_monitors.json`, `tests/synthetic/eks/009-noisy-healthy-restart-recovered/eks_events.json`, `tests/synthetic/eks/009-noisy-healthy-restart-recovered/eks_pods.json`, `tests/synthetic/eks/009-noisy-healthy-restart-recovered/scenario.yml`, `tests/synthetic/eks/010-red-herring-old-rollout/alert.json`, `tests/synthetic/eks/010-red-herring-old-rollout/answer.yml`, `tests/synthetic/eks/010-red-herring-old-rollout/datadog_logs.json`, and 23 more) * [#691](https://github.com/Tracer-Cloud/opensre/pull/691) feat(tests): Chaos Mesh lab via Make and tests/chaos\_engineering (author: Tan Wee Joe; contributors: Tan Wee Joe; labels: *none*; files: `Makefile`, `tests/chaos_engineering/README.md`, `tests/chaos_engineering/__init__.py`, `tests/chaos_engineering/__main__.py`, `tests/chaos_engineering/chaos-demo.yaml`, `tests/chaos_engineering/cli.py`, `tests/chaos_engineering/experiment_ops.py`, `tests/chaos_engineering/experiments/container-kill/container-kill-alert.json`, `tests/chaos_engineering/experiments/container-kill/container-kill-chaos.yaml`, `tests/chaos_engineering/experiments/container-kill/container-kill-demo.yaml`, and 63 more) * [#693](https://github.com/Tracer-Cloud/opensre/pull/693) feat: add VS Code devcontainer support (author: skredik; contributors: skredik and Yash Kumar Saini; labels: *none*; files: `.devcontainer/Dockerfile`, `.devcontainer/devcontainer.json`, `.dockerignore`, `.gitignore`, `CONTRIBUTING.md`, `README.md`, `SETUP.md`, `app/cli/wizard/local_grafana_stack/docker-compose.yml`, `tests/test_devcontainer.py`) * [#708](https://github.com/Tracer-Cloud/opensre/pull/708) chore: consolidate ruff config into ruff.toml, stop silently ignoring pyproject (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `pyproject.toml`) * [#692](https://github.com/Tracer-Cloud/opensre/pull/692) feat: Better Stack Telemetry (logs) integration (author: Afif Ahmed; contributors: Afif Ahmed and Yash Kumar Saini; labels: *none*; files: `.env.example`, `README.md`, `app/cli/constants.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/integrations/betterstack.py`, `app/integrations/catalog.py`, `app/integrations/cli.py`, `app/integrations/models.py`, `app/integrations/verify.py`, and 17 more) * [#707](https://github.com/Tracer-Cloud/opensre/pull/707) fix(azure-sql): wire is\_available + extract\_params into Azure SQL @tool decorators (author: Ebrahim Sameh; contributors: Ebrahim Sameh; labels: *none*; files: `app/integrations/azure_sql.py`, `tools/AzureSQLCurrentQueriesTool/__init__.py`, `tools/AzureSQLResourceStatsTool/__init__.py`, `tools/AzureSQLServerStatusTool/__init__.py`, `tools/AzureSQLSlowQueriesTool/__init__.py`, `tools/AzureSQLWaitStatsTool/__init__.py`, `tests/integrations/test_azure_sql.py`) * [#709](https://github.com/Tracer-Cloud/opensre/pull/709) feat: add GitDeployTimelineTool for incident-window deploy correlation (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `app/nodes/investigate/processing/post_process.py`, `app/nodes/plan_actions/build_prompt.py`, `tools/GitDeployTimelineTool/__init__.py`, `tests/tools/test_git_deploy_timeline_tool.py`) * [#713](https://github.com/Tracer-Cloud/opensre/pull/713) docs: Better Stack Telemetry integration page (author: Afif Ahmed; contributors: Afif Ahmed; labels: *none*; files: `docs/betterstack.mdx`, `docs/docs.json`, `docs/integrations-overview.mdx`) * [#712](https://github.com/Tracer-Cloud/opensre/pull/712) fix: surface checkpoint and audit\_log keywords in synthetic test fixtures (#607, #610) (author: Rohit Rajan; contributors: ANIRUDDHA ADAK, Rohit Rajan, Rohit Rajan, and vincenthus; labels: *none*; files: `app/nodes/plan_actions/build_prompt.py`, `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tools/AzureSQLCurrentQueriesTool/__init__.py`, `tools/AzureSQLResourceStatsTool/__init__.py`, `tools/AzureSQLServerStatusTool/__init__.py`, `tools/AzureSQLSlowQueriesTool/__init__.py`, `tools/AzureSQLWaitStatsTool/__init__.py`, `tools/MariaDBInnoDBStatusTool/__init__.py`, `tools/MariaDBProcessListTool/__init__.py`, `tools/MariaDBReplicationTool/__init__.py`, and 17 more) * [#716](https://github.com/Tracer-Cloud/opensre/pull/716) fix: improve RCA prompt for evidence-grounded diagnosis (Fixes #653) (author: Gaurav; contributors: Gaurav; labels: *none*; files: `app/nodes/root_cause_diagnosis/prompt_builder.py`, `services/llm_client.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-19T23:00:00+00:00` to `2026-04-20T23:00:00+00:00` # Daily Update — 2026-04-21 Source: https://opensre.com/docs/daily-updates/2026-04-21 OpenSRE engineering daily update for 2026-04-21 (Europe/London) Thanks to everyone who contributed yesterday: Abkari Mohammed Sayeem, Ankit Juneja, Ankit Juneja, Dax Patel, Dibyendu Sahoo, Dinesh Jinjala, Dinesh Jinjala, Divij Gera, gbsierra, Matt Van Horn, Som-0619, and Yash Kumar Saini 🙏🚀 ## Main updates shipped (April 21, 2026) * fix: add is\_available and extract\_params to MongoDB, ClickHouse, Kafka, and CloudWatch tools (#730) — Ankit Juneja * fix(detect-sources): populate sources\["azure\_sql"] from resolved integrations (#711) — Yash Kumar Saini * CI: add ruff format --check step to quality job — Divij Gera * chore: align Python baseline on 3.13 across mypy and ruff (#727) — Matt Van Horn * Fix onboarding .env bug when file lacks final newline — gbsierra * fix(sentry): safely parse tracing sample rate fallback — Som-0619 * docs: expand AGENTS.md with repo map, entry points, rules, and new integration checklist (#714) — Dax Patel * test(vercel): add runtime log stream parsing unit tests — Dinesh Jinjala * test(auth): add AsyncJWKSCache TTL unit test to prevent redundant JWKS fetches within cache window — Dibyendu Sahoo * docs: rename DEPLOYEMENT.md to DEPLOYMENT.md — Abkari Mohammed Sayeem ## Source pull requests * [#717](https://github.com/Tracer-Cloud/opensre/pull/717) CI: add ruff format --check step to quality job (author: Divij Gera; contributors: Divij Gera; labels: *none*; files: `.github/workflows/ci.yml`, `CONTRIBUTING.md`, `Makefile`, `app/nodes/publish_findings/formatters/evidence.py`, `app/nodes/root_cause_diagnosis/prompt_builder.py`, `services/llm_client.py`, `tools/AzureSQLCurrentQueriesTool/__init__.py`, `tools/AzureSQLResourceStatsTool/__init__.py`, `tools/AzureSQLServerStatusTool/__init__.py`, `tools/AzureSQLSlowQueriesTool/__init__.py`, and 19 more) * [#725](https://github.com/Tracer-Cloud/opensre/pull/725) docs: expand AGENTS.md with repo map, entry points, rules, and new integration checklist (#714) (author: Dax Patel; contributors: Dax Patel; labels: *none*; files: `AGENTS.md`) * [#719](https://github.com/Tracer-Cloud/opensre/pull/719) Fix onboarding .env bug when file lacks final newline (author: gbsierra; contributors: gbsierra; labels: *none*; files: `app/cli/wizard/env_sync.py`, `tests/cli/wizard/test_env_sync.py`) * [#728](https://github.com/Tracer-Cloud/opensre/pull/728) chore: align Python baseline on 3.13 across mypy and ruff (#727) (author: Matt Van Horn; contributors: Matt Van Horn; labels: *none*; files: `app/integrations/mcp_streamable_http_compat.py`, `mypy.ini`, `ruff.toml`, `tests/deployment/bedrock/conftest.py`, `tests/deployment/ec2/conftest.py`, hosted deployment conftest, `tests/deployment/vercel/conftest.py`, `tests/tools/test_registry.py`) * [#733](https://github.com/Tracer-Cloud/opensre/pull/733) fix(sentry): safely parse tracing sample rate fallback (author: Som-0619; contributors: Som-0619; labels: *none*; files: `utils/sentry_sdk.py`, `tests/test_sentry_init.py`) * [#732](https://github.com/Tracer-Cloud/opensre/pull/732) fix: add is\_available and extract\_params to MongoDB, ClickHouse, Kafka, and CloudWatch tools (author: Ankit Juneja; contributors: Ankit Juneja and Ankit Juneja; labels: *none*; files: `app/integrations/clickhouse.py`, `app/integrations/kafka.py`, `app/integrations/mongodb.py`, `tools/ClickHouseQueryActivityTool/__init__.py`, `tools/ClickHouseSystemHealthTool/__init__.py`, `tools/CloudWatchBatchMetricsTool/__init__.py`, `tools/KafkaConsumerGroupTool/__init__.py`, `tools/KafkaTopicHealthTool/__init__.py`, `tools/MongoDBCollectionStatsTool/__init__.py`, `tools/MongoDBCurrentOpsTool/__init__.py`, and 4 more) * [#722](https://github.com/Tracer-Cloud/opensre/pull/722) fix(detect-sources): populate sources\["azure\_sql"] from resolved integrations (author: Yash Kumar Saini; contributors: Yash Kumar Saini; labels: *none*; files: `app/nodes/plan_actions/detect_sources.py`, `tests/nodes/plan_actions/test_detect_sources_azure_sql.py`) * [#750](https://github.com/Tracer-Cloud/opensre/pull/750) docs: rename DEPLOYEMENT.md to DEPLOYMENT.md (author: Abkari Mohammed Sayeem; contributors: Abkari Mohammed Sayeem; labels: *none*; files: `DEPLOYMENT.md`) * [#749](https://github.com/Tracer-Cloud/opensre/pull/749) test(vercel): add runtime log stream parsing unit tests (author: Dinesh Jinjala; contributors: Dinesh Jinjala and Dinesh Jinjala; labels: *none*; files: `tests/services/vercel/test_client.py`) * [#754](https://github.com/Tracer-Cloud/opensre/pull/754) test(auth): add AsyncJWKSCache TTL unit test to prevent redundant JWKS fetches within cache window (author: Dibyendu Sahoo; contributors: Dibyendu Sahoo; labels: *none*; files: `tests/app/auth/test_jwt_auth.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-20T23:00:00+00:00` to `2026-04-21T23:00:00+00:00` # Daily Update — 2026-04-22 Source: https://opensre.com/docs/daily-updates/2026-04-22 OpenSRE engineering daily update for 2026-04-22 (Europe/London) Thanks to everyone who contributed yesterday: Abkari Mohammed Sayeem, Devesh, Dinesh Jinjala, Dinesh Jinjala, Francisco Angulo de Lafuente, Maharshi Patel, Mateus Scheuer Macedo, Nandana Dileep, qorex#dev, Ryjen1, and Udit 🙏🚀 ## Main updates shipped (April 22, 2026) * fix: use Optional\[RunnableConfig] on all pipeline nodes to silence runtime warning (#773) — Udit * feat(mcp): return stable error shape with error\_type in run\_rca (#746) — Nandana Dileep * fix(auth): narrow Exception catch in JWK parsing (#740) — qorex#dev * feat: add LLM\_MAX\_TOKENS env var override to LLMSettings.from\_env() (#739) — Francisco Angulo de Lafuente * fix: replace placeholder NVIDIA NIM model IDs in config.py (#738) — Maryjane Okafor * fix(posthog): include HTTP status code in validate\_posthog\_config error detail (#745) — Nandana Dileep * fix: handle empty alert\_name in \_make\_id() to avoid trailing underscores (#744) — Dinesh Jinjala * fix: health endpoint exceptions (#742) — Maharshi Patel * test(vercel): add unit tests for \_safe\_vercel\_path\_segment (#747) — Nandana Dileep * Revert "docs: update CONTRIBUTING.md to say OpenSRE" — Devesh * docs: update CONTRIBUTING.md to say OpenSRE (not Tracer) — Mateus Scheuer Macedo ## Source pull requests * [#756](https://github.com/Tracer-Cloud/opensre/pull/756) docs: update CONTRIBUTING.md to say OpenSRE (not Tracer) (author: Mateus Scheuer Macedo; contributors: Mateus Scheuer Macedo; labels: *none*; files: `CONTRIBUTING.md`) * [#762](https://github.com/Tracer-Cloud/opensre/pull/762) fix(auth): narrow Exception catch in JWK parsing (author: qorex#dev; contributors: qorex#dev; labels: *none*; files: `app/auth/jwt_auth.py`, `tests/app/auth/test_jwt_auth.py`) * [#767](https://github.com/Tracer-Cloud/opensre/pull/767) feat: add LLM\_MAX\_TOKENS env var override to LLMSettings.from\_env() (author: Francisco Angulo de Lafuente; contributors: Francisco Angulo de Lafuente; labels: *none*; files: `config/config.py`, `tests/test_config.py`) * [#764](https://github.com/Tracer-Cloud/opensre/pull/764) fix: replace placeholder NVIDIA NIM model IDs in config.py (#738) (author: Maryjane Okafor; contributors: Maryjane Okafor and Ryjen1; labels: *none*; files: `config/config.py`) * [#761](https://github.com/Tracer-Cloud/opensre/pull/761) fix(posthog): include HTTP status code in validate\_posthog\_config error detail (author: Nandana Dileep; contributors: Nandana Dileep; labels: *none*; files: `app/integrations/posthog.py`, `tests/integrations/test_posthog.py`) * [#759](https://github.com/Tracer-Cloud/opensre/pull/759) test(vercel): add unit tests for \_safe\_vercel\_path\_segment (author: Nandana Dileep; contributors: Nandana Dileep; labels: *none*; files: `app/integrations/posthog.py`, `tests/integrations/test_posthog.py`, `tests/services/vercel/test_client.py`, `tests/test_config.py`) * [#760](https://github.com/Tracer-Cloud/opensre/pull/760) feat(mcp): return stable error shape with error\_type in run\_rca (author: Nandana Dileep; contributors: Nandana Dileep; labels: *none*; files: `app/entrypoints/mcp.py`, `app/integrations/posthog.py`, `tests/entrypoints/test_mcp.py`, `tests/integrations/test_posthog.py`, `tests/test_config.py`) * [#753](https://github.com/Tracer-Cloud/opensre/pull/753) fix: handle empty alert\_name in \_make\_id() to avoid trailing underscores (author: Dinesh Jinjala; contributors: Dinesh Jinjala and Dinesh Jinjala; labels: *none*; files: `deployment/remote/server.py`, `tests/remote/test_server_utils.py`) * [#751](https://github.com/Tracer-Cloud/opensre/pull/751) docs: update CONTRIBUTING.md to say OpenSRE (author: Abkari Mohammed Sayeem; contributors: Abkari Mohammed Sayeem; labels: *none*; files: `CONTRIBUTING.md`) * [#775](https://github.com/Tracer-Cloud/opensre/pull/775) Revert "docs: update CONTRIBUTING.md to say OpenSRE" (author: Devesh; contributors: Devesh; labels: *none*; files: *No file list returned.*) * [#774](https://github.com/Tracer-Cloud/opensre/pull/774) fix: health endpoint exceptions (author: Maharshi Patel; contributors: Maharshi Patel; labels: *none*; files: `config/webapp.py`, `tests/test_webapp.py`) * [#776](https://github.com/Tracer-Cloud/opensre/pull/776) fix: use Optional\[RunnableConfig] on all pipeline nodes to silence runtime warning (author: Udit; contributors: Udit; labels: *none*; files: `app/nodes/extract_alert/extract_node.py`, `app/nodes/plan_actions/node.py`, `app/nodes/publish_findings/node.py`, `app/nodes/resolve_integrations/node.py`, `app/nodes/root_cause_diagnosis/node.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-21T23:00:00+00:00` to `2026-04-22T23:00:00+00:00` # Daily Update — 2026-04-23 Source: https://opensre.com/docs/daily-updates/2026-04-23 OpenSRE engineering daily update for 2026-04-23 (Europe/London) Thanks to everyone who contributed yesterday: Anwesh, Ceren Camkiran, chaosreload, gbsierra, mayankbharati-ops, paulovitorcl, RoomWithOutRoof, Sarah Salah, Vaibhav Upreti, weichao, and Yash Kumar Saini 🙏🚀 ## Main updates shipped (April 23, 2026) * feat: add CLI framework base and codex CLI integration (#642) — Anwesh * feat: add telegram integration to delivery messages — paulovitorcl * fix: emit healthy-short-circuit claims from investigation keys only — mayankbharati-ops * Improve investigation planning prompt for discriminating RCA evidence (#654) — Ceren Camkiran * fix(eks): detect EKS source when AWS integration uses IAM user credentials — chaosreload * fix: anchor investigation ID regex and add invalid ID tests — RoomWithOutRoof * Fix(makefile): support Windows — gbsierra * fix(wizard): bind Grafana provisioning relative to compose file — Anwesh * synthetic: enforce strict RDS event timeline reasoning for 005-failover — Ceren Camkiran * fix(mypy): remove stale NewGenericSyntax enable flag — Sarah Salah * docs(dev): align local checks with CI ruff format — Anwesh * fix: use relative provisioning path in local Grafana compose — Yash Kumar Saini * Update CONTRIBUTING.md with contribution guidelines — Anwesh ## Source pull requests * [#755](https://github.com/Tracer-Cloud/opensre/pull/755) feat: add telegram integration to delivery messages (author: paulovitorcl; contributors: paulovitorcl; labels: *none*; files: `app/cli/constants.py`, `app/integrations/catalog.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/nodes/publish_findings/node.py`, `app/state/agent_state.py`, `app/state/factory.py`, `utils/telegram_delivery.py`, `tests/utils/test_telegram_delivery.py`) * [#779](https://github.com/Tracer-Cloud/opensre/pull/779) Update CONTRIBUTING.md with contribution guidelines (author: Anwesh; contributors: Anwesh; labels: *none*; files: `CONTRIBUTING.md`) * [#777](https://github.com/Tracer-Cloud/opensre/pull/777) fix: emit healthy-short-circuit claims from investigation keys only (author: mayankbharati-ops; contributors: mayankbharati-ops; labels: *none*; files: `app/nodes/root_cause_diagnosis/evidence_checker.py`, `app/nodes/root_cause_diagnosis/node.py`, `tests/nodes/root_cause_diagnosis/test_evidence_checker.py`, `tests/nodes/root_cause_diagnosis/test_healthy_short_circuit.py`) * [#715](https://github.com/Tracer-Cloud/opensre/pull/715) Improve investigation planning prompt for discriminating RCA evidence (#654) (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/nodes/plan_actions/build_prompt.py`) * [#786](https://github.com/Tracer-Cloud/opensre/pull/786) fix: use relative provisioning path in local Grafana compose (author: Yash Kumar Saini; contributors: Yash Kumar Saini; labels: *none*; files: `app/cli/wizard/local_grafana_stack/docker-compose.yml`, `tests/test_devcontainer.py`) * [#787](https://github.com/Tracer-Cloud/opensre/pull/787) Revert "fix: use relative provisioning path in local Grafana compose" (author: Yash Kumar Saini; contributors: Yash Kumar Saini; labels: *none*; files: `app/cli/wizard/local_grafana_stack/docker-compose.yml`, `tests/test_devcontainer.py`) * [#785](https://github.com/Tracer-Cloud/opensre/pull/785) fix(wizard): bind Grafana provisioning relative to compose file (author: Anwesh; contributors: Anwesh; labels: *none*; files: `app/cli/wizard/local_grafana_stack/docker-compose.yml`, `tests/test_devcontainer.py`) * [#783](https://github.com/Tracer-Cloud/opensre/pull/783) fix: anchor investigation ID regex and add invalid ID tests (author: RoomWithOutRoof; contributors: RoomWithOutRoof and Vaibhav Upreti; labels: *none*; files: `deployment/remote/server.py`, `tests/remote/test_server_utils.py`) * [#721](https://github.com/Tracer-Cloud/opensre/pull/721) Fix(makefile): support Windows (author: gbsierra; contributors: gbsierra and Vaibhav Upreti; labels: *none*; files: `Makefile`) * [#781](https://github.com/Tracer-Cloud/opensre/pull/781) feat: add CLI framework base and codex CLI integration (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.env.example`, `.gitignore`, `AGENTS.md`, `app/cli/wizard/config.py`, `app/cli/wizard/env_sync.py`, `app/cli/wizard/flow.py`, `config/config.py`, `app/integrations/llm_cli/AGENTS.md`, `app/integrations/llm_cli/__init__.py`, `app/integrations/llm_cli/base.py`, and 11 more) * [#705](https://github.com/Tracer-Cloud/opensre/pull/705) synthetic: enforce strict RDS event timeline reasoning for 005-failover (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `tests/fixtures/airflow_alert.json`, `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tests/synthetic/rds_postgres/005-failover/answer.yml`, `tests/synthetic/rds_postgres/run_suite.py`) * [#789](https://github.com/Tracer-Cloud/opensre/pull/789) docs(dev): align local checks with CI ruff format (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.cursor/rules/code-style.mdc`, `AGENTS.md`, `CONTRIBUTING.md`, `Makefile`) * [#724](https://github.com/Tracer-Cloud/opensre/pull/724) fix(eks): detect EKS source when AWS integration uses IAM user credentials (author: chaosreload; contributors: chaosreload and weichao; labels: *none*; files: `app/nodes/plan_actions/detect_sources.py`) * [#798](https://github.com/Tracer-Cloud/opensre/pull/798) fix(mypy): remove stale NewGenericSyntax enable flag (author: Sarah Salah; contributors: Sarah Salah; labels: *none*; files: `mypy.ini`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-22T23:00:00+00:00` to `2026-04-23T23:00:00+00:00` # Daily Update — 2026-04-24 Source: https://opensre.com/docs/daily-updates/2026-04-24 OpenSRE engineering daily update for 2026-04-24 (Europe/London) Thanks to everyone who contributed yesterday: Adeoluwa Daniel Ademoye, ANIRUDDHA ADAK, Anwesh, Awokoya Olawale Davidson, Chris Chen, DIPSHA DAS, Het Patel, Jerome Wilson, John Zakkam, Just Harold, Kartik Sirohi, MichaelGurevich, Paarth, Ryjen1, S. B. | Software Developer, Tan Wee Joe, Vaibhav Upreti, vincenthus, Wahaj Ahmed, and Wahaj Ahmed 🙏🚀 ## Main updates shipped (April 24, 2026) * Feature: Add OpenSRE hugging face dataset integration & eval (#752) — Tan Wee Joe * Fix/restore readable investigation output after masking 479 — Adeoluwa Daniel Ademoye * fix: Database logic expansion for QA Edge Cases (Batch 2) — ANIRUDDHA ADAK * ci: speed up workflow with uv caching + single fast job for PRs (Fixes #647) — S. B. | Software Developer * test(integrations): add direct unit tests for grafana\_wire\_format.pytest(integrations): add direct unit tests for grafana\_wire\_format.py — Awokoya Olawale Davidson * feat: add direct unit tests for health\_view rendering helpers — Maryjane Okafor * test: add direct unit tests for data validation (#831) — Kartik Sirohi * test(tools): consolidate PrefectWorkerHealthTool tests — Het Patel * test(plan-actions): cover extract\_keywords behavior — MichaelGurevich * fix: remove redundant NewGenericSyntax from mypy.ini — Chris Chen * Add unit tests for availability helpers #830 — DIPSHA DAS * test: add comprehensive coverage for invalid investigation IDs with newline characters — Jerome Wilson * test: deduplicate Dockerfile guard tests — John Zakkam * test: remove inline PrefectFlowRunsTool test file after migration — Paarth * chore(github): edit PR template to add a demo/screenshot for the feature/bugs — Anwesh * Fix #824: Correct typo in docs filename and references — Wahaj Ahmed * Fix: Fix broken issue template links in CONTRIBUTING.md — Just Harold ## Source pull requests * [#814](https://github.com/Tracer-Cloud/opensre/pull/814) test: add comprehensive coverage for invalid investigation IDs with newline characters (author: Jerome Wilson; contributors: Jerome Wilson; labels: *none*; files: `tests/remote/test_server_utils.py`) * [#752](https://github.com/Tracer-Cloud/opensre/pull/752) Feature: Add OpenSRE hugging face dataset integration & eval (author: Tan Wee Joe; contributors: Tan Wee Joe; labels: *none*; files: `Makefile`, `app/cli/args.py`, `app/cli/commands/general.py`, `app/cli/investigate.py`, `app/integrations/opensre/__init__.py`, `app/integrations/opensre/constants.py`, `app/integrations/opensre/csv_grafana_backend.py`, `app/integrations/opensre/grafana_wire_format.py`, `app/integrations/opensre/hf_remote.py`, `app/integrations/opensre/inject.py`, and 26 more) * [#845](https://github.com/Tracer-Cloud/opensre/pull/845) Fix: Fix broken issue template links in CONTRIBUTING.md (author: Just Harold; contributors: Just Harold; labels: *none*; files: `CONTRIBUTING.md`) * [#853](https://github.com/Tracer-Cloud/opensre/pull/853) test(tools): consolidate PrefectWorkerHealthTool tests (author: Het Patel; contributors: Het Patel; labels: *none*; files: `tools/PrefectWorkerHealthTool/tool_test.py`, `tests/tools/test_prefect_worker_health_tool.py`) * [#851](https://github.com/Tracer-Cloud/opensre/pull/851) feat: add direct unit tests for health\_view rendering helpers (author: Maryjane Okafor; contributors: Maryjane Okafor and Ryjen1; labels: *none*; files: `tests/cli/test_health_view.py`) * [#854](https://github.com/Tracer-Cloud/opensre/pull/854) Add unit tests for availability helpers #830 (author: DIPSHA DAS; contributors: DIPSHA DAS; labels: *none*; files: `tests/tools/test_availability.py`) * [#852](https://github.com/Tracer-Cloud/opensre/pull/852) test: add direct unit tests for data validation (#831) (author: Kartik Sirohi; contributors: Kartik Sirohi; labels: *none*; files: `tests/tools/utils/test_data_validation.py`) * [#848](https://github.com/Tracer-Cloud/opensre/pull/848) test: deduplicate Dockerfile guard tests (author: John Zakkam; contributors: John Zakkam; labels: *none*; files: `app/dockerfile_test.py`) * [#856](https://github.com/Tracer-Cloud/opensre/pull/856) test: remove inline PrefectFlowRunsTool test file after migration (author: Paarth; contributors: Paarth; labels: *none*; files: `tools/PrefectFlowRunsTool/tool_test.py`) * [#857](https://github.com/Tracer-Cloud/opensre/pull/857) Fix #824: Correct typo in docs filename and references (author: Wahaj Ahmed; contributors: Wahaj Ahmed and Wahaj Ahmed; labels: *none*; files: `docs/daily-updates/2026-04-09.mdx`, `docs/daily-updates/2026-04-13.mdx`, `docs/daily-updates/2026-04-17.mdx`, `docs/daily-updates/2026-04-20.mdx`, `docs/docs.json`, `docs/integrations-overview.mdx`) * [#626](https://github.com/Tracer-Cloud/opensre/pull/626) fix: Database logic expansion for QA Edge Cases (Batch 2) (author: ANIRUDDHA ADAK; contributors: ANIRUDDHA ADAK, Vaibhav Upreti, and vincenthus; labels: *none*; files: `app/nodes/root_cause_diagnosis/prompt_builder.py`) * [#802](https://github.com/Tracer-Cloud/opensre/pull/802) fix: remove redundant NewGenericSyntax from mypy.ini (author: Chris Chen; contributors: Chris Chen; labels: *none*; files: `mypy.ini`) * [#807](https://github.com/Tracer-Cloud/opensre/pull/807) Fix/restore readable investigation output after masking 479 (author: Adeoluwa Daniel Ademoye; contributors: Adeoluwa Daniel Ademoye and Vaibhav Upreti; labels: *none*; files: `Makefile`, `app/nodes/plan_actions/node.py`, `app/nodes/root_cause_diagnosis/prompt_builder.py`) * [#860](https://github.com/Tracer-Cloud/opensre/pull/860) test(plan-actions): cover extract\_keywords behavior (author: MichaelGurevich; contributors: MichaelGurevich and MichaelGurevich; labels: *none*; files: `app/nodes/plan_actions/extract_keywords.py`, `tests/nodes/plan_actions/test_extract_keywords.py`) * [#820](https://github.com/Tracer-Cloud/opensre/pull/820) chore(github): edit PR template to add a demo/screenshot for the feature/bugs (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/PULL_REQUEST_TEMPLATE.md`) * [#817](https://github.com/Tracer-Cloud/opensre/pull/817) ci: speed up workflow with uv caching + single fast job for PRs (Fixes #647) (author: S. B. | Software Developer; contributors: S. B. | Software Developer; labels: *none*; files: `.github/workflows/ci.yml`, `README.md`) * [#862](https://github.com/Tracer-Cloud/opensre/pull/862) test(integrations): add direct unit tests for grafana\_wire\_format.pytest(integrations): add direct unit tests for grafana\_wire\_format.py (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `tests/integrations/test_grafana_wire_format.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-23T23:00:00+00:00` to `2026-04-24T23:00:00+00:00` # Daily Update — 2026-04-25 Source: https://opensre.com/docs/daily-updates/2026-04-25 OpenSRE engineering daily update for 2026-04-25 (Europe/London) Thanks to everyone who contributed yesterday: ANIRUDDHA ADAK, Aryan Jain, GoDiao, Matt Van Horn, mayankbharati-ops, Turan Buyukkamaci, Vaibhav Upreti, and vignesh Gopikrishnan 🙏🚀 ## Main updates shipped (April 25, 2026) * test(services): add direct unit tests for s3\_client.py (#886) — GoDiao * test: add unit tests for TracerPipelinesMixin (#888) — Turan Buyukkamaci * refactor(wizard): migrate Slack webhook validation from requests to httpx — vignesh Gopikrishnan * docs: standardize package manager, deduplicate assets, and sync README — ANIRUDDHA ADAK * test(cli): move local\_llm\_test.py from app/ to tests/cli/ (#900) — Matt Van Horn * test(plan\_actions): repair \_InputStub after model\_copy/model\_dump usage — mayankbharati-ops * test: add unit tests for app/cli/context.py — Aryan Jain * docs: add contributor note about test locations — Aryan Jain ## Source pull requests * [#929](https://github.com/Tracer-Cloud/opensre/pull/929) test(plan\_actions): repair \_InputStub after model\_copy/model\_dump usage (author: mayankbharati-ops; contributors: mayankbharati-ops; labels: *none*; files: `tests/nodes/plan_actions/test_node_plan_actions.py`) * [#931](https://github.com/Tracer-Cloud/opensre/pull/931) test: add unit tests for app/cli/context.py (author: Aryan Jain; contributors: Aryan Jain; labels: *none*; files: `tests/cli/test_context.py`) * [#933](https://github.com/Tracer-Cloud/opensre/pull/933) docs: add contributor note about test locations (author: Aryan Jain; contributors: Aryan Jain; labels: *none*; files: `AGENTS.md`, `CONTRIBUTING.md`) * [#943](https://github.com/Tracer-Cloud/opensre/pull/943) test(cli): move local\_llm\_test.py from app/ to tests/cli/ (#900) (author: Matt Van Horn; contributors: Matt Van Horn; labels: *none*; files: `tests/cli/test_local_llm.py`) * [#924](https://github.com/Tracer-Cloud/opensre/pull/924) docs: standardize package manager, deduplicate assets, and sync README (author: ANIRUDDHA ADAK; contributors: ANIRUDDHA ADAK; labels: *none*; files: `README.md`, `docs/docs.json`, `docs/images/createaccount(1).png`, `docs/images/createworkspace(1).png`, `docs/images/createworkspace-1.png`, `docs/images/fonts/BrittiSansTrial-Light-BF6757bfd494951.otf`, `docs/images/fonts/BrittiSansTrial-LightItalic-BF6757bfd48c7c7.otf`, `docs/images/fonts/BrittiSansTrial-Regular-BF6757bfd47ffbf.otf`, `docs/images/fonts/BrittiSansTrial-Regular-BF6757bfd47ffbf.woff2`, `docs/images/fonts/BrittiSansTrial-RegularItalic-BF6757bfd44e013.otf`, and 4 more) * [#953](https://github.com/Tracer-Cloud/opensre/pull/953) test(services): add direct unit tests for s3\_client.py (#886) (author: GoDiao; contributors: GoDiao; labels: *none*; files: `tests/services/test_s3_client.py`) * [#957](https://github.com/Tracer-Cloud/opensre/pull/957) refactor(wizard): migrate Slack webhook validation from requests to httpx (author: vignesh Gopikrishnan; contributors: Vaibhav Upreti and vignesh Gopikrishnan; labels: *none*; files: `app/cli/wizard/integration_health.py`, `tests/cli/wizard/test_integration_health.py`) * [#950](https://github.com/Tracer-Cloud/opensre/pull/950) test: add unit tests for TracerPipelinesMixin (#888) (author: Turan Buyukkamaci; contributors: Turan Buyukkamaci; labels: *none*; files: `tests/services/tracer_client/test_tracer_pipelines.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-24T23:00:00+00:00` to `2026-04-25T23:00:00+00:00` # Daily Update — 2026-04-26 Source: https://opensre.com/docs/daily-updates/2026-04-26 OpenSRE engineering daily update for 2026-04-26 (Europe/London) Thanks to everyone who contributed yesterday: no human contributors recorded in merged PRs today 🙏🚀 ## Main updates shipped (April 26, 2026) * No pull requests were merged into `main` today. ## Source pull requests * No pull requests were merged during this London calendar day. ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `yes` * UTC window: `2026-04-25T23:00:00+00:00` to `2026-04-26T23:00:00+00:00` # Daily Update — 2026-04-27 Source: https://opensre.com/docs/daily-updates/2026-04-27 OpenSRE engineering daily update for 2026-04-27 (Europe/London) Thanks to everyone who contributed yesterday: Aarush Sharma, Aarush Sharma, Anwesh, Awokoya Olawale Davidson, Blut-agent, DevNinja, Ghraven, Gingiris, hruico, Ibocata, Jeel3011, Kevin Espiñeira, Lozsku, Malik Amir Hamza, Piyush Tiwari, Ryjen1, ShivaniNR, Vaibhav Upreti, vignesh Gopikrishnan, and Yash Kumar Saini 🙏🚀 ## Main updates shipped (April 27, 2026) * feat: add IncidentWindow foundation for shared incident time (#951) — Malik Amir Hamza * Refactor: Parallelize investigation dispatch — Jeel3011 * docs(azure-monitor): add Azure Monitor integration guide — DevNinja * feat(llm-cli): modularize CLI binary resolution — Anwesh * fix: stop run\_diagnostic\_code from crashing every investigation dispatch — Aarush Sharma * test: add unit tests for AWSBatchJobsMixin and fix timezone bug — Kevin Espiñeira * Fix :Add deprecation warning for prefect integration — Ibocata * fix(CI): lazy-load GitHub MCP in CLI setup — Anwesh * test(tools): add unit tests for Azure Monitor Logs tool — vignesh Gopikrishnan * fix: add missing **init**.py to tests/integrations and tests/nodes packages — Ghraven * refactor(tests): move detect\_sources GitLab tests into tests/nodes/plan\_actions/ — Awokoya Olawale Davidson * test(tracer): add unit tests for get\_logs parameter construction — Piyush Tiwari * Fix: Moved pwhtool tests into tests dir @hruico — hruico * docs: replace stale Mintlify starter content — Maryjane Okafor * test(nodes): move publish\_findings node\_test.py into tests/ tree — Lozsku * test(llm-cli): fix Windows CI codex path assertions — Anwesh * docs: add Table of Contents to README — Gingiris * test(tools): move log\_compaction\_test.py into tests/ tree — Lozsku * fix: correct misleading FileNotFoundError message in example — Blut-agent * fix: rename tests/AGENTS.MD to tests/AGENTS.md — ShivaniNR ## Source pull requests * [#979](https://github.com/Tracer-Cloud/opensre/pull/979) fix: rename tests/AGENTS.MD to tests/AGENTS.md (author: ShivaniNR; contributors: ShivaniNR; labels: *none*; files: `tests/AGENTS.md`) * [#951](https://github.com/Tracer-Cloud/opensre/pull/951) feat: add IncidentWindow foundation for shared incident time (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `app/incident_window.py`, `app/nodes/extract_alert/extract_node.py`, `app/state/agent_state.py`, `tests/app/test_incident_window.py`, `tests/app/test_incident_window_cloudwatch_depth.py`, `tests/app/test_incident_window_extract_wiring.py`) * [#999](https://github.com/Tracer-Cloud/opensre/pull/999) docs(azure-monitor): add Azure Monitor integration guide (author: DevNinja; contributors: DevNinja; labels: *none*; files: `docs/azure-monitor.mdx`, `docs/docs.json`) * [#1004](https://github.com/Tracer-Cloud/opensre/pull/1004) fix: correct misleading FileNotFoundError message in example (author: Blut-agent; contributors: Blut-agent; labels: *none*; files: `tests/AGENTS.md`) * [#1000](https://github.com/Tracer-Cloud/opensre/pull/1000) fix: add missing **init**.py to tests/integrations and tests/nodes packages (author: Ghraven; contributors: Ghraven; labels: *none*; files: `tests/integrations/__init__.py`, `tests/nodes/__init__.py`) * [#1006](https://github.com/Tracer-Cloud/opensre/pull/1006) test: add unit tests for AWSBatchJobsMixin and fix timezone bug (author: Kevin Espiñeira; contributors: Kevin Espiñeira; labels: *none*; files: `services/tracer_client/aws_batch_jobs.py`, `tests/services/tracer_client/test_aws_batch_jobs.py`) * [#795](https://github.com/Tracer-Cloud/opensre/pull/795) feat(llm-cli): modularize CLI binary resolution (author: Anwesh; contributors: Anwesh; labels: *none*; files: `app/integrations/llm_cli/AGENTS.md`, `app/integrations/llm_cli/binary_resolver.py`, `app/integrations/llm_cli/codex.py`, `tests/integrations/llm_cli/conftest.py`, `tests/integrations/llm_cli/test_codex_adapter.py`) * [#988](https://github.com/Tracer-Cloud/opensre/pull/988) fix: stop run\_diagnostic\_code from crashing every investigation dispatch (author: Aarush Sharma; contributors: Aarush Sharma and Aarush Sharma; labels: *none*; files: `tools/run_diagnostic_code.py`, `tests/tools/test_run_diagnostic_code_tool.py`) * [#964](https://github.com/Tracer-Cloud/opensre/pull/964) test(tools): move log\_compaction\_test.py into tests/ tree (author: Lozsku; contributors: Lozsku; labels: *none*; files: `tests/tools/test_log_compaction.py`) * [#983](https://github.com/Tracer-Cloud/opensre/pull/983) test(tracer): add unit tests for get\_logs parameter construction (author: Piyush Tiwari; contributors: Piyush Tiwari; labels: *none*; files: `tests/services/tracer_client/test_tracer_logs.py`) * [#847](https://github.com/Tracer-Cloud/opensre/pull/847) Fix: Moved pwhtool tests into tests dir @hruico (author: hruico; contributors: hruico and Vaibhav Upreti; labels: *none*; files: `tests/tools/test_prefect_worker_health_tool.py`) * [#937](https://github.com/Tracer-Cloud/opensre/pull/937) Fix :Add deprecation warning for prefect integration (author: Ibocata; contributors: Ibocata; labels: *none*; files: `app/integrations/clients/prefect/__init__.py`) * [#936](https://github.com/Tracer-Cloud/opensre/pull/936) Refactor: Parallelize investigation dispatch (author: Jeel3011; contributors: Jeel3011; labels: *none*; files: `app/nodes/__init__.py`, `app/nodes/investigate/__init__.py`, `app/nodes/investigate/merge.py`, `app/nodes/investigate/parallel.py`, `app/pipeline/graph.py`, `app/pipeline/routing.py`, `app/state/agent_state.py`, `tests/nodes/test_parallel_investigate.py`) * [#965](https://github.com/Tracer-Cloud/opensre/pull/965) test(nodes): move publish\_findings node\_test.py into tests/ tree (author: Lozsku; contributors: Lozsku; labels: *none*; files: `tests/nodes/publish_findings/__init__.py`, `tests/nodes/publish_findings/test_node.py`) * [#959](https://github.com/Tracer-Cloud/opensre/pull/959) refactor(tests): move detect\_sources GitLab tests into tests/nodes/plan\_actions/ (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `app/nodes/plan_actions/detect_sources_test.py`, `tests/nodes/plan_actions/test_detect_sources.py`) * [#1009](https://github.com/Tracer-Cloud/opensre/pull/1009) test(llm-cli): fix Windows CI codex path assertions (author: Anwesh; contributors: Anwesh; labels: *none*; files: `tests/integrations/llm_cli/test_codex_adapter.py`) * [#1011](https://github.com/Tracer-Cloud/opensre/pull/1011) fix(CI): lazy-load GitHub MCP in CLI setup (author: Anwesh; contributors: Anwesh; labels: *none*; files: `app/integrations/cli.py`, `tests/cli/test_integrations_setup_github.py`) * [#1012](https://github.com/Tracer-Cloud/opensre/pull/1012) docs: add Table of Contents to README (author: Gingiris; contributors: Gingiris; labels: *none*; files: `README.md`) * [#1010](https://github.com/Tracer-Cloud/opensre/pull/1010) docs: replace stale Mintlify starter content (author: Maryjane Okafor; contributors: Maryjane Okafor and Ryjen1; labels: *none*; files: `Makefile`, `docs/README.md`) * [#1007](https://github.com/Tracer-Cloud/opensre/pull/1007) test(tools): add unit tests for Azure Monitor Logs tool (author: vignesh Gopikrishnan; contributors: vignesh Gopikrishnan and Yash Kumar Saini; labels: *none*; files: `tests/tools/test_azure_monitor_logs_tool.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-26T23:00:00+00:00` to `2026-04-27T23:00:00+00:00` # Daily Update — 2026-04-28 Source: https://opensre.com/docs/daily-updates/2026-04-28 OpenSRE engineering daily update for 2026-04-28 (Europe/London) Thanks to everyone who contributed yesterday: Ankush, Anwesh, Aryan Jain, Awokoya Olawale Davidson, Ceren Camkiran, chaosreload, DevNinja, DIPSHA DAS, Ghraven, GoDiao, Irfan Ansari, John Zakkam, Kartik Sirohi, Malik Amir Hamza, Mayank Bharati, Piyush Tiwari, rameshkumarkoyya, Udit, vignesh Gopikrishnan, and weichao 🙏🚀 ## Main updates shipped (April 28, 2026) * Incident window tool integration (#1036) — Malik Amir Hamza * fix(eks): forward stored AWS integration credentials to k8s client builder — chaosreload * refactor(utils): extract shared HTTP-post helper for delivery modules — Mayank Bharati * refactor(publish): extract GitLab MR write-back into dedicated helper module — Udit * refactor(tools): extract shared default\_db\_warning helper for SQL too… — Awokoya Olawale Davidson * synthetic: prevent overdiagnosis in noisy-but-healthy scenario (007) — Ceren Camkiran * synthetic: tighten validation and fix RCA reasoning for replication lag red-herring scenario (006) — Ceren Camkiran * test(tools): add unit tests for the four GitLab tools — DevNinja * refactor(tools): centralize code-host unavailable payload for code-host tools — vignesh Gopikrishnan * fix: move app/cli/deploy\_test.py into tests/cli/ — Ghraven * refactor: move gitlab integration tests to tests/integrations (#901) — Kartik Sirohi * test(utils): add direct unit tests for ingest\_delivery — DevNinja * test: add direct unit tests for services/honeycomb/client.py — Aryan Jain * Test/add lambda client unit tests — Aryan Jain * test(services): add direct unit tests for cloudwatch\_client — GoDiao * Add tracer integration service tests — rameshkumarkoyya * test(cli): add regression test for command/help sync, fix missing guardrails entry — Awokoya Olawale Davidson * test: add direct unit tests for Mimir mixin (#881) — Piyush Tiwari * Add direct unit tests for services/tracer\_client/tracer\_tools.py — Aryan Jain * docs(readme): sync integration matrix with shipped integrations — Ankush ## Source pull requests * [#952](https://github.com/Tracer-Cloud/opensre/pull/952) refactor(utils): extract shared HTTP-post helper for delivery modules (author: Mayank Bharati; contributors: Mayank Bharati; labels: *none*; files: `utils/delivery_transport.py`, `utils/discord_delivery.py`, `utils/slack_delivery.py`, `utils/telegram_delivery.py`, `tests/utils/test_delivery_transport.py`, `tests/utils/test_discord_delivery.py`, `tests/utils/test_slack_delivery.py`, `tests/utils/test_telegram_delivery.py`) * [#975](https://github.com/Tracer-Cloud/opensre/pull/975) Add direct unit tests for services/tracer\_client/tracer\_tools.py (author: Aryan Jain; contributors: Aryan Jain; labels: *none*; files: `tests/services/tracer_client/test_tracer_tools.py`) * [#960](https://github.com/Tracer-Cloud/opensre/pull/960) extended RailwayRemoteOpsProvider tests #907 (author: DIPSHA DAS; contributors: DIPSHA DAS; labels: *none*; files: `tests/remote/test_ops.py`) * [#1018](https://github.com/Tracer-Cloud/opensre/pull/1018) fix: move app/cli/deploy\_test.py into tests/cli/ (author: Ghraven; contributors: Ghraven; labels: *none*; files: `app/cli/deploy_test.py`, `tests/cli/test_deploy.py`) * [#1021](https://github.com/Tracer-Cloud/opensre/pull/1021) test(utils): add direct unit tests for ingest\_delivery (author: DevNinja; contributors: DevNinja; labels: *none*; files: `tests/utils/test_ingest_delivery.py`) * [#859](https://github.com/Tracer-Cloud/opensre/pull/859) test(cli): add direct unit coverage for args helpers (author: John Zakkam; contributors: John Zakkam; labels: *none*; files: `tests/cli/test_args.py`) * [#849](https://github.com/Tracer-Cloud/opensre/pull/849) test: move poll\_deployment\_health tests out of e2e (author: John Zakkam; contributors: John Zakkam; labels: *none*; files: `tests/test_deployment_health.py`) * [#917](https://github.com/Tracer-Cloud/opensre/pull/917) refactor: move gitlab integration tests to tests/integrations (#901) (author: Kartik Sirohi; contributors: Kartik Sirohi; labels: *none*; files: `tests/integrations/test_gitlab.py`) * [#1026](https://github.com/Tracer-Cloud/opensre/pull/1026) refactor(publish): extract GitLab MR write-back into dedicated helper module (author: Udit; contributors: Udit; labels: *none*; files: `app/nodes/publish_findings/gitlab_writeback.py`, `app/nodes/publish_findings/node.py`, `tests/nodes/publish_findings/test_gitlab_writeback.py`, `tests/nodes/publish_findings/test_node.py`) * [#1025](https://github.com/Tracer-Cloud/opensre/pull/1025) test(tools): add unit tests for ClickHouseQueryActivity and ClickHous… (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `tests/tools/test_clickhouse_query_activity_tool.py`, `tests/tools/test_clickhouse_system_health_tool.py`) * [#1003](https://github.com/Tracer-Cloud/opensre/pull/1003) test(tools): add unit tests for the four GitLab tools (author: DevNinja; contributors: DevNinja; labels: *none*; files: `tools/GitLabMRsTool/__init__.py`, `tools/GitLabPipelinesTool/__init__.py`, `tests/tools/test_gitlab_commits_tool.py`, `tests/tools/test_gitlab_file_tool.py`, `tests/tools/test_gitlab_mrs_tool.py`, `tests/tools/test_gitlab_pipelines_tool.py`) * [#1027](https://github.com/Tracer-Cloud/opensre/pull/1027) refactor(tools): extract shared default\_db\_warning helper for SQL too… (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `tools/AzureSQLCurrentQueriesTool/__init__.py`, `tools/AzureSQLSlowQueriesTool/__init__.py`, `tools/MariaDBProcessListTool/__init__.py`, `tools/MySQLCurrentProcessesTool/__init__.py`, `tools/PostgreSQLCurrentQueriesTool/__init__.py`, `tools/PostgreSQLSlowQueriesTool/__init__.py`, `tools/utils/__init__.py`, `tools/utils/db_warnings.py`, `tests/tools/test_azure_sql_current_queries_tool.py`, `tests/tools/test_azure_sql_slow_queries_tool.py`, and 5 more) * [#974](https://github.com/Tracer-Cloud/opensre/pull/974) test: add direct unit tests for services/honeycomb/client.py (author: Aryan Jain; contributors: Aryan Jain; labels: *none*; files: `tests/services/test_honeycomb_client.py`) * [#954](https://github.com/Tracer-Cloud/opensre/pull/954) Incident window tool integration (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `app/incident_window.py`, `app/nodes/extract_alert/extract_node.py`, `app/nodes/investigate/models.py`, `app/nodes/plan_actions/build_prompt.py`, `app/nodes/plan_actions/plan_actions.py`, `app/state/agent_state.py`, `tools/GitDeployTimelineTool/__init__.py`, `tests/app/test_incident_window.py`, `tests/app/test_incident_window_cloudwatch_depth.py`, `tests/app/test_incident_window_extract_wiring.py`, and 2 more) * [#1029](https://github.com/Tracer-Cloud/opensre/pull/1029) test(cli): add direct unit tests for ollama lifecycle helpers (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `tests/cli/test_local_llm_ollama_lifecycle.py`) * [#863](https://github.com/Tracer-Cloud/opensre/pull/863) test(cli): add regression test for command/help sync, fix missing guardrails entry (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `app/cli/layout.py`, `tests/cli/test_command_help_sync.py`) * [#1015](https://github.com/Tracer-Cloud/opensre/pull/1015) test(cli): add direct unit tests for CLI layout renderers (author: vignesh Gopikrishnan; contributors: vignesh Gopikrishnan; labels: *none*; files: `app/cli/layout.py`, `tests/cli/test_layout.py`) * [#1028](https://github.com/Tracer-Cloud/opensre/pull/1028) Test/add lambda client unit tests (author: Aryan Jain; contributors: Aryan Jain; labels: *none*; files: `tests/services/test_lambda_client.py`) * [#1032](https://github.com/Tracer-Cloud/opensre/pull/1032) test(services): add direct unit tests for cloudwatch\_client (author: GoDiao; contributors: GoDiao; labels: *none*; files: `tests/services/test_cloudwatch_client.py`) * [#1035](https://github.com/Tracer-Cloud/opensre/pull/1035) Add tracer integration service tests (author: rameshkumarkoyya; contributors: Irfan Ansari and rameshkumarkoyya; labels: *none*; files: `tests/services/tracer_client/test_tracer_integrations.py`) * [#1038](https://github.com/Tracer-Cloud/opensre/pull/1038) chore: smoke test post-merge celebration workflow (author: Anwesh; contributors: Anwesh; labels: *none*; files: `README.md`) * [#1037](https://github.com/Tracer-Cloud/opensre/pull/1037) refactor(tools): centralize code-host unavailable payload for code-host tools (author: vignesh Gopikrishnan; contributors: vignesh Gopikrishnan; labels: *none*; files: `tools/BitbucketSearchCodeTool/__init__.py`, `tools/GitHubFileContentsTool/__init__.py`, `tools/GitHubSearchCodeTool/__init__.py`, `tools/GitLabFileTool/__init__.py`, `tools/utils/__init__.py`, `tools/utils/code_host_unavailable.py`, `tests/tools/test_bitbucket_search_code_tool.py`, `tests/tools/test_code_host_unavailable.py`, `tests/tools/test_github_file_contents_tool.py`, `tests/tools/test_github_search_code_tool.py`, and 1 more) * [#1039](https://github.com/Tracer-Cloud/opensre/pull/1039) chore: spice up post-merge celebration messages (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/scripts/merged_pr_celebration_message.py`) * [#1040](https://github.com/Tracer-Cloud/opensre/pull/1040) test: add direct unit tests for Mimir mixin (#881) (author: Piyush Tiwari; contributors: Piyush Tiwari; labels: *none*; files: `tests/services/test_grafana_mimir.py`) * [#1045](https://github.com/Tracer-Cloud/opensre/pull/1045) fix(gifs): host celebration GIFs in repo to fix GitHub proxy rendering (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/assets/celebrations/celebrate.gif`, `.github/assets/celebrations/fireworks.gif`, `.github/assets/celebrations/party.gif`, `.github/assets/celebrations/ship.gif`, `.github/assets/celebrations/shipped.gif`, `.github/assets/celebrations/success.gif`, `.github/scripts/merged_pr_celebration_message.py`) * [#1047](https://github.com/Tracer-Cloud/opensre/pull/1047) Update comment for post-merge PR comment workflow test (author: Anwesh; contributors: Anwesh; labels: *none*; files: `README.md`) * [#1049](https://github.com/Tracer-Cloud/opensre/pull/1049) fix(github): add woohoo.gif and polish post-merge celebration messages (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/assets/celebrations/woohoo.gif`, `.github/scripts/merged_pr_celebration_message.py`) * [#978](https://github.com/Tracer-Cloud/opensre/pull/978) synthetic: prevent overdiagnosis in noisy-but-healthy scenario (007) (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tests/synthetic/rds_postgres/007-connection-pressure-noisy-healthy/QA_VALIDATION.md`, `tests/synthetic/rds_postgres/007-connection-pressure-noisy-healthy/answer.yml`) * [#942](https://github.com/Tracer-Cloud/opensre/pull/942) synthetic: enforce single shared root cause for connection leak scenario (009) (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `tests/synthetic/rds_postgres/009-dual-fault-connection-cpu/QA_VALIDATION.md`, `tests/synthetic/rds_postgres/009-dual-fault-connection-cpu/answer.yml`) * [#843](https://github.com/Tracer-Cloud/opensre/pull/843) synthetic: tighten validation and fix RCA reasoning for replication lag red-herring scenario (006) (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tests/synthetic/rds_postgres/006-replication-lag-cpu-redherring/QA_VALIDATION.md`, `tests/synthetic/rds_postgres/006-replication-lag-cpu-redherring/answer.yml`) * [#976](https://github.com/Tracer-Cloud/opensre/pull/976) fix(eks): forward stored AWS integration credentials to k8s client builder (author: chaosreload; contributors: chaosreload and weichao; labels: *none*; files: `app/nodes/plan_actions/detect_sources.py`, `services/eks/eks_client.py`, `services/eks/eks_k8s_client.py`, `services/eks/utils.py`, `tools/EKSDeploymentStatusTool/__init__.py`, `tools/EKSDescribeAddonTool/__init__.py`, `tools/EKSDescribeClusterTool/__init__.py`, `tools/EKSEventsTool/__init__.py`, `tools/EKSListClustersTool/__init__.py`, `tools/EKSListDeploymentsTool/__init__.py`, and 7 more) * [#1055](https://github.com/Tracer-Cloud/opensre/pull/1055) docs(readme): sync integration matrix with shipped integrations (author: Ankush; contributors: Ankush; labels: *none*; files: `README.md`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-27T23:00:00+00:00` to `2026-04-28T23:00:00+00:00` # Daily Update — 2026-04-29 Source: https://opensre.com/docs/daily-updates/2026-04-29 OpenSRE engineering daily update for 2026-04-29 (Europe/London) Thanks to everyone who contributed yesterday: Anwesh, Aryan Jain, Awokoya Olawale Davidson, Ceren Camkiran, Joey Roth, John Zakkam, Kartik Sirohi, Malik Amir Hamza, Mateus Scheuer Macedo, Richard Coker, Rohit Rajan, Rohit Rajan, Ryjen1, Udit, vignesh Gopikrishnan, vincenthus, WatchTree-19, Yash Kumar Saini, and Yassir Maknaoui 🙏🚀 ## Main updates shipped (April 29, 2026) * feat: interactive REPL — the first step toward a Claude-Code-style SRE terminal (#243) — Yash Kumar Saini * feat: add Argo CD deployment investigation integration (#320) — Mateus Scheuer Macedo * feat: add Apache Airflow integration with DAG/task evidence and tests (#330) — Ceren Camkiran * Adaptive incident window: expand-on-empty-deploy-timeline — Malik Amir Hamza * refactor(tools): resolve action labels from registry metadata (#870) — John Zakkam * fix: resolve synthetic RDS scenario 008 scoring and planner loops (#604) — Yassir Maknaoui * refactor(wizard): split integration health validators into grouped modules (#873) — vignesh Gopikrishnan * fix(airflow): correct tool source, StrictConfigModel, surfaces, exception-recovery test — Yash Kumar Saini * refactor: extract shared SQL wrapper helper for repeated tool flow (#894) — Maryjane Okafor * refactor(utils): extract shared truncate() helper, replace six inline \_truncate() implementations (#1056) — Udit * auto-configure PATH in shell profile after install (#1063) — Rohit Rajan * feat(cli): harden test inventory discovery and UX (#172) — Awokoya Olawale Davidson * refactor: move Notion client to services/ (#898) — Aryan Jain * refactor: centralize credential and param extraction for eks workload list (#895) — Richard Coker * test: add direct unit tests for tempo mixin (#882) — Kartik Sirohi * test(grafana): add direct unit tests for LokiMixin.query\_loki (#880) — WatchTree-19 * test(tools): add unit tests for Bitbucket tools (#992) — vignesh Gopikrishnan * Create airflow\_integration.mdx (#1020) — Ceren Camkiran * Fix Makefile Ruff targets to use configured Python (#176) — Joey Roth * chore(github): trim celebration GIFs and add three new merge GIFs — Anwesh ## Source pull requests * [#1062](https://github.com/Tracer-Cloud/opensre/pull/1062) Fix Makefile Ruff targets to use configured Python (author: Joey Roth; contributors: Joey Roth; labels: *none*; files: `Makefile`) * [#844](https://github.com/Tracer-Cloud/opensre/pull/844) fix: resolve synthetic RDS scenario 008 scoring and planner loops (author: Yassir Maknaoui; contributors: Yassir Maknaoui; labels: *none*; files: `app/nodes/investigate/processing/post_process.py`, `app/nodes/investigate/types.py`, `app/nodes/plan_actions/build_prompt.py`, `app/nodes/plan_actions/plan_actions.py`, `tests/nodes/plan_actions/test_reroute_and_budget.py`, `tests/synthetic/rds_postgres/run_suite.py`, `tests/synthetic/rds_postgres/test_suite.py`) * [#945](https://github.com/Tracer-Cloud/opensre/pull/945) refactor: move Notion client to services/ (author: Aryan Jain; contributors: Aryan Jain; labels: *none*; files: `.cursor/rules/integrations.mdc`, `app/integrations/clients/notion/__init__.py`, `services/notion/__init__.py`, `services/notion/client.py`, `tests/integrations/test_notion_client.py`) * [#915](https://github.com/Tracer-Cloud/opensre/pull/915) refactor(tools): resolve action labels from registry metadata (author: John Zakkam; contributors: John Zakkam; labels: *none*; files: `app/output.py`, `deployment/remote/reasoning.py`, `tools/BetterStackLogsTool/__init__.py`, `tools/CloudWatchLogsTool/__init__.py`, `tools/DataDogContextTool/__init__.py`, `tools/DataDogEventsTool/__init__.py`, `tools/DataDogLogsTool/__init__.py`, `tools/DataDogMonitorsTool/__init__.py`, `tools/GrafanaAlertRulesTool/__init__.py`, `tools/GrafanaLogsTool/__init__.py`, and 18 more) * [#1074](https://github.com/Tracer-Cloud/opensre/pull/1074) Adaptive incident window: expand-on-empty-deploy-timeline (author: Malik Amir Hamza; contributors: Malik Amir Hamza; labels: *none*; files: `app/incident_window.py`, `app/investigation_constants.py`, `app/nodes/__init__.py`, `app/nodes/adapt_window/__init__.py`, `app/nodes/adapt_window/node.py`, `app/nodes/adapt_window/rules.py`, `app/pipeline/graph.py`, `app/state/agent_state.py`, `tests/app/test_incident_window.py`, `tests/nodes/adapt_window/__init__.py`, and 3 more) * [#958](https://github.com/Tracer-Cloud/opensre/pull/958) refactor(wizard): split integration health validators into grouped modules (author: vignesh Gopikrishnan; contributors: vignesh Gopikrishnan; labels: *none*; files: `app/cli/wizard/integration_health.py`, `app/cli/wizard/integration_validators/__init__.py`, `app/cli/wizard/integration_validators/client_validators.py`, `app/cli/wizard/integration_validators/http_probe_validators.py`, `app/cli/wizard/integration_validators/mcp_validators.py`, `app/cli/wizard/integration_validators/shared.py`, `tests/cli/wizard/test_integration_health.py`) * [#1059](https://github.com/Tracer-Cloud/opensre/pull/1059) test: add direct unit tests for tempo mixin (#882) (author: Kartik Sirohi; contributors: Kartik Sirohi; labels: *none*; files: `services/grafana/tempo.py`, `tests/services/test_grafana_tempo.py`) * [#570](https://github.com/Tracer-Cloud/opensre/pull/570) feat: add Apache Airflow integration with DAG/task evidence and tests (author: Ceren Camkiran; contributors: Ceren Camkiran and vincenthus; labels: *none*; files: `app/integrations/airflow.py`, `app/integrations/catalog.py`, `app/integrations/models.py`, `app/nodes/plan_actions/build_prompt.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/plan_actions/plan_actions.py`, `tools/TracerAirflowDAGTool/__init__.py`, `packaging/sync_release_version.py`, `tests/e2e/airflow/__init__.py`, `tests/e2e/airflow/fixtures/airflow_task_failure_alert.json`, and 4 more) * [#1075](https://github.com/Tracer-Cloud/opensre/pull/1075) chore(github): trim celebration GIFs and add three new merge GIFs (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/assets/celebrations/merge-celebrate-1.gif`, `.github/assets/celebrations/merge-celebrate-2.gif`, `.github/assets/celebrations/merge-celebrate-3.gif`, `.github/assets/celebrations/office-dance.gif`, `.github/assets/celebrations/office-win.gif`, `.github/assets/celebrations/office-yes.gif`, `.github/assets/celebrations/winner.gif`, `.github/scripts/merged_pr_celebration_message.py`) * [#1066](https://github.com/Tracer-Cloud/opensre/pull/1066) feat(cli): harden test inventory discovery and UX (#172) (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `app/cli/commands/tests.py`, `app/cli/tests/discover.py`, `app/cli/tests/runner.py`, `tests/cli/test_cli_inventory.py`, `tests/cli/test_discover.py`) * [#1077](https://github.com/Tracer-Cloud/opensre/pull/1077) fix(airflow): correct tool source, StrictConfigModel, surfaces, exception-recovery test (author: Yash Kumar Saini; contributors: Yash Kumar Saini; labels: *none*; files: `app/integrations/airflow.py`, `tools/TracerAirflowDAGTool/__init__.py`, `app/types/evidence.py`, `tests/integrations/test_airflow.py`) * [#591](https://github.com/Tracer-Cloud/opensre/pull/591) feat: interactive REPL — the first step toward a Claude-Code-style SRE terminal (author: Yash Kumar Saini; contributors: Rohit Rajan, vincenthus, and Yash Kumar Saini; labels: `product direction`; files: `README.md`, `app/cli/__main__.py`, `app/cli/commands/__init__.py`, `app/cli/commands/agent.py`, `app/cli/commands/general.py`, `app/cli/investigate.py`, `app/cli/layout.py`, `app/cli/interactive_shell/__init__.py`, `app/cli/interactive_shell/banner.py`, `app/cli/interactive_shell/commands.py`, and 15 more) * [#948](https://github.com/Tracer-Cloud/opensre/pull/948) feat: add Argo CD deployment investigation integration (author: Mateus Scheuer Macedo; contributors: Mateus Scheuer Macedo; labels: *none*; files: `.env.example`, `app/cli/constants.py`, `app/integrations/catalog.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/nodes/investigate/processing/post_process.py`, `app/nodes/plan_actions/detect_sources.py`, `app/nodes/root_cause_diagnosis/evidence_checker.py`, `services/argocd/__init__.py`, `services/argocd/client.py`, and 13 more) * [#1058](https://github.com/Tracer-Cloud/opensre/pull/1058) Create airflow\_integration.mdx (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `docs/airflow.mdx`, `docs/docs.json`) * [#1080](https://github.com/Tracer-Cloud/opensre/pull/1080) refactor(utils): extract shared truncate() helper, replace six inline \_truncate() implementations (author: Udit; contributors: Udit; labels: *none*; files: `app/integrations/azure_sql.py`, `app/integrations/mariadb.py`, `app/integrations/mysql.py`, `app/nodes/publish_findings/gitlab_writeback.py`, `utils/discord_delivery.py`, `utils/telegram_delivery.py`, `utils/truncation.py`, `tests/nodes/publish_findings/test_gitlab_writeback.py`, `tests/test_mariadb_integration.py`, `tests/utils/test_truncation.py`) * [#1084](https://github.com/Tracer-Cloud/opensre/pull/1084) refactor: extract shared SQL wrapper helper for repeated tool flow (author: Maryjane Okafor; contributors: Maryjane Okafor and Ryjen1; labels: *none*; files: `tools/AzureSQLCurrentQueriesTool/__init__.py`, `tools/AzureSQLSlowQueriesTool/__init__.py`, `tools/MariaDBProcessListTool/__init__.py`, `tools/MySQLCurrentProcessesTool/__init__.py`, `tools/PostgreSQLCurrentQueriesTool/__init__.py`, `tools/PostgreSQLSlowQueriesTool/__init__.py`, `tools/utils/sql_wrapper.py`, `tests/tools/test_sql_wrapper.py`) * [#1088](https://github.com/Tracer-Cloud/opensre/pull/1088) feat(github): add 16 PR celebration message templates (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/scripts/merged_pr_celebration_message.py`) * [#1017](https://github.com/Tracer-Cloud/opensre/pull/1017) test(tools): add unit tests for Bitbucket tools (author: vignesh Gopikrishnan; contributors: vignesh Gopikrishnan; labels: *none*; files: `tools/BitbucketSearchCodeTool/__init__.py`, `tests/tools/test_bitbucket_commits_tool.py`, `tests/tools/test_bitbucket_file_contents_tool.py`, `tests/tools/test_bitbucket_search_code_tool.py`) * [#1082](https://github.com/Tracer-Cloud/opensre/pull/1082) test(grafana): add direct unit tests for LokiMixin.query\_loki (author: WatchTree-19; contributors: WatchTree-19; labels: *none*; files: `tests/services/test_grafana_loki.py`) * [#1064](https://github.com/Tracer-Cloud/opensre/pull/1064) auto-configure PATH in shell profile after install (author: Rohit Rajan; contributors: Rohit Rajan and Rohit Rajan; labels: *none*; files: `install.sh`, `tests/cli/test_install_sh_path.py`) * [#1089](https://github.com/Tracer-Cloud/opensre/pull/1089) refactor: centralize credential and param extraction for eks workload list (author: Richard Coker; contributors: Richard Coker; labels: *none*; files: `tools/EKSListClustersTool/__init__.py`, `tools/EKSListDeploymentsTool/__init__.py`, `tools/EKSListPodsTool/__init__.py`, `tools/utils/eks_workload_helper.py`, `tests/tools/utils/test_eks_workload_helper.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-28T23:00:00+00:00` to `2026-04-29T23:00:00+00:00` # Daily Update — 2026-04-30 Source: https://opensre.com/docs/daily-updates/2026-04-30 OpenSRE engineering daily update for 2026-04-30 (Europe/London) Thanks to everyone who contributed yesterday: Aarush Sharma, Aarush Sharma, Aaryan, Abhishek Marathe, Anwesh, Bhavarth Bhangdia, Hariswar, Kevin Espiñeira, Mayank Bharati, Michael Emefienem, Prakhar Jain, Vaibhav Upreti, wen, Yajush Srivastava, and Zeeshan 🙏🚀 ## Main updates shipped (April 30, 2026) * Issue/319 splunk integration — Abhishek Marathe * fix(discord,slack): harden delivery against non-JSON failures and token leakage (#865) — Kevin Espiñeira * fix(cli): `opensre tests` crashes with FileNotFoundError in bundled binary (#1078) — Mayank Bharati * fix(llm-cli): address post-review issues in CLI integration — Anwesh * fix: Ctrl+C double-exit in interactive shell — wen * Fix: attach data\_quality\_issues for list payloads in wrapper validation — Michael Emefienem * Issue 1127 harden remote tool end payloads — Yajush Srivastava * fix(extract): prevent health checks with state=normal from being marked as noise — Aarush Sharma * ci(github): split default CI and label-triggered optional checks — Anwesh * Add registry regression test for duplicate tool names across modules — Aaryan * Feat/add registry import failure test — Aaryan * Fix/check disk health tests — Bhavarth Bhangdia * test: add unit tests for parse\_vercel\_url to extract log and deployme… — Zeeshan * test(cli): skip install.sh tests on Windows runner (#1099) — Mayank Bharati * ci(github): split Windows and k8s label workflows — Anwesh * fix(github): disambiguate contributors workflow push refspec — Anwesh * Tighten OpenClaw docs with verify/fail-fast section (#1132) — Hariswar * docs: add troubleshooting block to quickstart — Prakhar Jain * docs(agents): add LLM API provider guide in services — Anwesh ## Source pull requests * [#1096](https://github.com/Tracer-Cloud/opensre/pull/1096) fix(extract): prevent health checks with state=normal from being marked as noise (author: Aarush Sharma; contributors: Aarush Sharma and Aarush Sharma; labels: *none*; files: `app/nodes/extract_alert/extract.py`) * [#1090](https://github.com/Tracer-Cloud/opensre/pull/1090) fix(cli): `opensre tests` crashes with FileNotFoundError in bundled binary (#1078) (author: Mayank Bharati; contributors: Mayank Bharati; labels: *none*; files: `app/cli/commands/deploy.py`, `app/cli/commands/tests.py`, `app/cli/tests/discover.py`, `tests/cli/test_cli_inventory.py`, `tests/cli/test_discover.py`) * [#1100](https://github.com/Tracer-Cloud/opensre/pull/1100) ci(github): split default CI and label-triggered optional checks (author: Anwesh; contributors: Anwesh; labels: `ci:windows`; files: `.github/workflows/ci-labels.yml`, `.github/workflows/ci.yml`) * [#1101](https://github.com/Tracer-Cloud/opensre/pull/1101) test(cli): skip install.sh tests on Windows runner (#1099) (author: Mayank Bharati; contributors: Mayank Bharati; labels: `ci:windows`; files: `tests/cli/test_install_sh_path.py`, `tests/cli/test_install_sh_resolution.py`) * [#1103](https://github.com/Tracer-Cloud/opensre/pull/1103) ci(github): split Windows and k8s label workflows (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/workflows/ci-labels-windows.yml`, `.github/workflows/ci.yml`) * [#1097](https://github.com/Tracer-Cloud/opensre/pull/1097) fix(llm-cli): address post-review issues in CLI integration (author: Anwesh; contributors: Anwesh; labels: *none*; files: `app/cli/wizard/config.py`, `app/cli/wizard/flow.py`, `app/integrations/llm_cli/AGENTS.md`, `app/integrations/llm_cli/base.py`, `app/integrations/llm_cli/binary_resolver.py`, `app/integrations/llm_cli/codex.py`, `app/integrations/llm_cli/registry.py`, `app/integrations/llm_cli/runner.py`, `services/llm_client.py`, `tests/cli/wizard/test_flow.py`, and 1 more) * [#1105](https://github.com/Tracer-Cloud/opensre/pull/1105) fix: Ctrl+C double-exit in interactive shell (author: wen; contributors: wen; labels: *none*; files: `app/cli/__main__.py`, `app/cli/prompt_support.py`, `app/cli/wizard/prompts.py`, `tests/cli/test_prompt_support.py`, `tests/cli_smoke_test.py`) * [#1137](https://github.com/Tracer-Cloud/opensre/pull/1137) fix(github): disambiguate contributors workflow push refspec (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/workflows/contributors.yml`) * [#1135](https://github.com/Tracer-Cloud/opensre/pull/1135) docs(agents): add LLM API provider guide in services (author: Anwesh; contributors: Anwesh; labels: *none*; files: `AGENTS.md`, `services/AGENTS.md`) * [#1140](https://github.com/Tracer-Cloud/opensre/pull/1140) Issue 1127 harden remote tool end payloads (author: Yajush Srivastava; contributors: Yajush Srivastava; labels: *none*; files: `deployment/remote/reasoning.py`, `tests/deployment/remote/test_reasoning.py`) * [#1141](https://github.com/Tracer-Cloud/opensre/pull/1141) Add registry regression test for duplicate tool names across modules (author: Aaryan; contributors: Aaryan and Vaibhav Upreti; labels: *none*; files: `tests/tools/test_registry.py`) * [#1147](https://github.com/Tracer-Cloud/opensre/pull/1147) Tighten OpenClaw docs with verify/fail-fast section (#1132) (author: Hariswar; contributors: Hariswar; labels: *none*; files: `docs/openclaw.mdx`) * [#791](https://github.com/Tracer-Cloud/opensre/pull/791) Issue/319 splunk integration (author: Abhishek Marathe; contributors: Abhishek Marathe; labels: *none*; files: `app/cli/alert_templates.py`, `app/cli/wizard/flow.py`, `app/cli/wizard/integration_health.py`, `app/cli/wizard/integration_validators/client_validators.py`, `app/integrations/catalog.py`, `app/integrations/models.py`, `app/integrations/verify.py`, `app/nodes/plan_actions/build_prompt.py`, `app/nodes/plan_actions/detect_sources.py`, `services/splunk/__init__.py`, and 13 more) * [#1148](https://github.com/Tracer-Cloud/opensre/pull/1148) test: add unit tests for parse\_vercel\_url to extract log and deployme… (author: Zeeshan; contributors: Zeeshan; labels: *none*; files: `tests/deployment/remote/test_vercel_poller.py`) * [#1138](https://github.com/Tracer-Cloud/opensre/pull/1138) fix(discord,slack): harden delivery against non-JSON failures and token leakage (#865) (author: Kevin Espiñeira; contributors: Kevin Espiñeira; labels: *none*; files: `utils/discord_delivery.py`, `utils/slack_delivery.py`, `tests/utils/test_discord_delivery.py`, `tests/utils/test_slack_delivery.py`) * [#1142](https://github.com/Tracer-Cloud/opensre/pull/1142) Feat/add registry import failure test (author: Aaryan; contributors: Aaryan; labels: *none*; files: `tools/registry.py`, `tests/tools/test_registry.py`) * [#1152](https://github.com/Tracer-Cloud/opensre/pull/1152) docs: add troubleshooting block to quickstart (author: Prakhar Jain; contributors: Prakhar Jain; labels: *none*; files: `docs/quickstart.mdx`) * [#1155](https://github.com/Tracer-Cloud/opensre/pull/1155) Fix/check disk health tests (author: Bhavarth Bhangdia; contributors: Bhavarth Bhangdia; labels: *none*; files: `tests/deployment/remote/test_server.py`) * [#1151](https://github.com/Tracer-Cloud/opensre/pull/1151) Fix: attach data\_quality\_issues for list payloads in wrapper validation (author: Michael Emefienem; contributors: Michael Emefienem; labels: *none*; files: `tools/utils/data_validation.py`, `tests/tools/utils/test_data_validation.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-29T23:00:00+00:00` to `2026-04-30T23:00:00+00:00` # Daily Update — 2026-05-01 Source: https://opensre.com/docs/daily-updates/2026-05-01 OpenSRE engineering daily update for 2026-05-01 (Europe/London) Thanks to everyone who contributed yesterday: akshat1074, Aniket Rawat, Anwesh, Awokoya Olawale Davidson, Divyansh Rawat, Diwansu-Pilania, Mayank Bharati, Prakhar Jain, Ryjen1, S. B. | Software Developer, Tejas, Vaibhav Upreti, vignesh Gopikrishnan, and vincenthus 🙏🚀 ## Main updates shipped (May 1, 2026) * chore: refactor integrations — Vaibhav Upreti * feat(cli): interactive shell UX improvements — vincenthus * feat(cli): improve interactive shell automation — vincenthus * feat(llm\_cli): integrate Claude Code CLI as LLM\_PROVIDER — Awokoya Olawale Davidson * fix(guardrails): redact union of overlapping spans, never leak sensitive bookends — Mayank Bharati * feat(alerts): normalize incoming payloads to OpenSRE canonical format — vignesh Gopikrishnan * feat(ci): auto-assign good first issues for new contributors — Anwesh * fix(ci): single assignee per good first issue — Anwesh * test(output): add direct unit tests for app/output.py (#871) — Awokoya Olawale Davidson * feat: print onboarding hint after successful install (#1153) — Maryjane Okafor * API Response Test: direct unit tests for services/datadog/client.py — S. B. | Software Developer * Fix/1114 health summary failure aliases — Aniket Rawat * test: add failure path coverage for IMDS helpers — Tejas * tests: add Escape-cancel coverage for confirm, text, and path prompts — Divyansh Rawat * test: add \_id\_to\_iso helper coverage in remote server tests — Prakhar Jain * test: add Vercel deployment sort regression tests for bad timestamps — Dark\_Pheonix * docs: add 'Run one focused test' section to CONTRIBUTING.md — akshat1074 ## Source pull requests * [#1159](https://github.com/Tracer-Cloud/opensre/pull/1159) feat(cli): interactive shell UX improvements (author: vincenthus; contributors: vincenthus; labels: *none*; files: `AGENTS.md`, `app/cli/__init__.py`, `app/cli/__main__.py`, `app/cli/commands/agent.py`, `app/cli/commands/deploy.py`, `app/cli/commands/doctor.py`, `app/cli/commands/general.py`, `app/cli/commands/integrations.py`, `app/cli/commands/remote.py`, `app/cli/commands/remote_health.py`, and 93 more) * [#1165](https://github.com/Tracer-Cloud/opensre/pull/1165) chore: refactor integrations (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `app/integrations/__main__.py`, `app/integrations/_catalog_impl.py`, `app/integrations/_relational.py`, `app/integrations/_validators.py`, `app/integrations/_verification_adapters.py`, `app/integrations/catalog.py`, `app/integrations/cli.py`, `app/integrations/config_models.py`, `app/integrations/effective_models.py`, `app/integrations/mariadb.py`, and 26 more) * [#1167](https://github.com/Tracer-Cloud/opensre/pull/1167) feat(cli): improve interactive shell automation (author: vincenthus; contributors: vincenthus; labels: *none*; files: `app/cli/interactive_shell/agent_actions.py`, `app/cli/interactive_shell/banner.py`, `app/cli/interactive_shell/cli_agent.py`, `app/cli/interactive_shell/commands.py`, `app/cli/interactive_shell/follow_up.py`, `app/cli/interactive_shell/history.py`, `app/cli/interactive_shell/loop.py`, `app/cli/interactive_shell/router.py`, `app/cli/interactive_shell/session.py`, `app/cli/interactive_shell/terminal_intent.py`, and 11 more) * [#1168](https://github.com/Tracer-Cloud/opensre/pull/1168) feat(llm\_cli): integrate Claude Code CLI as LLM\_PROVIDER (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `.env.example`, `app/cli/wizard/config.py`, `config/config.py`, `app/integrations/llm_cli/claude_code.py`, `app/integrations/llm_cli/registry.py`, `app/integrations/llm_cli/runner.py`, `tests/integrations/llm_cli/test_claude_code_adapter.py`) * [#1169](https://github.com/Tracer-Cloud/opensre/pull/1169) test(output): add direct unit tests for app/output.py (#871) (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `tests/test_output.py`) * [#1158](https://github.com/Tracer-Cloud/opensre/pull/1158) test: add \_id\_to\_iso helper coverage in remote server tests (author: Prakhar Jain; contributors: Prakhar Jain; labels: *none*; files: `tests/deployment/remote/test_server.py`) * [#1157](https://github.com/Tracer-Cloud/opensre/pull/1157) feat: print onboarding hint after successful install (#1153) (author: Maryjane Okafor; contributors: Maryjane Okafor and Ryjen1; labels: *none*; files: `install.ps1`, `install.sh`, `tests/cli/test_install_sh_path.py`) * [#1154](https://github.com/Tracer-Cloud/opensre/pull/1154) test: add failure path coverage for IMDS helpers (author: Tejas; contributors: Tejas; labels: *none*; files: `tests/deployment/remote/test_server.py`) * [#1170](https://github.com/Tracer-Cloud/opensre/pull/1170) tests: add Escape-cancel coverage for confirm, text, and path prompts (author: Divyansh Rawat; contributors: Divyansh Rawat; labels: *none*; files: `tests/cli/test_prompt_support.py`) * [#977](https://github.com/Tracer-Cloud/opensre/pull/977) feat(alerts): normalize incoming payloads to OpenSRE canonical format (author: vignesh Gopikrishnan; contributors: vignesh Gopikrishnan; labels: *none*; files: `app/alerts/__init__.py`, `app/alerts/normalize.py`, `app/state/factory.py`, `tests/test_state.py`) * [#1177](https://github.com/Tracer-Cloud/opensre/pull/1177) docs: add 'Run one focused test' section to CONTRIBUTING.md (author: akshat1074; contributors: akshat1074; labels: *none*; files: `CONTRIBUTING.md`) * [#780](https://github.com/Tracer-Cloud/opensre/pull/780) fix(guardrails): redact union of overlapping spans, never leak sensitive bookends (author: Mayank Bharati; contributors: Mayank Bharati; labels: *none*; files: `app/guardrails/engine.py`, `tests/test_guardrails/test_engine.py`, `tests/test_guardrails/test_llm_integration.py`) * [#1179](https://github.com/Tracer-Cloud/opensre/pull/1179) feat(ci): auto-assign good first issues for new contributors (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/scripts/good_first_issue_assign.py`, `.github/workflows/good-first-issue-assign.yml`, `tests/github/test_good_first_issue_assign.py`) * [#1178](https://github.com/Tracer-Cloud/opensre/pull/1178) test: add Vercel deployment sort regression tests for bad timestamps (author: Dark\_Pheonix; contributors: Dark\_Pheonix and Diwansu-Pilania; labels: *none*; files: `tests/deployment/remote/test_vercel_poller.py`) * [#1104](https://github.com/Tracer-Cloud/opensre/pull/1104) API Response Test: direct unit tests for services/datadog/client.py (author: S. B. | Software Developer; contributors: S. B. | Software Developer; labels: *none*; files: `tests/services/test_datadog_async_client.py`, `tests/services/test_datadog_client.py`) * [#1183](https://github.com/Tracer-Cloud/opensre/pull/1183) fix(ci): single assignee per good first issue (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/scripts/good_first_issue_assign.py`, `.github/workflows/good-first-issue-assign.yml`, `tests/github/test_good_first_issue_assign.py`) * [#1184](https://github.com/Tracer-Cloud/opensre/pull/1184) Fix/1114 health summary failure aliases (author: Aniket Rawat; contributors: Aniket Rawat; labels: *none*; files: `app/cli/support/health_view.py`, `tests/cli/test_health_view.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-04-30T23:00:00+00:00` to `2026-05-01T23:00:00+00:00` # Daily Update — 2026-05-02 Source: https://opensre.com/docs/daily-updates/2026-05-02 OpenSRE engineering daily update for 2026-05-02 (Europe/London) Thanks to everyone who contributed yesterday: Anwesh, Awokoya Olawale Davidson, Diwansu-Pilania, Jaimin Godhani, Just Harold, Krish Srivastava, Prakhar Jain, S. B. | Software Developer, Shivam Behl, and Turan Buyukkamaci 🙏🚀 ## Main updates shipped (May 2, 2026) * feat(cli): make interactive shell documentation-aware (#1166) — Awokoya Olawale Davidson * refactor: Ingest flow cleanup & Fix ingest duplication streamline investigation URL handling — S. B. | Software Developer * test: add unit tests for AlertmanagerClient (#876) — Turan Buyukkamaci * Fix: keep CLI analytics disabled when users opt out (#1118) — Shivam Behl * Fix: Integrations overview pages — Just Harold * test: add direct unit tests for grafana config normalization and auth… — Prakhar Jain * fix: move stdlib imports to module level in hot-path functions — Jaimin Godhani * test: consolidate remote-runtime tests under tests/remote/ (#872) — Awokoya Olawale Davidson * Revert "feat: Add analytics opt-out environment variable tests (#1118)" — Anwesh * test(cli): use dummy app session in prompt-session tests so they pass… — Awokoya Olawale Davidson * docs: add quick-start commands for testing in tests/README.md — Krish Srivastava * feat: Add analytics opt-out environment variable tests (#1118) — Dark\_Pheonix ## Source pull requests * [#1172](https://github.com/Tracer-Cloud/opensre/pull/1172) feat(cli): make interactive shell documentation-aware (#1166) (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `app/cli/interactive_shell/cli_help.py`, `app/cli/interactive_shell/docs_reference.py`, `app/cli/interactive_shell/router.py`, `tests/cli/interactive_shell/test_cli_help.py`, `tests/cli/interactive_shell/test_docs_reference.py`, `tests/cli/interactive_shell/test_router.py`) * [#1186](https://github.com/Tracer-Cloud/opensre/pull/1186) test: add direct unit tests for grafana config normalization and auth… (author: Prakhar Jain; contributors: Prakhar Jain; labels: *none*; files: `tests/services/test_grafana_config.py`) * [#1191](https://github.com/Tracer-Cloud/opensre/pull/1191) refactor: Ingest flow cleanup & Fix ingest duplication streamline investigation URL handling (author: S. B. | Software Developer; contributors: S. B. | Software Developer; labels: *none*; files: `app/nodes/publish_findings/formatters/report.py`, `app/nodes/publish_findings/node.py`, `utils/ingest_delivery.py`, `tests/nodes/publish_findings/test_masking_unmask.py`, `tests/nodes/publish_findings/test_node.py`, `tests/utils/test_ingest_delivery.py`) * [#1193](https://github.com/Tracer-Cloud/opensre/pull/1193) Fix: Integrations overview pages (author: Just Harold; contributors: Just Harold; labels: *none*; files: `docs/docs.json`, `docs/integrations/integration-workflow.mdx`) * [#1194](https://github.com/Tracer-Cloud/opensre/pull/1194) test: add unit tests for AlertmanagerClient (#876) (author: Turan Buyukkamaci; contributors: Turan Buyukkamaci; labels: *none*; files: `tests/services/test_alertmanager_client.py`) * [#1189](https://github.com/Tracer-Cloud/opensre/pull/1189) feat: Add analytics opt-out environment variable tests (#1118) (author: Dark\_Pheonix; contributors: Dark\_Pheonix and Diwansu-Pilania; labels: *none*; files: `tests/analytics/test_provider.py`) * [#1195](https://github.com/Tracer-Cloud/opensre/pull/1195) Revert "feat: Add analytics opt-out environment variable tests (#1118)" (author: Anwesh; contributors: Anwesh; labels: *none*; files: `tests/analytics/test_provider.py`) * [#1216](https://github.com/Tracer-Cloud/opensre/pull/1216) fix: move stdlib imports to module level in hot-path functions (author: Jaimin Godhani; contributors: Jaimin Godhani; labels: *none*; files: `app/nodes/chat.py`, `app/nodes/root_cause_diagnosis/prompt_builder.py`) * [#1215](https://github.com/Tracer-Cloud/opensre/pull/1215) test: consolidate remote-runtime tests under tests/remote/ (#872) (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: `ci:windows`; files: `tests/deployment/remote/__init__.py`, `tests/remote/test_client.py`, `tests/remote/test_reasoning.py`, `tests/remote/test_renderer.py`, `tests/remote/test_server.py`, `tests/remote/test_stream.py`, `tests/remote/test_system_metrics.py`, `tests/remote/test_vercel_poller.py`) * [#1224](https://github.com/Tracer-Cloud/opensre/pull/1224) test(cli): use dummy app session in prompt-session tests so they pass… (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: `ci:windows`; files: `tests/cli/interactive_shell/test_loop.py`) * [#1225](https://github.com/Tracer-Cloud/opensre/pull/1225) Fix: keep CLI analytics disabled when users opt out (#1118) (author: Shivam Behl; contributors: Shivam Behl; labels: *none*; files: `tests/analytics/test_provider.py`) * [#1171](https://github.com/Tracer-Cloud/opensre/pull/1171) docs: add quick-start commands for testing in tests/README.md (author: Krish Srivastava; contributors: Krish Srivastava; labels: *none*; files: `tests/README.md`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-05-01T23:00:00+00:00` to `2026-05-02T23:00:00+00:00` # Daily Update — 2026-05-03 Source: https://opensre.com/docs/daily-updates/2026-05-03 OpenSRE engineering daily update for 2026-05-03 (Europe/London) Thanks to everyone who contributed yesterday: Aakash Kumar, aksKrWRef, Ankit Juneja, Ankit Juneja, Anwesh, Arjun, Ceren Camkiran, Devesh, Enoch, Isaac, Raj Gajjar, Thibault Jaigu, Umer Aamir, Vaibhav Upreti, and Yash Kumar Saini 🙏🚀 ## Main updates shipped (May 3, 2026) * feat(integrations): add VictoriaLogs integration (#126) — Yash Kumar Saini * feat: deprecate tracer path and migrate to opensre — Vaibhav Upreti * feat: add Requesty as LLM provider — Thibault Jaigu * eval: harden OpenSRE LLM judge JSON extraction — Ceren Camkiran * fix: live-probe macOS Keychain auth for Claude Code CLI onboarding (#1199) — Ankit Juneja * Fix detection of shell built-in commands (e.g. pwd) in interactive CLI — Aakash Kumar * Revert "Fix detection of shell built-in commands (e.g. pwd) in interactive CLI" — Anwesh * fix(tests): add default\_headers param to \_FakeOpenAI stub — Anwesh * fix(ci): restore CLI help sync + uninstall help text — Isaac * fix: move stdlib json import to module level in hot-path tracer + lambda functions — Enoch * feat: add release announcements workflow for Discord and X (Twitter) — Devesh * test: added direct unit tests for app/cli/local\_llm/hardware.py — Umer Aamir * test: add unit tests for aws sdk service client — Arjun * test: add \_check\_memory\_health missing and incomplete data tests — Raj Gajjar ## Source pull requests * [#1144](https://github.com/Tracer-Cloud/opensre/pull/1144) feat(integrations): add VictoriaLogs integration (author: Yash Kumar Saini; contributors: Yash Kumar Saini; labels: *none*; files: `app/cli/support/constants.py`, `app/integrations/_catalog_impl.py`, `app/integrations/_verification_adapters.py`, `app/integrations/config_models.py`, `app/integrations/effective_models.py`, `app/integrations/models.py`, `app/integrations/registry.py`, `app/nodes/plan_actions/detect_sources.py`, `services/victoria_logs/__init__.py`, `services/victoria_logs/client.py`, and 9 more) * [#1231](https://github.com/Tracer-Cloud/opensre/pull/1231) fix(ci): restore CLI help sync + uninstall help text (author: Isaac; contributors: Isaac; labels: *none*; files: `app/cli/commands/general.py`, `tests/cli/test_command_help_sync.py`) * [#1212](https://github.com/Tracer-Cloud/opensre/pull/1212) test: add \_check\_memory\_health missing and incomplete data tests (author: Raj Gajjar; contributors: Raj Gajjar; labels: *none*; files: `tests/remote/test_server.py`) * [#1240](https://github.com/Tracer-Cloud/opensre/pull/1240) test: add unit tests for aws sdk service client (author: Arjun; contributors: Arjun; labels: *none*; files: `tests/services/test_aws_sdk_client.py`) * [#1243](https://github.com/Tracer-Cloud/opensre/pull/1243) fix: move stdlib json import to module level in hot-path tracer + lambda functions (author: Enoch; contributors: Enoch; labels: *none*; files: `services/lambda_client.py`, `services/tracer_client/tracer_integrations.py`) * [#1245](https://github.com/Tracer-Cloud/opensre/pull/1245) feat: add Requesty as LLM provider (author: Thibault Jaigu; contributors: Thibault Jaigu; labels: *none*; files: `.env.example`, `app/cli/commands/doctor.py`, `app/cli/interactive_shell/agent_actions.py`, `app/cli/wizard/config.py`, `app/cli/wizard/validation.py`, `config/config.py`, `services/llm_client.py`) * [#1187](https://github.com/Tracer-Cloud/opensre/pull/1187) test: added direct unit tests for app/cli/local\_llm/hardware.py (author: Umer Aamir; contributors: Umer Aamir; labels: *none*; files: `tests/cli/test_local_llm_hardware.py`, `tests/services/test_honeycomb_client.py`) * [#989](https://github.com/Tracer-Cloud/opensre/pull/989) eval: harden OpenSRE LLM judge JSON extraction (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: *none*; files: `app/integrations/opensre/llm_eval_judge.py`, `tests/integrations/opensre/test_llm_eval_judge.py`) * [#1235](https://github.com/Tracer-Cloud/opensre/pull/1235) feat: add release announcements workflow for Discord and X (Twitter) (author: Devesh; contributors: Devesh; labels: *none*; files: `.github/workflows/release-announcements.yml`, `.gitignore`) * [#1217](https://github.com/Tracer-Cloud/opensre/pull/1217) fix: live-probe macOS Keychain auth for Claude Code CLI onboarding (#1199) (author: Ankit Juneja; contributors: Ankit Juneja and Ankit Juneja; labels: *none*; files: `app/integrations/llm_cli/claude_code.py`, `tests/integrations/llm_cli/test_claude_code_adapter.py`) * [#1251](https://github.com/Tracer-Cloud/opensre/pull/1251) fix(tests): add default\_headers param to \_FakeOpenAI stub (author: Anwesh; contributors: Anwesh; labels: *none*; files: `tests/services/test_llm_client.py`) * [#1252](https://github.com/Tracer-Cloud/opensre/pull/1252) Fix detection of shell built-in commands (e.g. pwd) in interactive CLI (author: Aakash Kumar; contributors: Aakash Kumar and aksKrWRef; labels: *none*; files: `app/cli/interactive_shell/agent_actions.py`) * [#1254](https://github.com/Tracer-Cloud/opensre/pull/1254) Revert "Fix detection of shell built-in commands (e.g. pwd) in interactive CLI" (author: Anwesh; contributors: Anwesh; labels: *none*; files: `app/cli/interactive_shell/agent_actions.py`) * [#1232](https://github.com/Tracer-Cloud/opensre/pull/1232) feat: deprecate tracer path and migrate to opensre (author: Vaibhav Upreti; contributors: Vaibhav Upreti; labels: *none*; files: `.env.example`, `app/cli/commands/general.py`, `app/cli/support/uninstall.py`, `app/cli/wizard/flow.py`, `app/constants/__init__.py`, `app/integrations/_catalog_impl.py`, `app/integrations/cli.py`, `app/integrations/store.py`, `app/integrations/verify.py`, `app/llm_credentials.py`, and 27 more) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-05-02T23:00:00+00:00` to `2026-05-03T23:00:00+00:00` # Daily Update — 2026-05-04 Source: https://opensre.com/docs/daily-updates/2026-05-04 OpenSRE engineering daily update for 2026-05-04 (Europe/London) Thanks to everyone who contributed yesterday: Aniket Rawat, Anwesh, Awokoya Olawale Davidson, Devesh, DevNinja, Enoch, Kevin Espiñeira, Paul Demarecaux, Powlisher, Sundaram Kumar Jha, Yash Kumar Saini, Yash Kumar Saini, and zerone0x 🙏🚀 ## Main updates shipped (May 4, 2026) * feat(cli): allow switching toolcall model at runtime (#1182) — Awokoya Olawale Davidson * refactor: improve RDS storage-full RCA evidence handling — Sundaram Kumar Jha * test(tools): add dedicated unit tests for Snowflake, OpenObserve, OpenSearch tools (#790) — Paul Demarecaux * tests(kafka): add unit tests for KafkaTopicHealthTool and KafkaConsumerGroupTool — Yash Kumar Saini * docs: add LLM providers reference page — DevNinja * test(coralogix): add direct unit tests for CoralogixClient — Aniket Rawat * test: cover \_check\_memory\_health threshold branches — Kevin Espiñeira * clearing backlogs -unit tests for app/cli/tests/discover.py — Devesh * clearing backloag -test: add shutdown analytics tests for idempotency… — Devesh * fix: move stdlib base64 and json imports to module level in resolve\_integrations node — Enoch * fix: make test-cov use shell-safe Windows venv path — Sundaram Kumar Jha * fix(test): set TEST\_KEY before OpenAILLMClient in guardrail test — Anwesh * fix(tests): skip localhost-bound update smoke test when sockets are blocked — zerone0x ## Source pull requests * [#1257](https://github.com/Tracer-Cloud/opensre/pull/1257) fix(tests): skip localhost-bound update smoke test when sockets are blocked (author: zerone0x; contributors: zerone0x; labels: *none*; files: `tests/cli_smoke_test.py`) * [#1238](https://github.com/Tracer-Cloud/opensre/pull/1238) test(tools): add dedicated unit tests for Snowflake, OpenObserve, OpenSearch tools (#790) (author: Powlisher; contributors: Paul Demarecaux and Powlisher; labels: *none*; files: `tests/tools/conftest.py`, `tests/tools/test_openobserve_logs_tool.py`, `tests/tools/test_opensearch_analytics_tool.py`, `tests/tools/test_snowflake_query_history_tool.py`) * [#1236](https://github.com/Tracer-Cloud/opensre/pull/1236) fix: make test-cov use shell-safe Windows venv path (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `Makefile`) * [#1192](https://github.com/Tracer-Cloud/opensre/pull/1192) feat(cli): allow switching toolcall model at runtime (#1182) (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `app/cli/interactive_shell/cli_agent.py`, `app/cli/interactive_shell/commands.py`, `app/cli/wizard/config.py`, `tests/cli/interactive_shell/test_cli_agent.py`, `tests/cli/interactive_shell/test_commands.py`) * [#1262](https://github.com/Tracer-Cloud/opensre/pull/1262) test: cover \_check\_memory\_health threshold branches (author: Kevin Espiñeira; contributors: Kevin Espiñeira; labels: *none*; files: `tests/remote/test_server.py`) * [#1266](https://github.com/Tracer-Cloud/opensre/pull/1266) fix: move stdlib base64 and json imports to module level in resolve\_integrations node (author: Enoch; contributors: Enoch; labels: *none*; files: `app/nodes/resolve_integrations/node.py`) * [#1270](https://github.com/Tracer-Cloud/opensre/pull/1270) clearing backlogs -unit tests for app/cli/tests/discover.py (author: Devesh; contributors: Devesh; labels: *none*; files: `tests/cli/test_discover.py`) * [#1271](https://github.com/Tracer-Cloud/opensre/pull/1271) clearing backloag -test: add shutdown analytics tests for idempotency… (author: Devesh; contributors: Devesh; labels: *none*; files: `tests/analytics/test_provider.py`) * [#1264](https://github.com/Tracer-Cloud/opensre/pull/1264) tests(kafka): add unit tests for KafkaTopicHealthTool and KafkaConsumerGroupTool (author: Yash Kumar Saini; contributors: Yash Kumar Saini and Yash Kumar Saini; labels: *none*; files: `tests/tools/test_kafka_consumer_group_tool.py`, `tests/tools/test_kafka_topic_health_tool.py`) * [#1279](https://github.com/Tracer-Cloud/opensre/pull/1279) refactor: improve RDS storage-full RCA evidence handling (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: *none*; files: `app/nodes/investigate/processing/post_process.py`, `app/nodes/plan_actions/build_prompt.py`, `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tools/GrafanaAlertRulesTool/__init__.py`, `tools/utils/metric_summary.py`, `tests/nodes/root_cause_diagnosis/test_rds_grafana_evidence.py`, `tests/tools/test_grafana_alert_rules_tool.py`, `tests/tools/test_grafana_metrics_tool.py`) * [#1280](https://github.com/Tracer-Cloud/opensre/pull/1280) fix(test): set TEST\_KEY before OpenAILLMClient in guardrail test (author: Anwesh; contributors: Anwesh; labels: *none*; files: `tests/test_guardrails/test_llm_integration.py`) * [#1261](https://github.com/Tracer-Cloud/opensre/pull/1261) docs: add LLM providers reference page (author: DevNinja; contributors: DevNinja; labels: `pending triage`; files: `docs/docs.json`, `docs/llm-providers.mdx`) * [#1282](https://github.com/Tracer-Cloud/opensre/pull/1282) test(coralogix): add direct unit tests for CoralogixClient (author: Aniket Rawat; contributors: Aniket Rawat; labels: *none*; files: `tests/services/test_coralogix_client.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-05-03T23:00:00+00:00` to `2026-05-04T23:00:00+00:00` # Daily Update — 2026-05-05 Source: https://opensre.com/docs/daily-updates/2026-05-05 OpenSRE engineering daily update for 2026-05-05 (Europe/London) Thanks to everyone who contributed yesterday: Aayush Pratap Singh, Aayush Pratap Singh, Anwesh, Aryan Jain, Awokoya Olawale Davidson, Ceren Camkiran, Devesh, igor-berger, Jet, MAREDDY SAI TEJAS, mazenessam77, Richard Coker, Rohit Rajan, Rohit Rajan, Shubham Solanke, Sundaram Kumar Jha, vidhishah2209, vincenthus, and yashksaini-coder 🙏🚀 ## Main updates shipped (May 5, 2026) * feat(integrations): support Cursor Agent CLI as non-interactive LLM backend — Ceren Camkiran * feat(opencode-cli): add opencode cli as llm cli provider — Richard Coker * feat(rds): RDS integration with describe instance and events tools — mazenessam77 * feat(integrations): support Kimi Code CLI as non-interactive LLM backend — Aryan Jain * fix(analytics): persist unique CLI install IDs — vincenthus * fix(llm-cli): harden codex and claude-code auth probe paths — Anwesh * fix(claude-code): parse auth status JSON before exit code; use authMethod (#1260) — Awokoya Olawale Davidson * fix/added universal Bedrock model support using Converse API. fixes #1306 — Shubham Solanke * fix(rca): surface bad-query Performance Insights evidence — Sundaram Kumar Jha * fix(analytics): honour OPENSRE\_NO\_TELEMETRY to prevent CLI smoke test timeouts — MAREDDY SAI TEJAS * chore(deps): commit uv.lock and lock CI to uv sync — Anwesh * fix(llm): lazy OpenAI client init fix regression tests and Run full tests in PR CI — Anwesh * fix(cli): remove summary block from onboard completion output — Rohit Rajan * fix: improve WAL-driven replication lag RCA (scenario 001) + add QA validation — Ceren Camkiran * fix(release-announcements): include role mention in Discord release announcement — Devesh * test(airflow-routing): require connection\_verified on openclaw fixture — Awokoya Olawale Davidson * test: add filesystem fallback tests for analytics provider (#1119) — vidhishah2209 * docs: fix contributing guide links — Aayush Pratap Singh * revert(ci): skip coverage artifact upload on fork PRs — Anwesh * test(wizard): fix Claude Code flow test after demo step removal — Anwesh ## Source pull requests * [#1291](https://github.com/Tracer-Cloud/opensre/pull/1291) fix(llm): lazy OpenAI client init fix regression tests and Run full tests in PR CI (author: Anwesh; contributors: Anwesh; labels: `ci:windows`; files: `.github/workflows/ci.yml`, `services/llm_client.py`, `tests/services/test_llm_client.py`, `tests/test_guardrails/test_llm_integration.py`) * [#1276](https://github.com/Tracer-Cloud/opensre/pull/1276) test: add filesystem fallback tests for analytics provider (#1119) (author: vidhishah2209; contributors: Anwesh and vidhishah2209; labels: *none*; files: `tests/analytics/test_provider.py`) * [#1296](https://github.com/Tracer-Cloud/opensre/pull/1296) revert(ci): skip coverage artifact upload on fork PRs (author: Anwesh; contributors: Anwesh; labels: *none*; files: `.github/workflows/ci.yml`) * [#1293](https://github.com/Tracer-Cloud/opensre/pull/1293) fix(cli): remove summary block from onboard completion output (author: Rohit Rajan; contributors: Rohit Rajan and Rohit Rajan; labels: *none*; files: `app/cli/wizard/flow.py`, `tests/cli/wizard/test_flow.py`, `tests/cli_smoke_test.py`) * [#1295](https://github.com/Tracer-Cloud/opensre/pull/1295) fix(llm-cli): harden codex and claude-code auth probe paths (author: Anwesh; contributors: Anwesh; labels: `ci:windows`; files: `app/cli/commands/doctor.py`, `app/cli/interactive_shell/cli_agent.py`, `app/integrations/llm_cli/AGENTS.md`, `app/integrations/llm_cli/binary_resolver.py`, `app/integrations/llm_cli/claude_code.py`, `app/integrations/llm_cli/codex.py`, `app/integrations/llm_cli/runner.py`, `app/integrations/llm_cli/subprocess_env.py`, `tests/cli/interactive_shell/test_cli_agent.py`, `tests/cli/test_doctor.py`, and 6 more) * [#1297](https://github.com/Tracer-Cloud/opensre/pull/1297) test(wizard): fix Claude Code flow test after demo step removal (author: Anwesh; contributors: Anwesh; labels: *none*; files: `tests/cli/wizard/test_flow.py`) * [#1303](https://github.com/Tracer-Cloud/opensre/pull/1303) test(airflow-routing): require connection\_verified on openclaw fixture (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `tests/nodes/plan_actions/test_airflow_routing.py`) * [#1302](https://github.com/Tracer-Cloud/opensre/pull/1302) fix(claude-code): parse auth status JSON before exit code; use authMethod (#1260) (author: Awokoya Olawale Davidson; contributors: Awokoya Olawale Davidson; labels: *none*; files: `app/integrations/llm_cli/claude_code.py`, `tests/integrations/llm_cli/test_claude_code_adapter.py`) * [#1149](https://github.com/Tracer-Cloud/opensre/pull/1149) feat(integrations): support Cursor Agent CLI as non-interactive LLM backend (author: Ceren Camkiran; contributors: Anwesh and Ceren Camkiran; labels: `llm-cli`; files: `.env.example`, `app/cli/investigation/investigate.py`, `app/cli/wizard/config.py`, `config/config.py`, `app/integrations/llm_cli/AGENTS.md`, `app/integrations/llm_cli/__init__.py`, `app/integrations/llm_cli/cursor.py`, `app/integrations/llm_cli/errors.py`, `app/integrations/llm_cli/registry.py`, `app/integrations/llm_cli/runner.py`, and 4 more) * [#1310](https://github.com/Tracer-Cloud/opensre/pull/1310) fix(release-announcements): include role mention in Discord release announcement (author: Devesh; contributors: Devesh; labels: *none*; files: `.github/workflows/release-announcements.yml`) * [#1312](https://github.com/Tracer-Cloud/opensre/pull/1312) docs: fix contributing guide links (author: Aayush Pratap Singh; contributors: Aayush Pratap Singh and Aayush Pratap Singh; labels: *none*; files: `CONTRIBUTING.md`) * [#1196](https://github.com/Tracer-Cloud/opensre/pull/1196) feat(opencode-cli): add opencode cli as llm cli provider (author: Richard Coker; contributors: Anwesh and Richard Coker; labels: `llm-cli`, `needs-work`; files: `.env.example`, `app/cli/wizard/config.py`, `app/cli/wizard/flow.py`, `config/config.py`, `app/integrations/llm_cli/AGENTS.md`, `app/integrations/llm_cli/claude_code.py`, `app/integrations/llm_cli/codex.py`, `app/integrations/llm_cli/cursor.py`, `app/integrations/llm_cli/env_overrides.py`, `app/integrations/llm_cli/opencode.py`, and 6 more) * [#1292](https://github.com/Tracer-Cloud/opensre/pull/1292) chore(deps): commit uv.lock and lock CI to uv sync (author: Anwesh; contributors: Anwesh; labels: `ci:windows`, `pending triage`; files: `.github/dependabot.yml`, `.github/workflows/ci-labels-windows.yml`, `.github/workflows/ci.yml`, `.github/workflows/synthetic-tests.yml`, `.github/workflows/weekly-synthetic-and-deps-canary.yml`, `.gitignore`, `CONTRIBUTING.md`, `Makefile`, `README.md`, `SETUP.md`, and 1 more) * [#1321](https://github.com/Tracer-Cloud/opensre/pull/1321) chore(deps): bump types-requests from 2.33.0.20260408 to 2.33.0.20260503 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1331](https://github.com/Tracer-Cloud/opensre/pull/1331) chore(deps): bump ruff from 0.15.11 to 0.15.12 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1327](https://github.com/Tracer-Cloud/opensre/pull/1327) chore(deps): bump opentelemetry-instrumentation from 0.62b0 to 0.62b1 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1322](https://github.com/Tracer-Cloud/opensre/pull/1322) chore(deps): bump google-api-python-client from 2.194.0 to 2.195.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1333](https://github.com/Tracer-Cloud/opensre/pull/1333) chore(deps): bump legacy Anthropic integration package (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1332](https://github.com/Tracer-Cloud/opensre/pull/1332) chore(deps): bump google-auth from 2.49.2 to 2.50.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1330](https://github.com/Tracer-Cloud/opensre/pull/1330) chore(deps): bump tzdata from 2026.1 to 2026.2 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1329](https://github.com/Tracer-Cloud/opensre/pull/1329) chore(deps): bump hosted tracing package (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1328](https://github.com/Tracer-Cloud/opensre/pull/1328) chore(deps): bump legacy LLM framework core package (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1326](https://github.com/Tracer-Cloud/opensre/pull/1326) chore(deps): bump cryptography from 46.0.7 to 48.0.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1325](https://github.com/Tracer-Cloud/opensre/pull/1325) chore(deps): bump boto3 from 1.42.94 to 1.43.3 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1324](https://github.com/Tracer-Cloud/opensre/pull/1324) chore(deps): bump opentelemetry-api from 1.41.0 to 1.41.1 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1323](https://github.com/Tracer-Cloud/opensre/pull/1323) chore(deps): bump fastapi from 0.136.0 to 0.136.1 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1337](https://github.com/Tracer-Cloud/opensre/pull/1337) chore(deps): bump opentelemetry-instrumentation-requests from 0.62b0 to 0.62b1 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1336](https://github.com/Tracer-Cloud/opensre/pull/1336) chore(deps): bump opentelemetry-instrumentation-botocore from 0.62b0 to 0.62b1 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1335](https://github.com/Tracer-Cloud/opensre/pull/1335) chore(deps): bump anthropic from 0.96.0 to 0.99.0 (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1334](https://github.com/Tracer-Cloud/opensre/pull/1334) chore(deps): bump legacy OpenAI integration package (author: dependabot\[bot]; contributors: no human contributors recorded in merged PRs today; labels: `dependencies`, `python:uv`; files: `uv.lock`) * [#1214](https://github.com/Tracer-Cloud/opensre/pull/1214) fix: improve WAL-driven replication lag RCA (scenario 001) + add QA validation (author: Ceren Camkiran; contributors: Ceren Camkiran; labels: `pending triage`; files: `tests/synthetic/rds_postgres/001-replication-lag/QA_VALIDATION.md`, `tests/synthetic/rds_postgres/001-replication-lag/answer.yml`, `tests/synthetic/rds_postgres/run_suite.py`) * [#1268](https://github.com/Tracer-Cloud/opensre/pull/1268) fix(rca): surface bad-query Performance Insights evidence (author: Sundaram Kumar Jha; contributors: Sundaram Kumar Jha; labels: `pending triage`; files: `app/nodes/root_cause_diagnosis/prompt_builder.py`, `tests/nodes/root_cause_diagnosis/test_prompt_builder.py`) * [#1308](https://github.com/Tracer-Cloud/opensre/pull/1308) fix/added universal Bedrock model support using Converse API. fixes #1306 (author: Shubham Solanke; contributors: Anwesh and Shubham Solanke; labels: `needs-work`; files: `.env.example`, `services/llm_client.py`, `docs/llm-providers.mdx`, `tests/services/test_llm_client.py`) * [#1287](https://github.com/Tracer-Cloud/opensre/pull/1287) feat(rds): RDS integration with describe instance and events tools (author: mazenessam77; contributors: igor-berger and mazenessam77; labels: `pending triage`; files: `app/integrations/_catalog_impl.py`, `app/integrations/rds.py`, `app/nodes/plan_actions/detect_sources.py`, `tools/RDSDescribeInstanceTool/__init__.py`, `tools/RDSEventsTool/__init__.py`, `app/types/evidence.py`, `docs/docs.json`, `docs/rds.mdx`, `tests/integrations/test_rds.py`, `tests/tools/test_rds_tools.py`) * [#1139](https://github.com/Tracer-Cloud/opensre/pull/1139) feat(integrations): support Kimi Code CLI as non-interactive LLM backend (author: Aryan Jain; contributors: Anwesh, Aryan Jain, and yashksaini-coder; labels: `llm-cli`, `needs-work`; files: `.env.example`, `app/cli/wizard/config.py`, `app/cli/wizard/flow.py`, `config/config.py`, `app/integrations/llm_cli/AGENTS.md`, `app/integrations/llm_cli/binary_resolver.py`, `app/integrations/llm_cli/kimi.py`, `app/integrations/llm_cli/registry.py`, `app/integrations/llm_cli/runner.py`, `app/integrations/llm_cli/subprocess_env.py`, and 2 more) * [#1348](https://github.com/Tracer-Cloud/opensre/pull/1348) fix(analytics): persist unique CLI install IDs (author: vincenthus; contributors: vincenthus; labels: *none*; files: `.env.example`, `README.md`, `app/analytics/events.py`, `app/analytics/provider.py`, `app/cli/__main__.py`, `app/cli/commands/agent.py`, `app/cli/commands/deploy.py`, `app/cli/commands/general.py`, `app/cli/commands/integrations.py`, `app/cli/commands/tests.py`, and 32 more) * [#1350](https://github.com/Tracer-Cloud/opensre/pull/1350) fix(analytics): honour OPENSRE\_NO\_TELEMETRY to prevent CLI smoke test timeouts (author: MAREDDY SAI TEJAS; contributors: Jet, MAREDDY SAI TEJAS, and vincenthus; labels: *none*; files: `app/analytics/provider.py`, `tests/analytics/test_provider.py`, `tests/cli/test_main.py`) ## Generation metadata * Generator version: `opensre 2026.4.5` * Fallback summary used: `no` * UTC window: `2026-05-04T23:00:00+00:00` to `2026-05-05T23:00:00+00:00` # Daily Updates Source: https://opensre.com/docs/daily-updates/overview OpenSRE engineering daily updates from merged pull requests Daily updates are generated each evening (Europe/London) from the pull requests merged that day. ## Latest Updates | Date | Highlights | | - | - | | 2026-05-05 | [View](/docs/daily-updates/2026-05-05) — feat(integrations): support Cursor Agent CLI as... | | 2026-05-04 | [View](/docs/daily-updates/2026-05-04) — feat(cli): allow switching toolcall model at runtime (#1182) | | 2026-05-03 | [View](/docs/daily-updates/2026-05-03) — feat(integrations): add VictoriaLogs integration (#126) | | 2026-05-02 | [View](/docs/daily-updates/2026-05-02) — feat(cli): make interactive shell documentation-aware... | | 2026-05-01 | [View](/docs/daily-updates/2026-05-01) — chore: refactor integrations | | 2026-04-30 | [View](/docs/daily-updates/2026-04-30) — Issue/319 splunk integration | ## Archive | Date | Link | | - | - | | 2026-04-29 | [View](/docs/daily-updates/2026-04-29) | | 2026-04-28 | [View](/docs/daily-updates/2026-04-28) | | 2026-04-27 | [View](/docs/daily-updates/2026-04-27) | | 2026-04-26 | [View](/docs/daily-updates/2026-04-26) | | 2026-04-25 | [View](/docs/daily-updates/2026-04-25) | | 2026-04-24 | [View](/docs/daily-updates/2026-04-24) | | 2026-04-23 | [View](/docs/daily-updates/2026-04-23) | | 2026-04-22 | [View](/docs/daily-updates/2026-04-22) | | 2026-04-21 | [View](/docs/daily-updates/2026-04-21) | | 2026-04-20 | [View](/docs/daily-updates/2026-04-20) | | 2026-04-19 | [View](/docs/daily-updates/2026-04-19) | | 2026-04-18 | [View](/docs/daily-updates/2026-04-18) | | 2026-04-17 | [View](/docs/daily-updates/2026-04-17) | | 2026-04-16 | [View](/docs/daily-updates/2026-04-16) | | 2026-04-15 | [View](/docs/daily-updates/2026-04-15) | | 2026-04-14 | [View](/docs/daily-updates/2026-04-14) | | 2026-04-13 | [View](/docs/daily-updates/2026-04-13) | | 2026-04-12 | [View](/docs/daily-updates/2026-04-12) | | 2026-04-11 | [View](/docs/daily-updates/2026-04-11) | | 2026-04-10 | [View](/docs/daily-updates/2026-04-10) | | 2026-04-09 | [View](/docs/daily-updates/2026-04-09) | | 2026-04-08 | [View](/docs/daily-updates/2026-04-08) | | 2026-04-07 | [View](/docs/daily-updates/2026-04-07) | | 2026-04-06 | [View](/docs/daily-updates/2026-04-06) | | 2026-04-05 | [View](/docs/daily-updates/2026-04-05) | | 2026-04-04 | [View](/docs/daily-updates/2026-04-04) | | 2026-04-03 | [View](/docs/daily-updates/2026-04-03) | | 2026-04-02 | [View](/docs/daily-updates/2026-04-02) | # Features Source: https://opensre.com/docs/getting-started/features What OpenSRE can do today — tools, shell, privacy, and more. ## Highlights > **[CI/CD](/docs/cicd/overview)** > > Measure CI reliability, fix failing checks, and keep pull requests green with an agent. > **[Interactive shell](/docs/getting-started/interactive-shell-commands)** > > Describe incidents in plain English, or drive everything with slash commands. > **[Integrations](/docs/integrations)** > > 60+ tools across monitoring, incidents, cloud, code, chat, and data. > **[Goals](/docs/getting-started/goals)** > > Give the agent a finish line and let it work across several turns. > **[Masking](/docs/platform/masking)** > > Hide pods, hosts, and account IDs from external models — restore them in the answer. > **[Messaging](/docs/integrations/messaging)** > > Deliver results to Slack, Discord, Telegram, and more. > **[APIs](/docs/guides/api)** > > HTTP and Python APIs for alert intake and in-process agents. ## Full list **CI/CD:** * 30-day CI reliability report for one repository, with developer hours lost and a comparison to two open-source projects — [CI/CD analytics demo](/docs/cicd/cicd-analytics-demo) * Fix failing GitHub Actions checks on a pull request or branch and push the repair — [Fix GitHub CI](/docs/cicd/github-ci-fix) * A recurring reliability agent, a bounded repair loop for one PR, and the managed service for organizations — [Keep CI green with an agent](/docs/cicd/ci-repair-agent) * One read-only scan for every failing PR and branch across your organizations — [Scan CI health](/docs/cicd/github-ci-health-scan) **Integrations:** * 60+ tools across observability, incidents, cloud, code, messaging, databases, and workflows — [Integrations overview](/docs/integrations) * Setup and verify with `opensre integrations setup` and `opensre integrations verify` * Multi-instance providers for prod/staging pairs — [Multi-instance integrations](/docs/platform/multi-instance-integrations) * Bring-your-own LLM: Anthropic, OpenAI, Gemini, Ollama, Bedrock, OpenRouter, and more — [LLM providers](/docs/platform/llm-providers) * MCP connections from the shell (`/mcp`) **Interactive shell:** * Plain-language prompts that map to tools and slash commands — [Assistant](/docs/getting-started/interactive-shell-assistant) * Full slash-command surface via `/help` — [Shell commands](/docs/getting-started/interactive-shell-commands) * Goals that persist across turns until the finish line is met — [Goals](/docs/getting-started/goals) * Sessions you can list, resume, compact, and branch — [Sessions](/docs/platform/sessions) * Long-term memory across chats — [Memory](/docs/platform/memory) * Reasoning effort control (`/effort` or `OPENSRE_REASONING_EFFORT`) **Privacy and safety:** * Reversible identifier masking before external LLM calls — [Masking](/docs/platform/masking) * Command-history privacy controls — [Interactive shell privacy](/docs/platform/interactive-shell-privacy) * Guardrails for hard-blocked secrets and PII **Messaging and automation:** * Chat delivery and triggers: Slack, Discord, Telegram, and more — [Messaging](/docs/integrations/messaging) * Scheduled digests, recurring prompt loops, and delivery jobs — [Cron](/docs/platform/cron) **APIs and deployment:** * HTTP API for health checks and alert intake — [HTTP API](/docs/guides/api) * Python API for in-process agents — [Python API](/docs/guides/python-api) * Local CLI, an organization-managed gateway, gateway daemon (`opensre gateway start`), or host on Railway / ECS / Vercel — [Deployment](/docs/guides/deployment) * Local and enterprise install paths — [Install](/docs/install) ## Related > **[Showcase](/docs/getting-started/showcase)** > > See a first run and sample output. > **[Quickstart](/docs/getting-started/quickstart)** > > Install, sign in, and ask your first question in a few minutes. > **[Hosting the agent](/docs/guides/hosting-the-agent)** > > Embed the agent with your own sink, tools, and prompts. > **[Integrations overview](/docs/integrations)** > > Browse the full tool catalog by category. # Goals Source: https://opensre.com/docs/getting-started/goals Give OpenSRE a finish line with /goal and let it work through multi-step questions on its own Most questions are answered in one reply. Some are not — "how many Windows users signed up last week?" may need OpenSRE to check whether PostHog is connected, write a query, run it, and only then answer. `/goal` gives OpenSRE a **finish line** instead of a single prompt. You state the condition once, and OpenSRE keeps taking turns until it reaches it, runs out of its turn budget, or you stop it. You can walk away and check the progress line when you come back. ## Commands at a glance | Command | What it does | | - | - | | `/goal` | Show the current goal, or say there is none | | `/goal show` | Same as bare `/goal` (`/goal status` also works) | | `/goal set ` | Set a finish line and start working on it immediately | | `/goal set --max-turns N ` | Same, but cap how many turns it may take | | `/goal ` | Shorthand for `/goal set ` | | `/goal pause` | Stop the next turn, keep the condition and progress | | `/goal resume` | Start working again from where it paused | | `/goal edit [condition]` | Review or change the condition without losing progress | | `/goal clear` | Drop the goal entirely (`/goal unset` also works) | ## Try it Paste this into the interactive shell: ```text theme={null} /goal set --max-turns 4 How many Windows users in the last 7 days? Prefer live PostHog if connected; otherwise draft the HogQL and end with one setup CTA. Do not ask whether to continue. ``` OpenSRE confirms the goal and immediately starts on it: ```text theme={null} ◎ /goal active · 0s · turn 0/4 · +0 tokens condition: How many Windows users in the last 7 days? Prefer live PostHog if connected; otherwise draft the HogQL and end with one setup CTA. Do not ask whether to continue. reason: waiting for an achieved signal → next: condition runs as its own prompt turn (look for [N] ❯ below). /goal · pause · resume · edit · clear ``` Setting a goal **starts the work** — you do not need to retype the condition as a prompt. The confirmation turn itself does not count against the budget, so the first real attempt is turn 1 of 4. > [!TIP] > > Write the condition the way you would brief a teammate: say what "done" looks > like, and say what to do when something is missing ("otherwise draft the query > and end with one setup CTA"). After every turn a separate small model reads > what OpenSRE reported and decides met, not yet, or impossible, so the finish > line has to be something the reply can show: a count, a table, a passing > check. Numbered steps in the condition become the checklist. ## Watching progress Run `/goal` at any time: ```text theme={null} ◎ /goal active · 1m 12s · turn 2/4 · +18.4k tokens condition: How many Windows users in the last 7 days? … reason: waiting for an achieved signal ``` | Part of the line | Meaning | | - | - | | `active` | Current state — see [How a goal ends](#how-a-goal-ends) | | `1m 12s` | Time since you set the goal | | `turn 2/4` | Turns used out of the budget | | `+18.4k tokens` | Tokens spent on this goal since you set it | | `condition` | The finish line you asked for | | `reason` | Why OpenSRE is still going, or why it stopped | When OpenSRE breaks the work into steps, `/goal` also prints a checklist with `[x]` for finished items and `→` marking the one it is on. If no goal is set, you get an empty state instead: ```text theme={null} no active goal. Set one with /goal set . ``` ## Pause and resume `/goal pause` stops the next turn from firing but keeps everything else — condition, turn count, and progress so far: ```text theme={null} ◎ /goal paused · 2m 05s · turn 2/4 · +18.4k tokens condition: How many Windows users in the last 7 days? … reason: paused by you paused. /goal · resume · edit · clear ``` While paused you can still chat normally — OpenSRE answers your message and then stays put instead of continuing the goal. This is the way to ask a quick side question without losing your place. `/goal resume` picks the work back up from the same turn count: ```text theme={null} /goal resume ``` ## Change the condition `/goal edit` rewrites the finish line in place, keeping the turns already spent. Use it when the goal was nearly right but you want to redirect it. In an interactive terminal, bare `/goal edit` opens the current condition for review. Enter edits the condition; **Save changes** applies it and Esc cancels. The turn budget stays unchanged. You can also supply the replacement directly: ```text theme={null} /goal edit How many Windows users in the last 7 days, broken down by country? ``` If the goal is active, the updated condition starts running right away. If it is paused, the text changes and it stays paused until you `/goal resume`. Conditions longer than 400 characters are shortened. ## Stop the goal | You want to | Do this | | - | - | | Take a break, keep the goal | `/goal pause` | | Interrupt the turn running right now | **Ctrl+C** | | Forget the goal completely | `/goal clear` | `/goal clear` prints `goal cleared.` and removes it — there is nothing left to resume. ## How a goal ends | State | What happened | | - | - | | `active` | Still working | | `paused` | You ran `/goal pause`, the goal made no progress for two turns and is waiting for your pick, or a turn failed (for example the model call was rejected). Fix the cause, then `/goal resume` | | `achieved` | The judge confirmed the condition from what OpenSRE did and reported, backed by at least one successful tool call | | `impossible` | The judge decided this session cannot meet the condition (missing access, contradicted facts). Read the reason, then set a different goal | | `budget_exhausted` | It used every turn without getting there — read the last reply, then set a narrower goal or raise `--max-turns` | | `cancelled` | You interrupted the run with **Ctrl+C** | | `cleared` | You ran `/goal clear` | The turn budget defaults to **5** and exists so an impossible condition cannot loop forever. Raise it with `--max-turns` when the work genuinely needs more steps. ## `/goal` vs `/work` vs `/loops` | Use | When | | - | - | | `/goal` | One question that needs several turns to answer, and you want OpenSRE to keep going by itself | | [`/work`](/docs/platform/work-management) | A durable human todo list that outlives the session | | [`/loops`](/docs/platform/cron) | A recurring prompt that runs on a schedule and delivers its result to Slack, Telegram, or the shell inbox | ## Where goals live A goal belongs to the session it was set in and is restored with it, so a goal you set in the shell is still there after `/resume`. Goals are remembered in chat transports such as Slack and Telegram as well, and the same status line appears in the timeline while OpenSRE works. `/goal set` starts the condition on that turn — the same as the interactive shell — and continuation proceeds from there. > [!NOTE] > > After `/goal set`, OpenSRE begins the work immediately. You do not need a > follow-up message to start the first turn. ## Related docs * [Interactive Shell Commands](/docs/getting-started/interactive-shell-commands) — the full slash command reference * [Work Management](/docs/platform/work-management) — `/work` durable todos * [Scheduled deliveries](/docs/platform/cron) — `/loops` recurring prompts * [Session History](/docs/platform/sessions) — how sessions persist and resume * [PostHog](/docs/integrations/monitoring/posthog) — connect PostHog so product questions can run live # Agent with connected tools Source: https://opensre.com/docs/getting-started/interactive-shell-assistant Ask the interactive shell in plain language — it can pull live data from your integrations before answering. In the interactive shell (`opensre`), ask a free-form question. If the answer needs data from a connected integration (GitHub, Datadog, Sentry, Grafana, and others), the agent calls those tools and answers from their results. You do not pick a tool or run a slash command for this. Just ask. ## Example ```text theme={null} > can you check the github issues for the windows crashes in https://github.com/Tracer-Cloud/opensre? · gathering via GitHub · search issues — windows crashes… There are 3 open issues mentioning Windows crashes. The most recent, #482 "App crashes on launch (Windows 11)", reports a startup segfault after the 2.4.0 update… ``` ```text theme={null} > what do the datadog logs say about payment-service errors in the last hour? · gathering via Datadog · query logs — payment-service errors… Found 847 error log lines in the last hour for service:payment-service. Top pattern: ConnectionTimeout to postgres-primary… ``` The dim progress line shows which integration and tool ran. For GitHub, you can put the repo URL in the question, mention it earlier in the session, set `GITHUB_REPOSITORY`, or let OpenSRE use the current checkout's `origin` remote when GitHub is connected. ## How it works 1. **You ask** a free-form question (not a `/slash` command). 2. **OpenSRE decides** whether live data would help and calls the matching chat tools. 3. **The ReAct loop is bounded** so the shell stays responsive. Tool results are returned to the same model before it answers. 4. **Tools are read-only data fetches** — they run without a per-call confirmation (write actions ask for approval). 5. **Progress** shows on the dim `· gathering via <source> · <tool> — <hint>…` line while tools run. 6. **The agent answers** using that evidence and recent chat context. What happens when something goes wrong: | Situation | Result | | - | - | | Ctrl+C during tool use | The turn stops | | Tool or integration unavailable | The agent reports what is unavailable | | No usable tools for the question | The agent answers without a tool call | ## Setup You need an LLM and at least one integration for live tool calls. **1. LLM** Sign in when `opensre` starts; the signed-in shell uses the account-hosted model. For a self-hosted gateway or headless jobs on a machine that is not signed in, set `LLM_PROVIDER` plus the provider API key — see [LLM providers](/docs/platform/llm-providers). In the shell, `/model show` names the model in use. **2. Connect integrations** ```bash theme={null} opensre integrations setup datadog opensre integrations setup github # or in the shell: /integrations setup /integrations list ``` Only **connected and verified** integrations contribute tools. **3. Check that tools are available** ```text theme={null} /health # which integrations look healthy /verify datadog # one service /tools # registered tools ``` If `/health` is red or `/tools` is empty for that source, fix setup first. **4. Ask** ```text theme={null} what do the datadog logs say about payment-service errors in the last hour? ``` You should see a tool progress line when a tool runs. If you do not, the question may not need live data or that integration may be unavailable. With **no** integrations connected, behavior is unchanged: answers come from text and docs only. Connect a source and matching questions start returning live data. | You ask | Typical source | | - | - | | Check GitHub issues for Windows crashes | GitHub | | What do Datadog logs say about payment-service? | Datadog | | Recent Sentry errors on checkout? | Sentry | | Error rate in Grafana for prod-api? | Grafana | ## Related * [Shell commands](/docs/getting-started/interactive-shell-commands) * [Integrations overview](/docs/integrations) * [GitHub](/docs/integrations/code/github) # Shell commands Source: https://opensre.com/docs/getting-started/interactive-shell-commands Slash commands in the OpenSRE interactive shell — usage detail for each command. Start the shell with `opensre` (needs a real terminal). Type a `/command`, or describe what you want in plain language. ```bash theme={null} opensre /help ``` Type `/` to open the command tray attached to the input. Browse with the arrow keys and use Enter or Tab on a command with subcommands to add the separating space and open its subcommand tray. Enter runs a final command selection. Esc closes the tray without clearing your draft or stopping running work. On narrow terminals, the highlighted command's description appears below the list. > [!NOTE] > > Confirmation prompts come from **`/auto`** (tool-approval autonomy) and from > elevated slash commands. Default `/auto` is **high** (alpha: no tool prompts). > `/trust on` skips approval prompts for the rest of the session. Outside a TTY, > actions that would ask are blocked. ## How the REPL works | Input type | What happens | | - | - | | Slash command (`/status`) | Routed through the action agent and executed via `slash_invoke` | | Plain language (`verify datadog`) | Routed by the action agent to slash commands, tool calls, or doc-grounded answers | | Pasted alert JSON/text | The agent triages it in chat, calling integration tools as needed | | Shell one-liner (`kubectl get pods`) | Executed through shell policy when the action agent selects a shell action | ### Shell command syntax `shell_run` uses the command shell of the environment where OpenSRE is running: * Native Windows uses `cmd.exe`. Use `cmd` operators (`&`, `&&`, and `|`), `%NAME%` environment variables, and caret escaping (`^`). PowerShell syntax is not translated; invoke `powershell -NoProfile -Command "..."` or `pwsh -NoProfile -Command "..."` explicitly when you need it. * macOS, Linux, and OpenSRE running inside WSL use non-interactive `/bin/sh`. WSL therefore expects POSIX syntax, not native Windows syntax. Each `shell_run` call starts in OpenSRE's working directory with a fresh child shell. A `cd` affects only that call; prefix every command that needs another directory with `cd && ...` (`cd /d && ...` in native Windows when the path may be on another drive). The exact command `pwd` works on every platform and prints that call's starting directory. **TTY vs non-TTY:** Interactive menus, confirmations, and onboarding wizards need a real terminal. Piping input or running in CI is non-interactive — prefer explicit CLI commands outside the REPL for automation. **Unknown commands:** Typos suggest the closest registered command (`Did you mean /integrations?`). Run `/help` for the authoritative list — it always matches your installed OpenSRE version. **Keyboard shortcuts:** | Key | Effect | | - | - | | **Up/Down** | Browse suggestions while the command tray is open; otherwise recall persisted command history (when enabled) | | **Tab** | Complete the highlighted suggestion; for commands with subcommands, open their tray | | **Enter** | Open the highlighted command's subcommand tray, or run a final selection | | **Esc** | Close the command tray first; otherwise stop running work or clear the draft | | **Ctrl+C** once | Cancel in-flight streaming work or interrupt the current prompt | | **Ctrl+C** twice within 2s | Exit the REPL (prints `/resume` hint) | | **Ctrl+D** | Exit when the prompt is empty and nothing is running (same resume hint as `/exit`) | Every launch asks what you want to do: analyze a repository, keep CI/CD healthy automatically, or open the shell. Keeping CI/CD healthy then asks for a local repair, one repair in OpenSRE Cloud, or Slack. A local or cloud repair asks next for permission to create one private demo repository (`owner/opensre-ci-repair-demo-` plus four characters when your GitHub login is saved). Declining leaves the later repository or pull-request picker. Escape skips the picker. Run `opensre --skip-onboarding` to open the shell without it. Type `/demo` to start it later. See [CI/CD analytics demo](/docs/cicd/cicd-analytics-demo). The banner and prompt show **CI/CD fixes**: distinct GitHub repairs whose pushed checks passed, including merge-conflict repairs. A repair can count before it is merged; check the target branch before assuming it is green. Fixes in this shell update the prompt immediately. Use `/clear` to pick up fixes saved by another OpenSRE process in the same deployment. Zero also covers an unavailable count; the total is best-effort and can decrease after restarting if a save failed. ## Quick reference ### Help and exit | Command | What it does | | - | - | | `/help` | List commands or show help for one command or category | | `/?` | Shortcut for `/help` | | `/exit` | Leave the interactive shell | | `/quit` | Alias for `/exit` | | `/demo` | Open the guided demo picker (CI/CD analysis, reliability agent, managed service, Slack) | ### Session | Command | What it does | | - | - | | `/status` | Session summary — interactions, alerts, provider, effort, trust mode | | `/cost` | Token usage and LLM call count for the current session | | `/context` | What the next model call will contain, and how big each part is | | `/effort` | Set or show reasoning effort (`low` … `max`) | | `/auto` | Tool-approval autonomy (`off` / `low` / `med` / `high`; default `high`) | | `/trust` | Enable or disable trust mode (skip confirmation prompts) | | `/verbose` | Toggle verbose logging | | `/clear` | Clear the screen and re-render the banner | | `/compact` | Summarize older context into a replayable compaction entry | | `/sessions` | Browse recent sessions; select one with ↑/↓ and Enter in a terminal | | `/resume ` | Restore a past session's conversation context | | `/resume :` | Restore a specific session branch point | | `/new` | Start a new session file while keeping LLM context | | `/rename ` | Rename the current session (`/rename --reset` clears the name) | | `/goal` | Set a finish line and keep working toward it across turns — see [Goals](/docs/getting-started/goals) | ### Integrations, models, and tools | Command | What it does | | - | - | | `/health` | Integration and agent health check | | `/verify` | Verify integration connectivity (all or one service) | | `/integrations` | List, verify, show, setup, or remove integrations | | `/mcp` | List, connect, or disconnect MCP servers | | `/model` | Show the account-managed hosted model | | `/tools` | List registered chat and action tools | ### Privacy and history | Command | What it does | | - | - | | `/history` | Show or manage persisted command history | | `/privacy` | History persistence, redaction status, and threat model | | `/memory` | List and manage the agent's long-term memory | | `/knowledge` | Show workspace knowledge and check memory health (`show `, `audit`, `scan [service]`) | ### Tasks and background work | Command | What it does | | - | - | | `/loops` | List, create, run once, and debug recurring prompt loops | | `/tasks` | List recent and in-flight background tasks | | `/work` | List, add, complete, and prioritize human work items | | `/cancel ` | Cancel a running task by id **elevated** | | `/stop` | Guidance for stopping background work | | `/gateway` | Start, stop, or inspect the background gateway daemon | ### Alerts and selections | Command | What it does | | - | - | | `/alerts` | Alert listener inbox status | | `/choose` | Reopen the selection menu the agent is waiting on | ### CLI parity These commands delegate to the same Click CLI you would run outside the REPL. | Command | What it does | | - | - | | `/account` | Sign in to OpenSRE, inspect the local account, or sign out | | `/credits` | Remaining hosted credits for the signed-in account (same as `opensre credits`) | | `/setup` | First-run setup: account sign-in, hosted model, then the shell (same as `opensre setup`) | | `/auth` | Log in to LLM providers, show provider auth status, or clear credentials | | `/login` | Shortcut for `/auth login` (`/login openai`, `/login anthropic`, `/login deepseek`). Signs in an LLM provider, not the OpenSRE account | | `/logout` | Sign out of the OpenSRE account and close the shell (same as `/account logout`) | | `/onboard` | Interactive onboarding wizard | | `/remote` | Connect to and operate remote deployed agents | | `/remote-sync` | Manage remote sync to your object store. Subcommands: `status`, `sync`, `setup`. See [Sync to your own object store](/docs/install/remote-sync). | | `/config` | Show or edit `~/.opensre/config.yml` | | `/cron` | Manage scheduled delivery jobs | | `/sentry` | Schedule automated Sentry morning digests or uptime watches (Telegram or Slack required) | | `/messaging` | Telegram pairing and allowlist | | `/guardrails` | Sensitive-information guardrail rules | | `/update` | Check for updates and upgrade OpenSRE | | `/posthog` | Run or schedule PostHog per-metric summary reports | | `/runbooks` | Manage trusted runbook sources for guided investigations | | `/skills` | Publish skills and inspect the live skills release (`status`, `update`, `push `, `rollback`) | | `/debug` | Targeted runtime diagnostics | | `/uninstall` | Remove OpenSRE and local data **elevated** | ### System | Command | What it does | | - | - | | `/doctor` | Full local environment diagnostic | | `/version` | OpenSRE, Python, and OS versions | | `/theme` | Choose and persist the shell colour theme | ### Commands with interactive menus (TTY) These invocations open a picker or submenu: | Command | Menu behavior | | - | - | | `/help` | Browse categories; Enter runs the selected command | | `/integrations list` | Browse configured integrations; verify, setup, or remove the selected item | | `/mcp list` | Browse configured MCP servers; verify, connect, or disconnect | | `/model` | show / set / restore / toolcall | | `/theme` | Pick a colour theme from the installed list | | `/choose` | Reopen the selection menu the agent queued | | `/trust`, `/verbose` | on / off | | `/history` | show / clear / off / on / retention | | `/resume` | Arrow-key picker of recent conversations | `/integrations list` and `/mcp list` use the same searchable list as `/loops list`. Type to filter; click a row or use ↑↓/Tab to see its details below the list. New connections show **Checking…** while connectivity checks run in the background; results appear without pressing Enter. Search and navigation stay available while checks run. The list remembers this session’s last check result with a **Last checked** timestamp, including when you reopen it or switch between the two lists. **Enter** runs a fresh check and updates the result in place. Use **F2** for setup, **F3** to remove/disconnect (with confirmation), **F4** to add/connect, and **Esc** to close. Checks time out after 30 seconds; Esc cancels an in-progress check. These action keys leave ordinary typing available for search. Connections without a verifier show **Configured**, not a successful connectivity check. `/model` keeps its menu running: picking `show` prints the current models and hands you back to the menu, so leave it with **done** or Esc. ## Commands ### Help > **/help** > > ```text theme={null} > /help > /help /model > /help tasks > /help all > ``` > > | Form | Result | > | - | - | > | `/help` | Interactive picker in a TTY; category index otherwise | > | `/help <command>` | Detailed usage for one command (e.g. `/help /model`) | > | `/help <category>` | All commands in a section (`session`, `tasks`, …) | > | `/help all` | Full command index | > > Help categories mirror the REPL grouping: Quick Access, Session, Integrations/Models/Tools, Privacy, Tasks, Agents, Alerts, CLI parity, and System. > > **Quick Access** duplicates frequently used commands (`/integrations`, `/model`, `/health`, `/status`, `/help`) for faster discovery in the picker. > **/?** > > Shortcut for `/help`. Same forms and picker behavior. ### Session > **/status** > > Shows a snapshot of the **current** REPL session: > > ```text theme={null} > /status > ``` > > Typical fields: > > | Field | Meaning | > | - | - | > | `interactions` | Count of recorded turns (chat, slash, alerts) | > | `incoming alerts` | Alerts received by the local listener this session, plus age of the latest | > | `trust mode` | `on` / `off` | > | `reasoning effort` | Current `/effort` level (OpenAI/Codex only) | > | `provider` | Active `LLM_PROVIDER` | > | `accumulated context` | Comma-separated keys from prior turns (e.g. `service`, `cluster`) | > > Use `/status` for **session** state. Use `/health` for **integration connectivity**. > **/cost** > > Shows LLM usage tracked **locally** for the current REPL session. This is not cloud-provider billing and does not read your vendor invoice. > > ```text theme={null} > /cost > ``` > > Example output (measured + estimated mix): > > ```text theme={null} > Session cost (includes estimates) > > history entries 12 > llm calls 8 > input tokens 4,210 (measured: 1,200 · estimated: 3,010) > output tokens 890 (estimated: 890) > ``` > > | Field | Meaning | > | - | - | > | `history entries` | Lines recorded in this session's in-memory history | > | `llm calls` | LLM turns counted (planner + chat + help + follow-up) | > | `input tokens` / `output tokens` | Running totals for prompt and completion usage | > > **Measured vs estimated:** > > | Code path | Token source | > | - | - | > | Planner `invoke()` (action selection) | Provider usage metadata when the API returns it | > | Streaming chat, help, follow-up | Estimated from character length (\~4 characters per token) | > > When any estimate is included, the table title notes `(includes estimates)` and rows show measured vs estimated splits. > > **What increments the counter:** Normal LLM chat turns and planner calls. **What does not:** Non-LLM command handling such as history recall or prompt rendering. > > Token totals reset on `/new` or when session identity is rotated. They do **not** carry across `/resume`. See [Session History](/docs/platform/sessions). > **/context** > > Shows what the model will receive with your next message, without calling a model: > > ```text theme={null} > /context > ``` > > | Row | Meaning | > | - | - | > | one row per prompt block | The system prompt, rules, runtime facts, skills, connected integrations, memory, and the per-turn blocks, each with its tier, characters, and estimated tokens | > | `history` | Earlier turns replayed to the model, including their tool calls and results. Shown as `history after compaction` when the next turn compacts first: only the turns compaction keeps, without the summary it writes then | > | `total` | Blocks plus history; the message you type next and the tool schemas are not included | > > Below the table: how many messages and turns the history holds and how many of those turns carry tool results, whether a session summary opens it, how many tool schemas go with every call, and how close the history is to compaction (`OPENSRE_HISTORY_TOKEN_BUDGET` tokens or 60 turns). Past either, older turns are summarized before the next turn; `/compact` does that now. > **/effort** > > Set reasoning depth for **OpenAI and Codex** providers in this REPL session only. > > ```text theme={null} > /effort high > /effort > ``` > > | Level | When to use | > | - | - | > | `low` | Fast triage, simple lookups, lower token cost | > | `medium` | Default balance for most incident work | > | `high` | Deeper analysis when latency is acceptable | > | `xhigh`, `max` | Maximum reasoning; best with newer GPT-5 or Codex models — older models may reject these levels | > > Bare `/effort` prints the current level, the config default for your provider/model, and supported choices. `/status` includes the same field. > > Other providers (Anthropic, Ollama, etc.) ignore `/effort`; the shell prints a hint suggesting `/model set openai` or `/model set codex`. Set `OPENSRE_REASONING_EFFORT` in the environment for non-interactive defaults. See [LLM providers](/docs/platform/llm-providers). > **/auto** > > Set how often the REPL asks before running agent tools. Shown on the status line as `Auto (Med) · …`. > > ```text theme={null} > /auto > /auto med > /auto high > ``` > > | Level | Prompts before | > | - | - | > | `high` (default) | Nothing — alpha default; every tool runs immediately | > | `med` | Mutating agent tools: shell, code agent, slash/CLI, synthetic tests, LLM provider switch, Sentry issue fix, … | > | `low` | Everything `med` asks about, plus additional tool starts | > | `off` | Every tool type | > > `/auto` is **not** a shell-command allowlist: under alpha, shell strings still run when approved (or when High skips the prompt). Lower levels only promote the existing `ask` confirmation hook by `tool_type`. > > **With `/trust`:** `/trust on` skips approval prompts even when `/auto` would ask. Turn trust off to restore `/auto` prompts. > > Session preference only — not restored by `/resume`. Bare `/auto` prints the current level, the default, and each caption. > **/trust** > > Trust mode skips execution confirmation prompts for **elevated** commands (`/cancel`, `/uninstall`, integration remove, etc.). > > ```text theme={null} > /trust on > /trust off > /trust > ``` > > In a TTY, bare `/trust` opens an interactive on/off menu. Trust mode is a **session preference** — it is not restored by `/resume`. > **/verbose** > > Toggle `TRACER_VERBOSE` logging for the REPL process (deeper internal logs to stderr). Also shows the per-turn tool-call log above each reply; by default it is hidden and only reachable with `Ctrl+O`. > > ```text theme={null} > /verbose on > /verbose off > /verbose > ``` > **/clear** > > Clears the terminal and re-renders the OpenSRE banner. Does **not** reset session state, token usage, LLM conversation context, or accumulated infra context. > > ```text theme={null} > /clear > ``` > **/compact** > > Summarizes older conversation context, keeps recent messages, and persists a `compaction` entry in the session file. Future `/resume` calls replay the summary before the kept messages. > > ```text theme={null} > /compact > ``` > > If there are not enough messages to compact, the command reports nothing to compact. OpenSRE also compacts automatically before a shell turn when the replayed branch context exceeds the runtime threshold. > **/sessions** > > Browse up to **20** recent sessions stored on disk, newest first: > > ```text theme={null} > /sessions > ``` > > In a terminal, use **↑/↓** to highlight a session and **Enter** to resume it. **Esc** closes the picker. Rows show relative times, and the status line shows the selected session's ID, start time, duration, and turn count. The current session is marked; the most recent previous session is highlighted by default when one exists. > > When output is piped, `/sessions` prints a table instead. Use `/resume ` with the displayed 8-character ID. The current session's duration updates live. > **/resume** > > Restore LLM conversation context and accumulated infra context from a previous session. > > ```text theme={null} > /resume 9b2e4f7a > /resume 9b2e4f7a:abc123 > /resume redis > /resume > ``` > > | Form | Behavior | > | - | - | > | ID prefix | Match session UUID by prefix (must be unique) | > | ID + entry prefix | Replay the branch ending at the matching entry ID prefix | > | Name substring | Match when ≥3 characters and exactly one session matches (e.g. `/resume redis`) | > | Bare `/resume` | Arrow-key picker of recent conversations (TTY) | > > **Current session:** If the ID prefix matches the session you are already in, `/resume` is a no-op — OpenSRE prints a hint to pick a previous session from `/sessions`. > > When you resume a **different** session, OpenSRE: > > 1. Switches the active session file to the target session > 2. Restores `cli_agent_messages` so the assistant remembers prior turns > 3. Restores `accumulated_context` keys > 4. Reprints conversation history (user prompts, assistant replies, slash commands) > > Compaction entries are replayed as summary messages before the kept branch messages. > > **Restored vs not restored:** > > | | `/resume` | > | - | - | > | Conversation context | Yes | > | Infra context keys | Yes | > | Trust mode, reasoning effort | No — per-session preferences | > | Token usage / `/cost` counters | No — fresh counters for the resumed session identity | > | In-memory history count | Yes — turn stubs from the saved session | > > **Warning:** Resuming a different session replaces the current session's LLM context if messages already exist. > > Session replay reads the current version-2 session tree format. Older session files are ignored by `/sessions` and `/resume`. See [Session History](/docs/platform/sessions). > **/new** > > Rotate to a **new session file** while keeping the current LLM conversation thread and accumulated context. > > ```text theme={null} > /new > ``` > > ```text theme={null} > new session started — conversation context carried forward. > 14 messages in context · type to continue > ``` > > | Command | Session file | LLM context | Token counters | > | - | - | - | - | > | `/clear` | Same | Kept | Kept | > | `/new` | New UUID | Kept | Reset | > | `/resume` | Target session | Replaced with target | Reset for that session | > > Use `/new` after a long `/resume` so `/sessions` stays tidy without losing your place in the conversation. > **/goal** > > Set a finish line and keep working toward it across turns: > > ```text theme={null} > /goal > /goal set How many Windows users signed up last week? > /goal set --max-turns 4 Prefer live PostHog if connected > /goal pause > /goal resume > /goal edit > /goal edit > /goal clear > ``` > > Bare `/goal` shows the current goal. `/goal set` starts work immediately. > > In an interactive terminal, `/goal set` without a condition asks for one, then > shows the turn budget for review. The condition is required; max turns is > optional and empty leaves the goal unbounded. Use arrow keys or Tab/Shift-Tab > to move between rows, then Enter to edit a row or choose **Set goal**. Ctrl-S > attaches the goal from the summary and runs the condition as the next turn. > Esc returns from an edit without applying it; Esc on the summary cancels > without attaching anything. Fully specified commands still execute directly. > See [Goals](/docs/getting-started/goals) for the full workflow. > > Bare `/goal edit` reviews the current condition in the same form. Choose **Save changes** > to apply it. The existing turn budget and progress stay attached; an active goal > continues with the updated condition, while a paused goal stays paused. ### Integrations, models, and tools > **Choosing the right diagnostic command** > > | Question | Command | > | - | - | > | Are my integrations configured and reachable? | `/verify`, `/health`, or `/integrations verify` | > | Verify one integration by name? | `/verify datadog` or `/integrations verify datadog` | > | Show one integration's credentials/endpoints? | `/integrations show datadog` | > | Is Python/Docker/venv OK on this machine? | `/doctor` | > | What integrations exist in OpenSRE generally? | Ask in plain language (docs answer) — not `/integrations list` | > **/health** > > Read-only pass/fail report for the local OpenSRE agent, LLM connectivity, and each configured integration. > > ```text theme={null} > /health > ``` > > Runs live verification against the integration store. Use before incidents to confirm Datadog/Grafana/K8s credentials still work. For integration-only checks without the full agent/LLM report, prefer `/verify`. > **/verify** > > Shortcut for integration connectivity checks (same as `opensre integrations verify` / `make verify-integrations`). > > ```text theme={null} > /verify > /verify datadog > /verify telegram > ``` > > | Form | Behavior | > | - | - | > | Bare `/verify` | Verify all integrations; prints status table plus `all integrations ok` or `N integration(s) need attention` | > | `/verify <service>` | Verify one named integration | > **/integrations** > > ```text theme={null} > /integrations list > /integrations verify > /integrations verify datadog > /integrations show datadog > /integrations setup datadog > /integrations setup telegram > /integrations remove datadog > ``` > > | Subcommand | Behavior | > | - | - | > | `list` | Browse configured integrations in a TTY; verify and render a status table otherwise | > | `verify` | Verify all integrations; prints summary line | > | `verify <service>` | Verify one integration (same as `/verify <service>`) | > | `show <service>` | Verify one service and print key/value config (masked secrets) | > | `setup <service>` | Guided setup with sign-in instructions, discovery, and verification | > | `remove <service>` | Remove from local store **elevated** | > > GitHub setup opens the OpenSRE app, where you connect through OAuth or enter a token. Opening the app leaves setup pending until the connection is refreshed. > > You can ask “Can you configure Telegram/PostHog for me?” to start the same guided flow. It explains each human action before requesting input, discovers IDs where possible, and lets you retry failed checks. Enter credentials only in the setup flow's hidden input. > > `/integrations list` opens a searchable configured-integration browser in a TTY. Click a row or use **↑↓/Tab** to see its details; **Enter** runs a fresh connectivity check when supported. Use **F2** for setup, **F3** to remove with confirmation, **F4** to add, and **Esc** to close. Verification updates the list in place; setup and removal return to the conversation. > > Connections without a result are checked automatically in the background, up to three at a time. Last-known results and their **Last checked** timestamps are remembered for this REPL session only; reopening reuses them until you press Enter to recheck. Connections without a verifier show **Configured**, not a successful health check; setup and removal remain available. Use `/integrations verify` to check all connections again. > **/mcp** > > MCP-capable integrations only (subset of the full integration catalog). > > `/mcp list` opens the same searchable browser and automatically checks connections without a result in the background. It shows **Checking…** until results arrive, or this session’s last check result with its timestamp. Click a row or use **↑↓/Tab** to browse, **Enter** to check again, **F2** to reconfigure, **F3** to disconnect with confirmation, **F4** to connect, and **Esc** to close. Explicit subcommands remain available below. > > ```text theme={null} > /mcp list > /mcp connect github > /mcp disconnect github > ``` > > | Subcommand | Maps to | > | - | - | > | `list` | Browse configured MCP servers in a TTY; render a verified table otherwise | > | `connect <server>` | `opensre integrations setup <server>` | > | `disconnect <server>` | `opensre integrations remove <server>` | > **/model** > > Show the active hosted model. An authenticated interactive shell is account-managed, so local provider/model changes are locked until sign-out; signing out closes the shell. > > ```text theme={null} > /model show > /model set openai gpt-4.1 > /model set anthropic claude-sonnet-4-20250514 --toolcall-model claude-sonnet-4-20250514 > /model set claude-sonnet-4-20250514 > /model restore > /model restore anthropic > /model toolcall set gpt-4.1-mini > /model > ``` > > | Subcommand | Behavior | > | - | - | > | `show` (default) | Show provider and reasoning/toolcall model slots | > | `set <provider> [model] [--toolcall-model <m>]` | Refuse the local change while the account route is active | > | `set <model>` (no provider) | Refuse the local change while the account route is active | > | `restore [provider]` | Refuse the local change while the account route is active | > | `toolcall set <model>` | Refuse the local change while the account route is active | > > **Account lock:** `set`, `restore`, and `toolcall set` explain that the model is managed by the OpenSRE account and make no change. The account route overrides local provider variables for every model role while the session is signed in. > > **Bare `/model`:** in a terminal this opens a menu rather than printing the table. The menu stays open: `show` prints the current models and hands you back to it, so leave with **done** or Esc. `set`, `restore` and `toolcall` close it once the change succeeds. Use `/model show` when you only want the table. > > **Reasoning vs toolcall model:** Reasoning model drives chat and planning; toolcall model drives tool invocation when the provider exposes a separate slot (common on Anthropic/OpenAI). > > See [LLM providers](/docs/platform/llm-providers). > **/tools** > > Browse registered tools with a short purpose beside each name. The list grows > with your terminal height, showing up to 20 tools. A compact preview below the > list shows the selected tool's name and description. > > ```text theme={null} > /tools > ``` > > In an interactive terminal, use **↑/↓** to browse and **Enter** to toggle > chat/action surfaces, parameters, and the full description. Expanded details > use the available height below the list. If they still overflow, **←/→** changes > pages. **Esc** closes the browser and restores the conversation. Moving to > another tool returns to its compact preview. Narrow terminals omit the purpose column; > the selected tool's description remains visible below the list. > > The browser uses a temporary terminal screen, so browsing and resizing do not > add tool listings to your conversation's scrollback. > > Piped and noninteractive output stays a readable table without keyboard input. > > There is no global `/list` command — use domain-specific list commands (see [Natural language actions](#natural-language-actions)). > > *** ### Privacy and history Command **history** (up-arrow recall) is separate from **session** history (`/sessions`). Full redaction patterns and env vars: [Interactive Shell Privacy](/docs/platform/interactive-shell-privacy). > **/history** > > ```text theme={null} > /history > /history clear > /history off > /history on > /history retention 1000 > ``` > > | Subcommand | Behavior | > | - | - | > | (none) | Numbered list of persisted prompt lines | > | `clear` | Delete `~/.opensre/interactive_history`; up-arrow recall empty on next launch | > | `off` | Pause disk writes for this session (in-memory recall still works) | > | `on` | Resume persistence | > | `retention <N>` | Cap file entries; prunes immediately (requires redacting backend) | > > Bare `/history` opens an interactive menu in a TTY (presets: 100, 500, 1000, 5000 for retention). > > > [!WARNING] > > > > Redaction applies to the **history file**, not necessarily to what is sent to the LLM. Treat shared machines accordingly — run `/history clear` after sensitive sessions. > **/privacy** > > ```text theme={null} > /privacy > ``` > > Shows persistence on/off, redaction on/off, retention cap, history file path, built-in pattern count, and a short threat-model reminder (unencrypted local disk). > > *** ### Tasks and background Long-running work is tracked in a per-session task registry surfaced by `/tasks`. Human todos and reminders are tracked separately by `/work`. > **Task kinds** > > | Kind | Source | > | - | - | > | `cli_command` | Delegated CLI subprocesses | > | `code_agent` | Code-agent integrations (when used) | > **Task statuses** > > | Status | Meaning | > | - | - | > | `running` | In progress; `/cancel` eligible | > | `completed` | Finished successfully | > | `cancelled` | User or `/cancel` stopped it | > | `failed` | Non-zero exit or pipeline error | > | `pending` | Created but not yet started | > **/gateway** > > Control the background OpenSRE gateway daemon, which runs the web health app, Telegram chat, and the task scheduler. Scheduled work needs a scheduler running: this daemon, or `opensre cron start` in the foreground. > > ```text theme={null} > /gateway > /gateway start > /gateway stop > /gateway status > /gateway logs > /gateway logs 100 > ``` > > | Subcommand | Behavior | > | - | - | > | `status` (default) | Running state and pid, plus a line per component | > | `start` | Start the daemon in the background | > | `stop` | Stop the running daemon | > | `logs [lines]` | Tail the log, 30 lines by default | > > Logs are written to `~/.opensre/gateway/gateway.log`. Same daemon as `opensre gateway`. > **/tasks** > > ```text theme={null} > /tasks > ``` > > Example: > > ```text theme={null} > Tasks > > id kind status started (UTC) duration detail > def34 cli_command completed 2026-06-17 09:55 120.0s opensre health > ``` > > Shows up to **50** recent tasks, newest first. Use the **id** column (short prefix) with `/cancel`. > **/work** > > ```text theme={null} > /work > /work add <title> > /work edit > /work edit <id-or-title> --priority high --due 2026-10-20 > /work done <id-or-title> > /work next > /work path > ``` > > Interactive record lists (`/work list`, `/work next`, `/loops list`, `/cron list`, > `/memory list`, and `/tasks`, including their list aliases) share read-only search. > Type a name, ID, or displayed detail to filter; use arrow keys or Tab to move > through matches. Enter never changes a record. Esc closes search and leaves the > full listing in your terminal. Command filters still apply. CLI, piped, and > agent-requested listings remain plain output; logs and reports are unchanged. > > In an interactive terminal, `/work add` without a title asks for one, then shows > selectable detail rows with Priority selected. Use arrow keys or Tab/Shift-Tab to > move between rows, then Enter to edit a row or choose **Create work item**. > Ctrl-S creates the item from the summary. Priority is a selection; project supports existing names or a new > entry; due date supports Today, Tomorrow, or an ISO date/datetime. Optional fields > remain unset unless supplied, and the existing Normal priority is shown explicitly. > Esc returns from an edit without applying it; Esc on the summary cancels creation. > Fully specified commands still execute directly. > > Use `/work done` or `/work complete` without a target to choose an unfinished work > item in an interactive terminal. Type to search, use arrow keys to choose, and > Enter to mark it complete. You can still pass one or more IDs directly. Outside > the interactive terminal, a completion target is required. > > Use `/work edit` to choose an existing item, or `/work edit ` to open > its prefilled title, priority, project, owner, and due date. Choose **Save changes** > after reviewing. An explicit field command executes directly; use a quoted empty > value such as `--owner ""` or `--due ""` to clear an optional field. > Missing-target edit options such as `/work edit --priority high` are kept for > review after you choose an item. > > `/work add` creates durable human todos; it does not schedule reminder delivery. > For reminders, use the destination-aware `opensre work add` command with > `--remind-at` and `--target provider:chat-id`, or ask from a connected chat whose > destination OpenSRE can reuse. These work items are separate from `/tasks` > (runtime jobs). See [Work Management](/docs/platform/work-management). > > You can also ask naturally: > > ```text theme={null} > Add task "finish Slack reminder smoke test" for the hackathon, priority high. > What should I focus on next? > ``` > **/cancel** > > ```text theme={null} > /cancel abc12 > ``` > > * Matches task id by **prefix** (must be unique among active/recent tasks) > * Only **running** tasks accept cancellation > **/stop** > > Prints guidance only — does not kill processes: > > ```text theme={null} > /stop > ``` > > | Situation | Action | > | - | - | > | Streaming turn | **Ctrl+C** | > | Background CLI task | `/tasks` → `/cancel <id>` | ### Alerts and selections > **/choose** > > Reopen the selection menu the agent is waiting on. Useful when the menu scrolled away, or you dismissed it with Esc and want it back. > > ```text theme={null} > /choose > ``` > > Prints `No selection menu is pending.` when the agent has not asked anything. Outside a TTY it lists the options as text for you to answer in a reply instead of opening a picker — except for a pending onboarding demo choice, which it drops with "Guided demo selection is unavailable here", leaving you to ask for what you want directly. > **/alerts** > > ```text theme={null} > /alerts > ``` > > When the alert listener is active: > > | Field | Meaning | > | - | - | > | `status` | `listening` | > | `queue depth` | Alerts waiting for REPL consumption | > | `dropped` | Alerts dropped when queue was full | > | `recent` | Up to 5 latest alert names/snippets | > > If the listener is inactive, prints a warning. Incoming alerts also appear in `/status` as `incoming alerts`. Post alerts to your configured webhook/listener URL during setup; the REPL triages them in chat. > > *** ### Setup and ops (CLI parity) These spawn `opensre …` subprocesses. Output is captured into the REPL buffer for non-interactive subcommands; wizards attach to the real TTY. > **/account** > > ```text theme={null} > /account > /account login > /account status > /account credits > /account usage > /account logout > ``` > > Sign in to a personal OpenSRE account, inspect the local login, show remaining hosted credits (`/account credits` or `/credits`, same as `opensre credits`), open the credits and top-up page (`/account usage`, opens your browser; from a chat it replies with the link), or sign out. Same as `opensre account`. The browser opens the OpenSRE sign-up page, where you can use email or any enabled sign-in provider. The shell starts only for an `active` account state. `/logout` (same as `/account logout`) revokes the saved session, invalidates the hosted-model route, and closes the shell before another turn. If `OPENSRE_ACCOUNT_TOKEN` is set in the environment, it overrides the token saved by login until you unset it. > **/credits** > > ```text theme={null} > /credits > /credits --dev > ``` > > Show remaining OpenSRE hosted credits for the signed-in account. Same as `opensre credits` and `/account credits`. Requires an active login. A fetch or auth failure is reported as unavailable — it is never shown as a zero balance. > **/setup** > > ```text theme={null} > /setup > /setup --dev > ``` > > Run the first-use webapp account flow. It creates or signs in to an OpenSRE account using email or another enabled provider and activates the account-hosted model. Use `--dev` for a webapp running at `http://localhost:3000`. Setup requires exclusive stdin because browser authorization prompts on the real TTY. > **/auth** > > ```text theme={null} > /auth > /auth status > /auth login deepseek > /auth logout deepseek > /auth verify anthropic > ``` > > Log in to LLM providers, show local auth status, or clear stored credentials. Same as `opensre auth`. Interactive login prompts on the real TTY. > **/login** > > ```text theme={null} > /login > /login openai > /login anthropic > /login deepseek > ``` > > Shortcut for `/auth login`. Same provider flows as `opensre auth login`. This signs in an LLM provider. Sign in to an OpenSRE account with `/account login`, and sign out of that account with `/logout`. > **/logout** > > ```text theme={null} > /logout > ``` > > Sign out of the OpenSRE account and close the shell. Same as `/account logout`: it revokes the saved session and drops the hosted-model route before another turn. It does not clear an LLM provider credential — use `/auth logout ` for that. > **/onboard** > > ```text theme={null} > /onboard > /onboard local_llm > ``` > > Optional local LLM-provider onboarding. The account-hosted route remains authoritative while signed in, so this does not switch the model used by the interactive shell. > **/remote** > > ```text theme={null} > /remote health > /remote ops > /remote pull > /remote trigger > ``` > > Operate remote deployed OpenSRE agents (EC2, Nitro, hosted runtime). > **/remote-sync** > > ```text theme={null} > /remote-sync > /remote-sync status > /remote-sync sync > /remote-sync sync --dry-run > /remote-sync setup --provider aws --bucket my-opensre-bucket > ``` > > Copy sessions and memory to an object store you own (S3, GCS, Azure, Vercel Blob, or S3-compatible). Credentials and API keys stay local. See [Sync to your own object store](/docs/install/remote-sync). > **/config** > > ```text theme={null} > /config show > /config set interactive.history.max_entries 1000 > ``` > > Read/write `~/.opensre/config.yml`. Prefer env vars for secrets; use config for structured interactive-shell settings. > **/cron** > > ```text theme={null} > /cron > /cron list > /cron status > /cron start [--service] > /cron add > /cron remove <id> > /cron run <id> > /cron logs <id> > ``` > > Scheduled report delivery. Press Enter or Tab after `/cron` to choose a subcommand. When `/cron add` is missing required values, the form opens the first missing input and preserves supplied options. Review the task, schedule and delivery in compact rows; use **More options** for settings such as timezone, name and lookback. Required values and defaults appear while editing. Choose **Create schedule** or press **Ctrl-S** from the main review to save. **Esc** returns from an editor or More options; from the main review it cancels without saving. Creating a schedule does not run it immediately. Complete commands and non-interactive calls keep their usual behavior. Choices needing task IDs still return to the composer for editing. `/cron list` displays all tasks with their IDs and timezone context; `/cron logs ` shows execution history. In the interactive shell, `/cron run ` waits up to five seconds before returning control while the task continues in the background; read its outcome with `/cron logs `. `/cron start` runs the scheduler in the foreground until interrupted. See [Cron](/docs/platform/cron). > **/loops** > > ```text theme={null} > /loops > /loops active > /loops all > /loops show > /loops show <name-or-id> > /loops show <name-or-id> --run <run-id> > /loops add --name "Morning ops" --time 08:30 --prompt "Check open incidents and summarize production risk" --run-now > /loops add --name "CI fix agent" --cron "*/30 * * * * *" --mode agent --prompt "Repair failing PR checks with fix_github_pr_ci" > /loops edit > /loops edit <name-or-id> --time 09:30 --weekdays > /loops run <loop_id> > /loops stop <loop_id> > /loops start <loop_id> > /loops delete <loop_id> > /loops next <loop_id> > /loops messages > ``` > > List recurring loops from the scheduler store, create manual prompt loops, run a loop once immediately, stop or restart a loop, delete a loop, and debug the next fire time. By default a loop tick only produces a report; add `--mode agent` when the tick should act with tools (for example edit and push CI fixes). Onboarding creates draft examples such as a weekday morning report; active loops run through the cron scheduler when the gateway or `opensre cron start` is running. > > Agent-mode prompts must contain the complete task and name the tools to use; each tick starts a fresh session without loading workflow skills. For the CLI equivalent, use `opensre cron add --kind manual_loop --mode agent` with your prompt and schedule. Existing loops retain their stored mode; check `/loops show ` before reusing one. > > The list shows each loop's latest finding and run status, plus its next scheduled time in your local timezone. Failed and quiet runs replace the previous result in the table. > > In an interactive terminal, omit the ID from `/loops run`, `/loops start`, > `/loops stop`, `/loops delete`, or `/loops next` to choose a loop from a searchable > list. Start offers stopped and draft loops; stop offers active loops. Running a > loop requires the **Run now** action after reviewing its configuration. Deleting > requires **Delete permanently** confirmation. Esc cancels selection; Esc during > action review returns to the list. Outside the interactive terminal, supply an ID. > > Use `/loops show` to pick a loop, or pass its name or ID directly. Details contain the full report, recent runs, and configuration. Open an earlier report with the run ID shown in its history. New reports are retained even when delivery fails; older reports may be unavailable. > > Use `/loops edit` to pick a recurring loop, or pass its name or ID to review the > current settings. Edit its name and schedule, with timezone and lookback under > **More options**. Ordinary prompt loops also expose their prompt and Report/Agent > mode; managed instructions remain tied to their template, report, or skill. > Choose **Save changes** to update the existing loop. Editing does not run it, > change its delivery destinations, or enable a stopped loop. Complete commands > with explicit edit options also work without an interactive terminal. > If you omit the target but supply options, such as `/loops edit --time 09:30`, > those values are kept for review after you choose a loop. > > You can also ask naturally: > > ```text theme={null} > Set up a manual loop called Morning ops at 08:30 UTC to check open incidents and summarize production risk, and run it once now. > ``` > > By default, new loops send to every configured default handle OpenSRE can reach: Telegram, Slack, and the local interactive-shell inbox. > **/sentry** > > ```text theme={null} > /sentry digest run > /sentry digest schedule list > /sentry digest schedule add --cron "0 8 * * *" --provider telegram --chat-id <id> > /sentry digest schedule run <id> > /sentry digest schedule remove <id> > /sentry uptime check > /sentry uptime watch list > /sentry uptime watch add --cron "*/5 * * * *" --provider telegram --chat-id <id> > /sentry uptime watch run <id> > /sentry uptime watch remove <id> > ``` > > Automated Sentry morning digest delivery (#10) and uptime watch notifications (#4032). > Requires Sentry plus Telegram or Slack configured for the chosen provider. See [Sentry](/docs/integrations/incidents/sentry). > **/messaging** > > ```text theme={null} > /messaging pair > /messaging allow > /messaging revoke > /messaging status > ``` > > Telegram bot pairing and sender allowlist for two-way chat and delivery features. > **/guardrails** > > ```text theme={null} > /guardrails audit > /guardrails init > /guardrails rules > /guardrails test > ``` > > | Subcommand | Purpose | > | - | - | > | `init` | Scaffold local guardrail config | > | `rules` | List active masking rules | > | `test` | Run sample text through rules | > | `audit` | Scan recent content for sensitive patterns | > > Related: [Masking](/docs/platform/masking). > **/update** > > ```text theme={null} > /update > ``` > > Checks PyPI and upgrades OpenSRE in-place (5-minute network timeout). Non-zero exit prints CLI error code. > **/posthog** > > Run or schedule automated PostHog per-metric summary reports. Requires the PostHog integration; see [PostHog](/docs/integrations/monitoring/posthog). > > ```text theme={null} > /posthog report run > /posthog report schedule list > /posthog report schedule add > /posthog report schedule run > /posthog report schedule remove > ``` > > A scheduled report only fires while a scheduler is running, so pair it with `/gateway start` or `opensre cron start`. Same as `opensre posthog`. > **/runbooks** > > Manage the trusted runbook sources the agent may read during guided investigations. See [Runbook-guided investigations](/docs/integrations/incidents/runbook-guided-investigations). > > ```text theme={null} > /runbooks list > /runbooks add github --name --repo > /runbooks verify > /runbooks remove > ``` > > The agent reads runbooks only from sources listed here; an untrusted repository, an unsafe path, or a non-Markdown document is rejected. Same as `opensre runbooks`. > **/debug** > > ```text theme={null} > /debug sentry > ``` > > Targeted smoke tests (Sentry, etc.) — subcommands match `opensre debug --help`. > **/uninstall** > > ```text theme={null} > /uninstall > ``` > > Removes OpenSRE and local data. **Destructive** — confirmation required unless trust mode. Delegates to interactive CLI uninstall flow. > > *** ### System > **/doctor** > > Environment diagnostic distinct from `/health`: > > ```text theme={null} > /doctor > ``` > > Checks Python version, venv, config paths, Docker availability, credential presence, and related local prerequisites. Each row: `ok`, `warn`, or `error` with detail text. > > Use **`/doctor`** before first install debugging; use **`/health`** before an incident to verify integrations. > **/theme** > > Choose the shell colour theme. The choice persists to `interactive.theme` in `~/.opensre/config.yml`, so it survives restarts. > > ```text theme={null} > /theme > /theme nord > ``` > > Bare `/theme` opens a picker with the current theme preselected, and needs a TTY. Passing a name sets it directly and reports an unknown name with the full list. > > Installed themes: `green`, `blue`, `amber`, `mono`, `red`, `pink`, `purple`, `orange`, `teal`, `lime`, `nord`, `dracula`, `solarized`, `gruvbox`, `webflux`, `sunset`. The default is `purple`. > **/version** > > ```text theme={null} > /version > ``` > > | Field | Example | > | - | - | > | `opensre` | Package version from install | > | `python` | `3.12.x` | > | `os` | `darwin (arm64)` | > **/exit and /quit** > > ```text theme={null} > /exit > /quit > ``` > > Clean shutdown with a `/resume` hint. Prefer this over killing the terminal so session files flush cleanly. > > `/exit`, `/quit`, double **Ctrl+C**, or **Ctrl+D** (empty prompt) all print something like: > > ```text theme={null} > Resume this session with: > /resume 3f8a1c2d > opensre --resume 3f8a1c2d > goodbye. > ``` ## Trust mode, `/auto`, and confirmations Two controls decide whether the REPL prompts before an action: | Control | Role | | - | - | | `/auto` | Which **agent tool types** need approval (default **high** = none) | | `/trust` | Session override that **skips** approval prompts when `/auto` (or an elevated slash command) would ask | | Tier | Examples | Confirmation | | - | - | - | | Exempt | `/help`, `/exit`, `/trust`, `/auto` | Never prompts | | Safe | `/status`, `/health`, `/integrations list` | Read-only or standard risk | | Elevated | `/cancel`, `/integrations remove`, `/uninstall` | Prompt unless trust on | Non-TTY: actions that would ask **fail closed** (error message, no side effect). ```text theme={null} /auto med # ask before mutating agent tools /auto high # alpha default — no tool prompts /trust on # skip prompts for this session /trust off # restore /auto (and elevated) prompts ``` *** ## Natural language actions You do not need to memorize every slash command. Examples: | You type | Typical action | | - | - | | "what's my token usage?" | `/cost` | | "verify datadog" | `/verify datadog` | | "show datadog config" | `/integrations show datadog` | | "show the active model" | `/model show` | | "list my past sessions" | `/sessions` | | "check health then list integrations" | `/health` then `/integrations list` | | "how do I configure Datadog?" | Doc-grounded answer (no mutating command) | **List intents** — there is no global `/list`: | Intent | Command | | - | - | | Connected integrations | `/integrations list` | | Tools | `/tools` | | Background tasks | `/tasks` | | Human todos and reminders | `/work` | | MCP servers | `/mcp list` | | Cron jobs | `/loops` or `/cron list` | | Past REPL sessions | `/sessions` | When the planner is uncertain, it asks for clarification or falls back to help — it does not silently run elevated commands. Compound requests ("health then integrations") execute as an ordered sequence of slash actions. *** ## Common workflows ### First-time setup ```text theme={null} /account status /health /verify /model show ``` Run `opensre` to sign in before this workflow is reachable. Every launch opens the demo picker; choose **E** or press Escape to continue to the prompt, or run `opensre --skip-onboarding` to bypass it. The recommended option analyzes CI/CD performance; see [CI/CD analytics demo](/docs/cicd/cicd-analytics-demo). ### Incident triage (local) ```text theme={null} /status /alerts /health ``` Paste alert text or describe the incident in plain language and the agent will triage it with your connected integrations. ### Pick up yesterday's thread ```text theme={null} /sessions /resume 9b2e4f7a ``` Continue chatting, then optionally: ```text theme={null} /new ``` ### Inspect account and model state ```text theme={null} /account status /model show /cost ``` ### Debug integration failures ```text theme={null} /integrations show datadog /doctor /verbose on ``` *** ## Troubleshooting | Symptom | Try | | - | - | | `unknown command` | `/help`; check spelling; use suggested correction | | Elevated command blocked in CI | Use the CLI equivalent or run interactively | | `/cost` shows zero | No LLM turns yet this session; chat once and retry | | `/resume` not found | Run `/sessions`; use longer ID prefix | | `/resume` ambiguous prefix | Add more characters from Session ID column | | `/model set` reports an account lock | The hosted model is working as designed; `/logout` closes the shell rather than switching it to a local provider | | `/integrations show` not found | Run `/integrations list` for exact service slug | | History pause not working | Requires redacting backend — see `/privacy` | | Garbage in next prompt after table | Known TTY issue — upgrade OpenSRE; report if persistent | *** ## Related docs * [Goals](/docs/getting-started/goals) — `/goal` set, pause, resume, edit, clear * [Session History](/docs/platform/sessions) — persistence format, privacy, `/new` vs `/clear` * [Interactive Shell Privacy](/docs/platform/interactive-shell-privacy) — redaction, env vars, threat model * [LLM providers](/docs/platform/llm-providers) — `/model` and `/effort` * [Cron](/docs/platform/cron) — `/cron` scheduled deliveries * [Sync to your own object store](/docs/install/remote-sync) — `/remote-sync` > **/demo** > > Open the demo picker that every launch shows: CI/CD analysis of a repository on this machine (recommended), the recurring CI/CD reliability agent, remote CI/CD repairs, or Slack setup. Menus need an interactive terminal. > > ```text theme={null} > /demo > ``` > > See [CI/CD with OpenSRE](/docs/cicd/overview). ## Scheduled CI repair demo Choose scheduled CI repair from `/demo` to try a tiny failure in a new private repository named `opensre-ci-repair-demo-` plus four letters or digits. OpenSRE creates it with a passing `main`, opens a pull request whose test fails, and schedules the repair of that pull request. A repository you name that is not an OpenSRE demo is left unchanged, and a new demo name is used instead. The repair has a ten-minute total limit. It uses the macOS or Linux background scheduler so it can finish after you close the terminal. Configure GitHub and authenticate your coding agent before starting. Creating the demo needs permission to create a private repository and workflow. * Read the complete outcome and evidence links with `/loops show `. * Stop an active repair with `/loops stop `. * The demo saves its evidence under `~/.opensre/demo-results/`. The repository, its pull request, and the stopped loop history remain; delete the repository on GitHub when you no longer need it. # Quickstart Source: https://opensre.com/docs/getting-started/quickstart ## Before you begin Make sure you have: * A terminal open in a code project * A Git repository (recommended for the full workflow demonstration) ## Step 1: Install and start opensre ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc ``` Then start OpenSRE: ```bash theme={null} opensre ``` ![OpenSRE CLI welcome screen with the sign-in prompt](https://mintcdn.com/tracer/N6OGfOUQjx4VJof5/images/opensre-welcome.png?fit=max\&auto=format\&n=N6OGfOUQjx4VJof5\&q=85\&s=a111e922dc38f18a550747d6974f2521) On Windows, open PowerShell and run: ```powershell theme={null} & ([scriptblock]::Create((irm https://install.opensre.com/install.ps1))) -dc ``` See [Windows](/docs/install/windows-local). Prebuilt Linux binaries require **glibc 2.35+** (Ubuntu 22.04+ or comparable). See [platform requirements](/docs/install/linux-local#binary-compatibility) before installing on another distribution. ## Step 2: Sign in Choose **Sign in or create account** in the welcome screen. The browser opens the OpenSRE sign-up page, where you can use email or another enabled provider. Signing in activates the OpenSRE-hosted model, so you need no LLM key. Back in the terminal, OpenSRE confirms the session with the web app and opens the shell. If the sign-in screen comes back instead, see [troubleshooting](#troubleshooting). ## Step 3: Pick what to do The first shell prompt asks what you want to do: ```text theme={null} What would you like to do? (A) Analyze & improve a repo (recommended) (B) Keep my CI/CD healthy automatically (C) Open the shell ``` Choosing **B** asks how that should run: ```text theme={null} Keep my CI/CD healthy automatically (A) Run continuously on this machine (recommended) (B) Run one repair in OpenSRE Cloud (C) Connect Slack ``` Choose **A** to see what OpenSRE does with a repository on your machine: it reads the last 30 days of GitHub Actions history and reports how reliable your CI is and how much developer time it costs. It needs GitHub access: connect it in the OpenSRE app through OAuth or its token-entry form. Run `opensre account login` to load that connection on this machine. If it is missing, the flow offers the app setup link. See [CI/CD analytics demo](/docs/cicd/cicd-analytics-demo). Choose **C** to go straight to the prompt. Type `/demo` to open the menu again at any time, or start with `opensre --skip-onboarding` to skip it. ## Step 4: Ask your first question Choose either path — both use the same local `opensre` binary: **Interactive shell** — start OpenSRE with no arguments and describe what's going on in plain English: ```bash theme={null} opensre ``` **One-shot headless run** — a single agent turn from your shell or a CI job: ```bash theme={null} opensre ask "why did checkout latency increase today?" ``` OpenSRE reasons across your connected systems, calling the tools it needs, and prints the answer. See [Headless CLI](/docs/guides/headless-cli) for tool approvals, `--json` output, and exit codes. Inside the interactive shell, `/help` lists commands, `/status` shows the current setup, `/effort` changes reasoning effort, and `/exit` closes the shell. ![Running an OpenSRE turn on a sample incident](https://mintcdn.com/tracer/p9QMaHUNKBbW3kp5/images/quickstart-investigate.gif?s=879ef2fc978da1f592e1d938d7ac6b4a) Connect the tools OpenSRE should read with `opensre integrations setup ` or `/integrations setup ` in the shell. Good first tasks, once GitHub is connected: ```text theme={null} analyze Tracer-Cloud/opensre CI/CD performance for the last 30 days how many PRs and branches currently have failing CI across all my orgs? fix failing checks on Tracer-Cloud/opensre#4597 ``` More in [CI/CD with OpenSRE](/docs/cicd/overview). ## Step 5: Keep up to date ```bash theme={null} opensre update ``` ## Uninstall To remove opensre and all its local data from your machine: ```bash theme={null} opensre uninstall ``` Pass `--yes` to skip the confirmation prompt: ```bash theme={null} opensre uninstall --yes ``` ## Troubleshooting * **`opensre` is not found**: Open a new terminal, or add the bin directory printed by the installer to `PATH`. * **The Linux binary does not start**: Prebuilt binaries require glibc 2.35 or newer; use a supported distribution or the [Docker guide](/docs/install/docker). * **`make` is missing**: See the [development setup guide](https://github.com/Tracer-Cloud/opensre/blob/main/SETUP.md) if you are working from source. * **The demo menu did not appear**: Menus need an interactive terminal. Run `opensre` in a terminal window rather than through a pipe or a CI job, then type `/demo`. * **The sign-in screen comes back after you signed in**: Run `opensre account status`; it names what is wrong with the session (expired, revoked, or the web app unreachable). # Showcase Source: https://opensre.com/docs/getting-started/showcase See what OpenSRE looks like in practice — install, chat, and share. Want to know what OpenSRE actually looks like when you run it? Here's a walkthrough of a first install, a first chat with the agent, and a way to put your own demos on the page. ## Watch a first run Install, sign in, and ask your first question. Most people are done in under ten minutes. ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc opensre ``` The first prompt offers a demo: the recommended one analyzes a repository on your machine and reports CI/CD reliability and the developer time lost to it. Skip it with **E** to go straight to the prompt. ![The CI/CD reliability report the recommended demo produces](https://mintcdn.com/tracer/pusIxCBwQ1b8M9V-/images/onboarding-ci-analysis-report.png?fit=max\&auto=format\&n=pusIxCBwQ1b8M9V-\&q=85\&s=9cd0e8970caeefc6f9900ba86f71b648) Ask: *Why is the payments-api pod in production crash-looping?* The agent calls your connected tools and answers in the shell. Prefer a step-by-step walkthrough with screenshots? Start at the [Quickstart](/docs/getting-started/quickstart). > **[Full quickstart](/docs/getting-started/quickstart)** > > Install, sign in, and ask your first question. > **[Install and start](/docs/getting-started/quickstart#step-1-install-and-start-opensre)** > > Start with the curl command and OpenSRE welcome screen. ## Show us what you built Got a demo, a runbook, or a post-mortem that used OpenSRE? 1. Post in [GitHub Discussions](https://github.com/Tracer-Cloud/opensre/discussions) with the **Showcase** tag — include the commands you ran, which tools were connected, and redacted output if you can. 2. Join the [bi-weekly giveaway](/docs/guides/bi-weekly-giveaway) when you share something — there's OpenSRE swag in the mix. # HTTP API Source: https://opensre.com/docs/guides/api Health checks and alert intake over HTTP. The OpenSRE backend serves one HTTP API (FastAPI, `gateway/web/webapp.py`). All routes live on a single port (default `8000`). To drive the agent in-process from your own code instead, see the [Python API](/docs/guides/python-api). In production, always call the API over HTTPS — deploy behind a TLS-terminating load balancer (the backend is Terraform-managed, separately from this repo). In production, terminate TLS at a load balancer or reverse proxy in front of the application. ## Authentication | Routes | Caller | Authentication | | - | - | - | | `POST /alerts` | Machines (alert sources, CI, schedulers) | Set `OPENSRE_ALERT_LISTENER_TOKEN` and send `Authorization: Bearer `. If unset, only loopback callers are accepted. | | `GET /`, `/health`, `/ok`, `/healthz`, `/readyz` | Health probes | None | ## Health | Route | Behavior | | - | - | | `GET /healthz` | Liveness. Always returns `{"status": "ok"}`. | | `GET /readyz` | Readiness after gateway startup. Returns `503` with `{"status": "not_ready"}` until ready. | | `GET /`, `/health`, `/ok` | LLM configuration check. Returns version and `llm_configured`. Status is `503` until an LLM is configured. | ```bash theme={null} curl https:///healthz curl https:///readyz curl https:///health ``` ## Alert intake Enqueue an alert for background handling. Returns `202`: ```bash theme={null} curl -X POST "https:///alerts" \ -H "Authorization: Bearer $OPENSRE_ALERT_LISTENER_TOKEN" \ -H "Content-Type: application/json" \ -d '{"text": "CPU above 90% on checkout-db", "source": "grafana"}' ``` Required field: `text`. Optional fields: `alert_name`, `severity`, `source`, `received_at`. Request bodies larger than 1 MiB return `413`. Queued alerts are picked up by the agent's alert inbox — the interactive shell surfaces them with `/alerts`, and chat gateways deliver them to the paired channel. ## Status codes | Code | Meaning | | - | - | | `202` | Accepted (`POST /alerts`) | | `400` | Malformed JSON or missing required fields | | `401` | Missing or invalid bearer token | | `403` | Non-loopback caller without `OPENSRE_ALERT_LISTENER_TOKEN` | | `413` | Alert body larger than 1 MiB | | `503` | LLM not configured (`/health`) or gateway not ready (`/readyz`) | # Community Giveaway Source: https://opensre.com/docs/guides/bi-weekly-giveaway Contribute to OpenSRE and become eligible for bi-weekly prizes. Every two weeks, the maintainer team selects three winners from contributors who were active in the current window. ![OpenSRE bi-weekly giveaway: eligibility, rules, and tiered prizes](https://mintcdn.com/tracer/PqagOwJqZl2Y6xHm/images/bi-weekly-giveaway.png?fit=max\&auto=format\&n=PqagOwJqZl2Y6xHm\&q=85\&s=966485637d78198f2a650e2e2686a701) > **What counts** > > * Merged pull requests and resolved issues > * Code reviews and documentation > * Consistent activity during the window > * Quality of contribution over volume alone > **Rules** > > * Contributions must fall within the active window > * Winners are selected by the maintainer team > * Both volume and quality are considered > * One win per person per month ## Prizes Each winner receives an Amazon gift card and an OpenSRE merch bundle. | Place | Reward | Value | | - | - | - | | 1st | Amazon gift card + merch bundle | \$50 | | 2nd | Amazon gift card + merch bundle | \$35 | | 3rd | Amazon gift card + merch bundle | \$25 | ## How to participate 1. Contribute during the current two-week window. Merged pull requests, resolved issues, code reviews, and documentation all count toward eligibility. 2. Maintain consistent activity through the window. Quality outweighs a large number of low-value changes. 3. Join [Discord](https://discord.gg/opensre) for winner announcements each cycle. If you are new to the repository, issues labeled `good first issue` or `help wanted` are a useful starting point. They are not required — any qualifying contribution in the window counts. | Resource | Link | | - | - | | Good first issues | [Browse on GitHub](https://github.com/Tracer-Cloud/opensre/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) | | Help wanted | [Browse on GitHub](https://github.com/Tracer-Cloud/opensre/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) | | Good First Issues guide | [Contributor guide](https://github.com/Tracer-Cloud/opensre/blob/main/docs/good-first-issues/README.md) | | Pull request review flow | [Greptile · human · CI](/docs/guides/pr-review-flow) | | Winner announcements | [OpenSRE Discord](https://discord.gg/opensre) | > [!NOTE] > > Winners are selected at maintainer discretion based on contribution quality and > activity in the active window. If you win, the team will contact you on Discord > to arrange your prize. # Deployment Source: https://opensre.com/docs/guides/deployment Run an organization-managed gateway or deploy the OpenSRE gateway yourself. Choose an OpenSRE Cloud managed gateway when your organization wants OpenSRE to operate the runtime, or deploy the repository `Dockerfile` when you want to own the process and storage. For local containers, see [Docker](/docs/install/docker). ## OpenSRE Cloud managed gateway An organization admin provisions the managed gateway from [Agent Backend settings](https://app.opensre.com/settings/agent-backend). The gateway belongs to the organization associated with the account used by the local shell; users never enter an organization or gateway ID. After signing in with `opensre`, ask the shell naturally: ```text theme={null} Check whether my organization's hosted gateway is running. Start my organization's hosted gateway. Stop my organization's hosted gateway. ``` The health check is read-only. Starting or stopping requires an explicit approval in the shell. Stopping preserves the gateway's configuration and credentials, but its remote loops and chat integrations do not run until it is started again. These requests control the organization-managed gateway. The `opensre gateway start|status|stop` commands below control a separate daemon on the current machine. ## Self-hosted container Use the repository `Dockerfile` with Railway, EC2, ECS, Vercel, or another container host. The image supports three values of `MODE`: | `MODE` | Runtime | | - | - | | `web` (default) | HTTP API — health checks and alert intake | | `gateway` | Web app, configured chat transports, and the scheduler | | `scheduler` | Dedicated cron/loop scheduler service (no gateway or web) | ## Environment setup 1. Deploy this repository using your host’s normal application workflow, or build the `Dockerfile` and set `MODE` to `web` or `gateway`. 2. Set `LLM_PROVIDER` to your model provider (`anthropic`, `openai`, `openrouter`, `trustedrouter`, `deepseek`, or `gemini`). 3. Set the corresponding API key: * `ANTHROPIC_API_KEY` for `anthropic` * `OPENAI_API_KEY` for `openai` * `OPENROUTER_API_KEY` for `openrouter` * `TRUSTEDROUTER_API_KEY` for `trustedrouter` * `DEEPSEEK_API_KEY` for `deepseek` * `GEMINI_API_KEY` for `gemini` 4. Add any integration or storage environment variables required by your deployment, then verify health with `GET /health` or `GET /ok`. Minimum configuration: ```bash theme={null} LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=... ``` Provider keys and optional model overrides are listed in `.env.example` and [Environment variables](/docs/install/environment-variables). Set `DATABASE_URL` when gateway records must be shared through Postgres. Slack Events API requires it unless you explicitly enable the single-replica local deduplication escape hatch. Scheduler state remains file-backed under `OPENSRE_HOME`. Organization-bound sessions use `OPENSRE_CONTEXT_ROOT` when it is set and otherwise fall back to `OPENSRE_HOME`; make both roots durable when they differ. See [HTTP API](/docs/guides/api) for the route reference. ## Local gateway Start a local HTTP server for health checks and alert intake (and optional chat transports): ```bash theme={null} uv run opensre gateway start # background daemon uv run opensre gateway start --foreground # attached to this terminal uv run opensre gateway stop uv run opensre gateway status ``` The web app listens on `0.0.0.0` using the `PORT` environment variable (default `8000`). ```bash theme={null} curl http://127.0.0.1:8000/ok ``` Example alert push: ```bash theme={null} curl -X POST http://127.0.0.1:8000/alerts \ -H "Content-Type: application/json" \ -d '{ "text": "Orders pipeline failed with timeout.", "alert_name": "etl-daily-orders-failure", "severity": "critical", "source": "dagster" }' ``` By default, `/alerts` accepts only loopback callers. To allow remote access, set `OPENSRE_ALERT_LISTENER_TOKEN` and send it as a bearer token: ```bash theme={null} curl -X POST http://127.0.0.1:8000/alerts \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENSRE_ALERT_LISTENER_TOKEN" \ -d '{"text":"test"}' ``` Complete route reference: [HTTP API](/docs/guides/api). ## Railway 1. Add Railway Postgres and set `DATABASE_URL` when you use Slack Events API or another layout that needs gateway records shared between replicas. 2. Add a shared `OPENSRE_HOME` volume when more than one process needs the same scheduler tasks. If `OPENSRE_CONTEXT_ROOT` is set, make that root durable and shared for organization-bound sessions too. 3. Optionally set `OPENSRE_DEPLOYMENT_METHOD=railway` for telemetry labeling. 4. Deploy the service through your Railway project. Redis is not used for gateway persistence. ## Horizontal scale-out Run **one** gateway task and leave `OPENSRE_SESSION_FILE_LOCK` off. More than one gateway task sharing a file-backed session store is not supported yet. Follow [#5474](https://github.com/Tracer-Cloud/opensre/issues/5474) and add a second task only after that lands. To serve more concurrent conversations today, raise concurrency on that single task (next section). After #5474, put every task on the same session store — a shared mount such as S3 Files or EFS — and set: ```bash theme={null} OPENSRE_SESSION_FILE_LOCK=1 ``` Slack Events API tasks must also share `DATABASE_URL`. Without it, a Slack retry that lands on another replica runs the same turn twice. Socket Mode, Telegram, and Discord do not use that store. Do not set `SLACK_GATEWAY_ALLOW_LOCAL_DEDUP=1` on more than one replica. Do not turn the lock on to make a multi-task fleet safe until then. Even after the fix, `fcntl.flock` is weak on some shared mounts (NFS, and EFS depending on configuration) — stay on a single task there, or use a mount that honors POSIX locks. ### Concurrency per task Each task has two caps, and the lower one wins. `OPENSRE_MAX_CONCURRENT_TURNS` limits every turn in the process (default 1 on SMALL). Slack, Telegram, and Discord each have their own pool (`SLACK_GATEWAY_MAX_CONCURRENT`, `TELEGRAM_GATEWAY_MAX_CONCURRENT`, `DISCORD_GATEWAY_MAX_CONCURRENT`), which also defaults to 1 on SMALL. Raising only the process cap leaves chat at one concurrent turn. Turns are I/O-bound — mostly waiting on the model — so a small task can run several; the ceiling is the task's memory, since each concurrent turn holds its context resident. Raise both without changing the task size, using the transport vars for the chats you run: ```bash theme={null} OPENSRE_MAX_CONCURRENT_TURNS=2 SLACK_GATEWAY_MAX_CONCURRENT=2 TELEGRAM_GATEWAY_MAX_CONCURRENT=2 DISCORD_GATEWAY_MAX_CONCURRENT=2 ``` Or set `OPENSRE_SIZE_PROFILE=MEDIUM` (2) or `LARGE` (4) to raise both defaults together. Remote prompts from the CLI and the web app need no separate setting: they run on as many workers as the process cap, and one conversation's prompts still run one at a time. Clones and coding agents (Codex, Claude Code) use far more memory than a turn that only waits on the model. However many turns run, at most `OPENSRE_MAX_CONCURRENT_HEAVY_WORK` (default 2) of these run at once; the rest wait up to five minutes, then the tool reports that the gateway is busy. On a task with little memory, set it to 1. Size the task from a real run before raising either cap. Every `gateway_turn_completed` and `gateway_turn_failed` event carries `container_memory_bytes` and `container_memory_peak_bytes`. These are the whole task's memory, child processes included, so compare the peak against the task's memory limit. ### Dedicated scheduler service Scheduled callbacks use a bounded worker pool. At most two distinct tasks run at once by default; set `OPENSRE_SCHEDULER_MAX_CONCURRENT_RUNS` to a positive integer to change that limit. A task never overlaps its own previous run. Runs waiting for a scheduler worker appear as `pending` in run history and resume after a scheduler restart. On restart, pending runs for paused tasks wait until the task is enabled again. Admitted runs wait until both a worker and the task are available. Cron ticks rejected because the same job is already queued or running are skipped. Invalid concurrency settings fall back to two workers. Agentic scheduled runs also take a permit from `OPENSRE_MAX_CONCURRENT_TURNS`, so the lower of the scheduler and process-wide limits bounds their effective concurrency; interactive traffic shares the process-wide limit. Recovery work uses the same scheduler worker pool. By default the gateway process also runs the cron/loop scheduler. With more than one gateway task that would fire every scheduled task once per task. Run the scheduler as its own single service instead: * Deploy one `MODE=scheduler` task. It runs `opensre cron start --service`, idling until tasks exist rather than exiting. * Set `OPENSRE_GATEWAY_HOST_SCHEDULER=0` on the gateway tasks so they stop hosting the scheduler in-process. Run exactly one scheduler service, never one per gateway task. The gateway image sets `CODING_AGENT_SANDBOX=host`: Fargate denies the user namespaces that the coding agent's own sandbox needs, so the task itself is the isolation boundary and the agent gets the whole process. Unset it (or set `agent`) only where the agent's sandbox can start, such as a VM or a container that allows user namespaces. With `host`, the agent can reach the task's filesystem and network, so treat the repair target as trusted code: approve repair loops only for repositories whose contributors you trust. Repairs run only on repositories an organization admin approved and only on branches inside that repository; pull requests from forks are refused before the agent starts. Set `ORGANIZATION_ID` on a single-organization deployment. Loops scheduled before organization stamping stay listed for that organization only when the deployment declares it. The scheduler service and every process that mutates tasks — the gateway, `opensre cron`, and `/loops` — must share the **same** task store: the same `OPENSRE_HOME` on a shared mount (S3 Files / EFS). The scheduler reconciles from that file, so a separate filesystem would leave it running a stale task set. ### Optional: conversation affinity Skip affinity until you run more than one gateway task (after #5474). Then it is a performance option: it keeps a conversation's warm agent in one task's memory instead of cold-rebuilding when a turn lands elsewhere. Affinity is a routing concern, not an app setting: * Behind an HTTP load balancer (Slack Events API), set the shared `DATABASE_URL` above, then enable consistent hashing on the conversation (Slack channel and thread) so a thread sticks to one task. Hashing does not replace event dedup. * With the pull transports (Slack Socket Mode, Telegram, Discord), each task connects independently and the provider spreads events across tasks, so there is no built-in affinity. Affinity would need an external dispatcher and is worth adding only when warm-agent reuse measurably helps. # FAQ Source: https://opensre.com/docs/guides/faq Common questions about OpenSRE. > **What is OpenSRE?** > > OpenSRE is an open-source framework for AI SRE agents that answer questions > about production systems. It uses your observability stack, cloud context, > and runbooks, and can deliver its findings to Slack, local files, or other > messaging channels. See the > [Integrations overview](/docs/integrations) for supported tools. > **How do I get started?** > > Install OpenSRE, sign in, then ask it something: > > ```bash theme={null} > curl -fsSL https://install.opensre.com | bash -s -- -dc > opensre > opensre ask "why did checkout latency increase today?" > ``` > > Full walkthrough: [Quickstart](/docs/getting-started/quickstart). > **Which deployment path should I use?** > > Most users start with the local CLI (`curl` install, then > `opensre` to sign in, plus `opensre integrations setup` for the > tools you want OpenSRE to query). For an organization-managed runtime, use > [Enterprise setup](/docs/install/install-enterprise) at > [app.opensre.com](https://app.opensre.com). Integrations connected in the > web app are also used by every signed-in shell and CLI, and take > precedence over local setup for the same service. > > Managed and self-hosted gateway options are covered in > [Deployment](/docs/guides/deployment). > **Can I self-host OpenSRE?** > > Yes. Deploy the repository Docker image in `web`, `gateway`, or `scheduler` > mode. Set `LLM_PROVIDER` and the > matching provider API key (for example `ANTHROPIC_API_KEY` when > `LLM_PROVIDER=anthropic`). See [Deployment](/docs/guides/deployment) and > [Environment variables](/docs/install/environment-variables). > **Which model providers are supported?** > > The signed-in interactive shell uses the account-hosted model. Other hosts > and account-independent CLI workflows support Anthropic, OpenAI, OpenRouter, > TrustedRouter, DeepSeek, Gemini, Bedrock, Azure OpenAI, and CLI-backed > providers through `LLM_PROVIDER`. Provider matrix and troubleshooting: > [LLM providers](/docs/platform/llm-providers). > **Does OpenSRE work with our existing tools?** > > In most cases, yes. OpenSRE integrates with observability platforms > (Datadog, Grafana, Sentry), cloud environments (AWS, Kubernetes), incident > tools (PagerDuty, incident.io), databases, and messaging (Slack, Telegram). > Browse the [Integrations overview](/docs/integrations) and run > `opensre integrations verify` after setup. > **What happens when I run opensre with no arguments?** > > Running `opensre` signs you in (once) and starts the interactive shell. The > first prompt offers a demo; pick the CI/CD analysis of one of your repositories > or skip it with **E**. Then describe issues in plain language. Slash commands > such as `/help`, `/status`, `/demo`, and `/verify datadog` are documented in > [Shell commands](/docs/getting-started/interactive-shell-commands). > **How are security and telemetry handled?** > > OpenSRE supports reversible [masking](/docs/platform/masking) before external LLM calls > and command-history redaction via > [Interactive shell privacy](/docs/platform/interactive-shell-privacy). Anonymous product > telemetry can be disabled with `OPENSRE_NO_TELEMETRY=1`. Report > vulnerabilities to `support@opensre.com`. # Headless CLI Source: https://opensre.com/docs/guides/headless-cli Run one OpenSRE agent turn non-interactively from a terminal, script, or CI job. `opensre ask` is OpenSRE’s headless CLI command. It runs one agent turn, prints the response, and exits instead of opening the interactive shell. Each run is saved as a resumable session unless you pass `--ephemeral`. It uses the hosted model of the signed-in account (`opensre account login`), or `LLM_PROVIDER` plus the provider API key on a machine that is not signed in, and any tools configured with `opensre integrations setup`. Without either it exits with status `1` before the agent runs and prints the command that fixes the setup, such as `opensre auth login `. ```bash theme={null} opensre ask "why did checkout latency increase today?" ``` ## Follow-ups and required input When the agent completes the request, a terminal run prints a session ID and a command you can use for a later follow-up: ```text theme={null} Session: 489e2ba8-3b7f-4fa2-bcdf-55a93177337c Continue: opensre ask --resume 489e2ba8-3b7f-4fa2-bcdf-55a93177337c "" ``` Use the full ID or an unambiguous prefix: ```bash theme={null} opensre ask --resume 489e2ba8 "now check the last deployment" ``` If the agent cannot continue without a decision, it exits with status `needs_input` and prints numbered options. Resume the same session with the option number, or a custom answer when the question allows one: ```bash theme={null} opensre ask --resume 489e2ba8 "1" ``` For a batch of questions, pass a JSON object keyed by the displayed labels: ```bash theme={null} opensre ask --resume 489e2ba8 '{"Repository":"1","Window":"2"}' ``` Use `--ephemeral` for scripts or one-off requests that should not write resumable session history or print a session ID: ```bash theme={null} opensre ask --ephemeral "summarize this public status page" ``` [Prompt and response logging](/docs/platform/interactive-shell-privacy#prompt-and-response-logging) has separate controls. Set `OPENSRE_PROMPT_LOG_DISABLED=1` to disable it, including for ephemeral requests. ## Live activity When both output streams are attached to a terminal, `ask` starts with a `Thinking…` spinner with elapsed time. It switches to the active diagnostic tool (or a neutral tool count for a batch) when tools are invoked. The final answer remains on standard output without echoing raw tool transcripts. Piped and `--json` runs do not emit progress, so their output remains machine-readable. Pass `-` as the prompt to read all of standard input: ```bash theme={null} cat incident-notes.txt | opensre ask - ``` ## File context Attach an alert, log excerpt, deployment note, or other regular UTF-8 text file with `-i` or `--context-file` while keeping the positional prompt as the instruction: ```bash theme={null} opensre ask -i alert.json \ "investigate this alert and explain the likely cause" ``` Repeat the option to correlate several files: ```bash theme={null} opensre ask \ -i alert.json \ -i deployment.md \ "correlate the alert with the recent deployment" ``` ### Worked example Given `alert.json`: ```json theme={null} { "alert": "CheckoutLatencyHigh", "service": "checkout-api", "started_at": "2026-09-22T09:15:00Z", "p95_latency_ms": 2410, "baseline_p95_latency_ms": 420, "error_rate_percent": 8.2 } ``` And `deployment.md`: ```markdown theme={null} # Recent deployment - Service: checkout-api - Version: 2026.09.22-rc3 - Completed: 2026-09-22T09:12:00Z - Change: Enabled the payment-routing client with a 3-second upstream timeout. ``` Run: ```bash theme={null} opensre ask \ -i alert.json \ -i deployment.md \ "Correlate the alert with the recent deployment" ``` Response wording varies by model, but the result should use facts from both files. For example: ```text theme={null} The deployment is a strong likely trigger for the alert: - checkout-api rc3 completed at 09:12 UTC; the alert began three minutes later. - P95 latency rose from 420 ms to 2,410 ms, about 5.7x baseline, with an 8.2% error rate. - The observed latency is consistent with requests waiting near the new 3-second upstream timeout. Next action: disable the new client or roll back rc3, then verify whether latency and errors return to baseline. ``` Attach up to 16 files. Files are treated as untrusted context, not operator instructions. Each file may contain up to 64 KiB, with a combined limit of 128 KiB. Empty, binary, and non-UTF-8 files are rejected before the agent starts. The combined limit also applies after JSON encoding, so content dominated by control characters, quotes, or backslashes may reach it earlier. Review attachments for secrets before running the command. Their contents are sent to the configured LLM provider as part of the prompt. When a resumed session is waiting for a numbered choice, submit that answer without context files. Attach files in the following turn instead. ## Tool approvals Read-only tools run automatically. Tools that mutate state, contact an external service, explicitly require approval, or do not declare their side effects are denied by default. Authorize only the tools needed for this invocation by repeating `--allowed-tool`: ```bash theme={null} opensre ask "inspect the repository and run its focused tests" \ --allowed-tool shell_run \ --allowed-tool github_cli ``` An unknown tool name is rejected before the agent starts. The authorization is not saved and applies only to that process. The root `--yes` (`-y`) option does not authorize agent tools. `--dangerously-bypass-approvals` authorizes every approval-gated tool for that invocation. Use it only in a trusted environment where the prompt and connected integrations are controlled: ```bash theme={null} opensre ask "perform the requested maintenance" --dangerously-bypass-approvals ``` Do not combine the bypass flag with `--allowed-tool`. Neither option bypasses the operating-system permissions or sandboxing that applies to OpenSRE. Commands authorized through `shell_run` use the host environment's shell. See [Shell command syntax](/docs/getting-started/interactive-shell-commands#shell-command-syntax) for native Windows `cmd.exe`, PowerShell, WSL, and working-directory behavior. ## JSON output and exit codes Put the global `--json` option before `ask` for machine-readable output: ```bash theme={null} opensre --json ask "summarize current service health" ``` The command writes one JSON object with `status`, `response`, `session_id`, `questions`, `denied_tools`, and `error`. Each question includes its title, label, options, whether it accepts multiple selections, and whether a custom answer is allowed. The `error` value is either `null` or an object with `message` and `suggestion`. For required input, read `session_id` and `questions` from that object, then pass the answer to another JSON invocation of the same session: ```bash theme={null} opensre --json ask "investigate the failing CI" opensre --json ask --resume 489e2ba8-3b7f-4fa2-bcdf-55a93177337c "1" ``` The first result can be `needs_input` (exit code `4`) with a question like: ```json theme={null} {"status":"needs_input","response":"Which repository?\n 1. Tracer-Cloud/opensre\n 2. Another repository","denied_tools":[],"session_id":"489e2ba8-3b7f-4fa2-bcdf-55a93177337c","questions":[{"label":"Repository","title":"Which repository?","options":["Tracer-Cloud/opensre","Another repository"],"multi_select":false,"allow_custom":true}],"error":null} ``` After the answer, a successful result has `status: "success"`, the same `session_id`, and an empty `questions` array. JSON mode emits no terminal-only continuation footer; use the returned ID. `--json` is a global option, so it must appear before `ask` on both invocations. Without `--json`, a required question also prints `Session: ID` on stderr when output is redirected, so the choice can still be resumed. | Exit code | Meaning | | - | - | | `0` | The agent completed the request. | | `1` | Setup or agent execution failed. | | `2` | The command arguments were invalid. | | `3` | A tool required approval and was denied. | | `4` | The agent needs structured user input; resume the returned session. | | `130` | The invocation was interrupted with `SIGINT`. | | `143` | The invocation was terminated with `SIGTERM`. | # Hosting the Agent Source: https://opensre.com/docs/guides/hosting-the-agent Build the OpenSRE agent on your own ports and drive it per turn — the same way the gateway and the interactive shell do [Python API](/docs/guides/python-api) covers **embedding**: `AgentSession.start()` and `chat()`, with the standard ports. This page is for a **host** — a program that supplies its own output sink, tools, prompts, or error reporting, and drives one agent across many turns. The gateway (Slack, Discord, Telegram, and Buzz) and the interactive shell are both hosts, and they are the same program with different ports. ## Four roles, five API modules Everything a host or tool author touches comes through five modules — the harness API. Nothing else under `core.agent_harness` is part of it. | Role | You want to… | Import | | - | - | - | | **Embed** | run turns with the standard ports | `core.agent_harness` — `AgentSession`, `SessionConfig`, `SessionCore`, `SessionManager`, `TurnResult`, `ToolCallingTurnResult`, `OutputSink` | | **Host** | build one agent from your ports, handle it per message | `core.agent_harness.runtime` — `DefaultHeadlessBuild`, `InMemoryHeadlessBuild`, `HeadlessAgent`, `TurnBinding`, `DefaultToolProvider`, `ActionTurnRunner`; also the ReAct engine itself (`AgentConfig`, `build_agent`, the default LLM factories) | | **Host** | call helpers around a turn | `core.agent_harness.spi.` — `session_goal`, `session_state`, `cancel`, `accounting`, `prompt_chrome`, `integrations`, `grounding`, `defaults` | | **Write a tool** | implement an action tool | `core.agent_harness.tools` — `ActionToolScope`, `execute_with_action_context`, `capability_available_from_sources`, `HandoffField`, `EVIDENCE_KIND_VALUES` | | **Implement** | satisfy a protocol the harness calls | `core.agent_harness.ports` — `OutputSink`, `ToolProvider`, `PromptContextProvider`, `ReasoningClientProvider`, `RunRecordFactory`, `ErrorReporter`, `TurnAccounting`, `SessionState`, and the rebind mix-ins `ConsoleBindable` / `OutputBindable` / `SessionBindable` | ## The host loop Build once, then one call per inbound message: ```python theme={null} from core.agent_harness.runtime import DefaultHeadlessBuild, TurnBinding # 1. Build — the default port family for this session, plus your ports. agent = DefaultHeadlessBuild( session=session, # a SessionCore from AgentSession / SessionManager output=my_sink, # your OutputSink error_reporter=my_reporter, # optional; default logs at debug ).agent( tools=my_tool_provider, # optional; a configured DefaultToolProvider or your own prompts=my_prompts, # optional; names the prompt surface ) # 2. Handle — state the whole turn; the agent binds it, dispatches, and keeps # going while a session goal is attached. Nothing carries over from the # previous message. result = agent.handle( text, TurnBinding( session=session, # None keeps the current session tool_hooks=approval_hooks, # None means "no hooks this turn" console=turn_console, # for cooperative cancel and progress lines confirm_fn=ask_user, # None means "no confirmation callback" is_tty=False, ), accounting_factory=None, # message -> TurnAccounting; None = default per message cancel_requested=lambda: cancel.is_set(), on_progress=render_goal_progress, ) ``` `DefaultHeadlessBuild` holds what does not vary per host — reasoning client, run records, the default tool provider and prompt provider — and takes only the inputs those defaults need (`console`, `logger`, `surface`). Every port you pass to `agent()` replaces that default. `TurnBinding` is a value, not a set of keyword arguments: a field you omit is *this turn's value* (no hooks, no callback), never "leave alone", so a pooled agent cannot inherit another conversation's hooks by omission. `handle` is the only per-message call a host makes; `dispatch` (one engine turn) sits underneath it. The in-memory family is `InMemoryHeadlessBuild` — a script or test says `InMemoryHeadlessBuild().agent(tools=NullToolProvider())` and has an agent with zero configuration. There is one construction verb, `.agent(...)`; hosts do not call `HeadlessAgent(...)` directly. ## The same loop in the two shipped hosts The gateway keeps one agent per logical session and calls `handle` per inbound message: ```python theme={null} agent = DefaultHeadlessBuild(session=session, output=live_sink, console=console, logger=log, surface="gateway").agent( tools=DefaultToolProvider(session, console, tool_action_logger=log, observer_factory=…, subprocess_presenter_factory=…, slash_ports_factory=…), ) result = agent.handle( text, TurnBinding(session=session, tool_hooks=sink.tool_hooks, console=turn_console, is_tty=False), cancel_requested=cancel.is_set, on_progress=post_status_line, ) ``` The interactive shell shares the same `TurnRunner` → `SessionAgentPool` → `handle` path as the gateway (TTY accounting / `is_tty=True`). Direct construction for tests or a custom host looks like: ```python theme={null} agent = DefaultHeadlessBuild(session=session, output=ShellOutputSink(console), console=console, error_reporter=ShellErrorReporter()).agent( tools=shell_tool_provider(session, console, request_exit=…), prompts=DefaultPromptContextProvider(session, surface="interactive_shell"), ) result = agent.handle( text, TurnBinding(session=session, output=sink, tool_hooks=hooks, console=console, confirm_fn=ask, is_tty=tty), accounting_factory=lambda message: ShellTurnAccounting(session, message, recorder), cancel_requested=lambda: host_cancel_requested(sink), on_progress=print_checklist, ) ``` Neither host replaces a stage of the agent (action selection, answering) and neither writes its own turn loop. They differ only in ports and in what they do with the result. Production gateway/shell submissions go through the pool; expand `AgentBuildConfig` with `resolve_agent_ports`. ## What you implement vs what you call | You want to… | Implement | Call | | - | - | - | | Send output somewhere new (a websocket, a chat thread) | `OutputSink` — `print`, `stream`, `render_error`, `render_response_header` | `DefaultHeadlessBuild(output=…)` | | Add tools or change how they are rendered/confirmed | a `ToolProvider` — usually `DefaultToolProvider(...)` with your port factories | `.agent(tools=…)` | | Run turns under another surface's prompt (setup facts, menus) | nothing, or a `PromptContextProvider` whose `surface()` names it | `DefaultHeadlessBuild(surface=…)`, or `.agent(prompts=…)` | | Route swallowed exceptions (Sentry, logs) | `ErrorReporter` — `report(exc, context=…, expected=…)` | `DefaultHeadlessBuild(error_reporter=…)` | | Account a turn (telemetry, token totals) | `TurnAccounting` — `record_action_result`, `finalize` | `TurnBinding(accounting=…)` | | Approve or deny tool calls per conversation | — | `TurnBinding(tool_hooks=…)` | | Follow the turn's console or sink from your own port | `ConsoleBindable.bind_console` / `OutputBindable.bind_output` | happens on `bind_turn` | | Drive a multi-turn goal, read or set session state | — | `spi.session_goal`, `spi.session_state` | ## Rules a host must keep * **One agent per logical session; one turn at a time on it.** The agent enforces this: a second `handle` while one is running raises `AgentBusyError` rather than interleaving. Serialize turns *before* calling `handle` (the gateway holds a per-session lock across it); do not catch the error to retry. * **State the whole turn.** Build a fresh `TurnBinding` per message and pass it to `handle`; do not call `bind_turn`/`dispatch` yourself. * **Bind on the worker thread you dispatch from.** Storage scope is a `ContextVar`; if you hand the turn to a thread pool, wrap it with `config.scope_context.in_current_scope` on the calling thread. * **Do not import past the API.** Modules under `core.agent_harness.turns`, `.tools`, `.session` and `.prompts` are internal; the shipped hosts pin their remaining deep imports in shrink-only tests. # OTel payment auto-triage pilot Source: https://opensre.com/docs/guides/otel-payment-triage Run the payment demo, understand what OpenSRE watches, and receive failure and recovery reports. This opt-in pilot runs the official OpenTelemetry Demo payment service, one ClickStack Collector, and ClickHouse. OpenSRE checks the payment error rate, investigates a new incident, and reports its recovery. It does not change your application or require a dashboard. Existing vendor integrations are unchanged. ## How the pieces fit together ```text theme={null} Payment service (standard OpenTelemetry SDK) → Collector: receives telemetry through OTLP → ClickHouse: stores metrics, traces, and logs OpenSRE triage watch → reads recent payment metrics from ClickHouse every 30 seconds → on a new incident, reads a small set of related traces and logs → asks the configured API model to summarize that evidence → saves the report locally and sends it to Telegram, if enabled ``` **OpenSRE reads ClickHouse, not the Collector.** The Collector receives and forwards telemetry; it is not the history store. You configure these separately: | File | What you configure | | - | - | | `examples/telemetry-payment/collector.yaml` | How the Collector receives telemetry, derives request counts, and writes to ClickHouse. | | `examples/telemetry-payment/watch.yaml` | Where OpenSRE reads the stored data, which service to watch, and when to open or recover an incident. | Raw telemetry stays in ClickHouse. The local `watch.db` SQLite file stores incident state, reports, and pending deliveries—not the telemetry stream. The Collector's exporter creates and fills the telemetry tables; OpenSRE's read-only account does not create them or manage their schema. ## Start the demo Use a source checkout with Docker Compose and a configured OpenSRE API model. A tool-enabled model CLI is not supported for this bounded investigation. For Telegram, first run `uv run opensre integrations setup telegram` and choose your own chat or a group the bot can access—not another bot's account. Omit `--telegram` for terminal-only reports. From the repository root: ```bash theme={null} uv run python examples/telemetry-payment/setup.py init docker compose -f examples/telemetry-payment/compose.yaml up -d ``` `init` creates random demo credentials in an owner-readable `.env` and starts with the fault disabled. It refuses to overwrite an existing credential file. Keep the checkout in a directory shared with Docker Desktop. Port `18123` must be free. Only ClickHouse HTTP is published, bound to `127.0.0.1`; OTLP, payment, and flagd stay inside the Compose network. Load the demo reader credential in a Bash-compatible terminal, then start the watch: ```bash theme={null} set -a source examples/telemetry-payment/.env set +a uv run opensre triage watch \ --config examples/telemetry-payment/watch.yaml \ --state examples/telemetry-payment/watch.db \ --telegram ``` The foreground watch polls every 30 seconds. `--once` processes one completed window and exits. `--chat-id` optionally overrides the configured Telegram destination. Stop with Ctrl-C; retained reports and pending deliveries survive restart. Run the same command with the same state file to resume. Changing source, rule, bot identity, or destination requires a new state file. ## Understand the watch configuration The included `watch.yaml` is ready for this demo: ```yaml theme={null} source: backend: clickhouse host: 127.0.0.1 port: 18123 database: default username: opensre_reader watch: service: payment environment: demo # Explicitly map the collector's unsampled, delta SERVER-span counter. metric: name: traces.span.metrics.calls status_attribute: status.code error_value: STATUS_CODE_ERROR kind_attribute: span.kind server_value: SPAN_KIND_SERVER window_seconds: 30 ingestion_delay_seconds: 15 minimum_requests: 5 error_ratio: 0.2 healthy_windows: 2 ``` Read this as: **“Check payment in demo every 30 seconds. With at least five requests, open an incident if 20% or more fail. Recover only after two consecutive healthy windows.”** | Setting | Meaning | | - | - | | `source` | The ClickHouse connection, using a read-only account. Port `18123` is ClickHouse HTTP, not an OTLP port. | | `service`, `environment` | Limit all metric, trace, and log reads to this service and environment. | | `metric` | Identify the request counter, its error label, and the label selecting incoming server requests. Keep this mapping unchanged for the demo. | | `window_seconds` | How much data each check covers and how often the watch checks. | | `ingestion_delay_seconds` | Wait before checking a completed window so telemetry has time to arrive. Late data can still be missed. | | `minimum_requests` | Below this request count, the result is unknown—not healthy. | | `error_ratio` | The fraction of requests that must fail to open an incident: `0.2` means 20%. | | `healthy_windows` | How many consecutive below-threshold windows, each with enough requests, are needed to recover. | Keep passwords out of YAML. The setup command creates the demo password; loading `.env` above supplies `OPENSRE_TELEMETRY_CLICKHOUSE_PASSWORD` to OpenSRE. ### What happens on each check? For example, a check at **12:01:15** examines **12:00:30–12:01:00**: a completed 30-second window with a 15-second allowance for ingestion. | Result in that window | What OpenSRE does | | - | - | | 30 requests, no errors, no open incident | Prints `Telemetry: healthy`; no model call or Telegram report. | | 30 requests, 30 errors, no open incident | Opens one incident, reads up to ten span rows and ten logs, and produces an opening report. | | Errors stay at or above 20% during that incident | Keeps the incident open; does not repeat model analysis or create another opening report. | | Two consecutive windows below 20%, each with at least five requests | Marks the incident recovered and produces a recovery report, without another model call. | | Missing data, fewer than five requests, or a failed query | Prints `Telemetry: unknown`; does not declare recovery. An unknown window resets the recovery streak. | This watches **application telemetry, not Collector health**. An unknown result does not tell you whether the Collector stopped, the application stopped, or there was simply no traffic. Previously stored data may remain eligible briefly after collection stops; missing data is not detected instantly. Keep the watch terminal running: closing it pauses detection even if the Collector keeps storing data. Restarting resumes recent checks, but does not backfill missed windows. A slow investigation can also skip windows. ## Reproduce failure and recovery In another terminal: ```bash theme={null} uv run python examples/telemetry-payment/setup.py fail # Wait for an OPENED report (normally one or two completed metric windows). uv run python examples/telemetry-payment/setup.py recover # Wait for two consecutive healthy windows and a RECOVERED report. ``` If a Docker Desktop bind mount does not propagate the flag change, run `docker compose -f examples/telemetry-payment/compose.yaml restart flagd`. The traffic generator sends synthetic requests to the unmodified upstream payment service. The official `paymentFailure` flag rejects these requests; `emitRawPii` stays disabled. No real payment provider is contacted. View complete retained reports without contacting ClickHouse or Telegram: ```bash theme={null} uv run opensre triage reports --state examples/telemetry-payment/watch.db ``` Inside the REPL, use `/triage reports --state `. The continuous watch runs in a separate terminal, not inside the REPL. Telegram delivery retries the saved report, not the completed model analysis. Delivery is at-least-once: a crash after Telegram accepts a message but before OpenSRE saves its acknowledgment can produce a duplicate. Notifications carry an incident ID. Keep the local state file private; it contains investigation reports. ## Measurement and investigation limits The demo's application counter counts successful payments only. The Collector therefore derives a delta request counter from **unsampled SERVER spans**. `watch.yaml` explicitly maps that metric and its status/kind dimensions. Do not use this rule on sampled traces or substitute a cumulative counter: the reader rejects non-delta data. An application-native all-requests/error counter is a better production source when available, but requires an appropriate mapping. The model sees only the selected evidence from the incident's service, environment, and time window, with a 90-second analysis deadline. It cannot run tools, execute SQL, or make changes to the application. If evidence or the model is unavailable, OpenSRE saves a partial report with the cause marked unknown. A report is an evidence-based explanation, not a guaranteed root cause. The span sample is not a collection of complete distributed traces. ## Connect your own application Use standard OpenTelemetry instrumentation and OTLP. Supply `service.name` and `deployment.environment.name`. Your SDK or existing Collector can send to a secured Collector configured to write the same tables. This demo's private, unauthenticated OTLP listeners must not be exposed publicly. The included Collector configuration is a complete, pinned ClickStack Map-schema configuration, not a HyperDX SDK requirement. OpenSRE is not an OTLP receiver: ClickHouse supplies the queryable history. Another backend needs an implementation of the evidence-reader contract; OTLP alone does not standardize historical queries. The current adapter supports the included Map-schema tables, not arbitrary ClickHouse or ClickStack JSON schemas. For another ClickHouse source, copy `watch.yaml`, set its connection and rule, and provide `OPENSRE_TELEMETRY_CLICKHOUSE_PASSWORD` through the environment or OpenSRE's credential store. Use a dedicated `readonly=2` account with SELECT only on `otel_metrics_sum`, `otel_traces`, and `otel_logs`; never use collector or administrator credentials. Use TLS for remote connections. Known credentials, card-like numbers, and emails are redacted before model synthesis. This is not comprehensive DLP: remove sensitive content at collection time before using real telemetry with a model or Telegram. Resource/attribute maps are not exported into the report. Treat this as a single-owner reference pilot, not a hosted multi-tenant ingestion service. ## Stop and remove demo data ```bash theme={null} docker compose -f examples/telemetry-payment/compose.yaml down # Optional, destructive: remove this demo's telemetry volume too. docker compose -f examples/telemetry-payment/compose.yaml down -v ``` Telemetry has a 24-hour TTL; expiry is asynchronous. Local reports persist until you remove the chosen SQLite state file and its sidecars. Preserve that state if you intend to resume the same watch. # Pull request review flow Source: https://opensre.com/docs/guides/pr-review-flow How Greptile, human review, and CI work on OpenSRE pull requests. This page describes what happens after you open a pull request: Greptile review, required CI, and maintainer review. Greptile helps maintainers manage the queue. It does not replace maintainer approval. ## Overview 1. Open a focused pull request with the [PR template](https://github.com/Tracer-Cloud/opensre/blob/main/.github/PULL_REQUEST_TEMPLATE.md) completed. 2. Leave **Allow edits from maintainers** enabled on fork pull requests. 3. Wait for CI. The required check is **CI Gate**. 4. Trigger Greptile and reach **Confidence Score: 5/5** with no unresolved threads. 5. A maintainer reviews and either merges or requests changes. Contribution rules: [CONTRIBUTING.md](https://github.com/Tracer-Cloud/opensre/blob/main/CONTRIBUTING.md). Local checks before you push: [CI.md](https://github.com/Tracer-Cloud/opensre/blob/main/CI.md). ## Greptile (AI review) OpenSRE uses [Greptile](https://greptile.com) for automated review. Opening or reopening a pull request posts a reminder comment. Treat Greptile as a human gate: aim for **5/5** before asking a maintainer to merge. Trigger a review with a pull request comment: ```text theme={null} @greptile review ``` Reviews typically take 5–10 minutes (sometimes longer). Address the feedback, resolve threads you have fixed, then comment `@greptile review` again until you reach **5/5** with no open threads. Optional: the [greploop skill](https://skills.sh/greptileai/skills/greploop) can run that loop for you. Treat Greptile comments like normal review comments. Reply and leave a thread open only when you need a human decision. ## CI Required checks must be green before merge. The branch-protection aggregator is **CI Gate** (quality and unit tests). If CI is red, fix the branch before requesting another Greptile or maintainer pass. Do not open a test-only or CI-only pull request solely to chase failures that are already red on `main`, unless a maintainer requested that work — see [CONTRIBUTING.md](https://github.com/Tracer-Cloud/opensre/blob/main/CONTRIBUTING.md). ## Human review A maintainer decides when the pull request is ready. Greptile 5/5 and green CI are necessary signals, not automatic approval to merge. Maintainers typically review: * Problem statement and evidence in the pull request body (demo, screenshot, or log) * AI-usage disclosure when you used AI assistance * Scope (one concern per pull request; avoid unrelated refactors) * Tests for behavioral changes Keep the pull request description current when the problem, solution, or evidence changes. Comments are for discussion; the body is what reviewers re-read. For coordination help, ask in Discord [#contribute](https://discord.gg/opensre). ## Improving a pull request during review 1. Treat Greptile and maintainer comments as the action list. 2. Push fixes; update the pull request description when the story or evidence changes. 3. Resolve threads you have addressed. 4. Re-run `@greptile review` only after the branch and description are current. 5. Keep discussion on the pull request when possible. Use Discord when you need maintainer coordination or believe automation is stuck. ## If progress stalls * Greptile can take several minutes; do not spam `@greptile review`. * If CI is pending, wait for the next green run. * If the pull request is behind `main`, update the branch (or ask a maintainer), then wait for CI again. * If a maintainer is actively editing the branch, let them finish before pushing competing commits. If you remain blocked after CI is green and Greptile is 5/5, ping [#contribute](https://discord.gg/opensre) with the pull request link and what you are waiting on. # Python API Source: https://opensre.com/docs/guides/python-api Drive the OpenSRE agent in-process from Python. Use the Python API when your code runs on the same machine as OpenSRE and you want agent responses without invoking the CLI. For network access, use the [HTTP API](/docs/guides/api). To supply your own output sink, tools, prompts, or error reporting and drive one agent across many turns, see [Hosting the Agent](/docs/guides/hosting-the-agent). ## OpenSRE as a teammate in your daily loop OpenSRE functions as an autonomous engineering teammate embedded directly into your daily operational workflows. Rather than treating the agent as an isolated chat interface, you can embed the Python API into scheduled cron jobs, CI/CD pipelines, and automated incident triage loops to perform recurring tasks — auditing repository health, triaging production alerts, and delivering digests to team channels like Slack, webhooks, or ticketing systems. ## Prerequisites The standalone CLI installer does not provide an importable package. Install from a source checkout: ```bash theme={null} git clone https://github.com/Tracer-Cloud/opensre.git cd opensre && make install ``` Configure a provider once (`opensre onboard`) — the session API reuses the same config and credentials as the CLI. Run your script inside the checkout's environment with `uv run python your_script.py`. Register adapters before the first turn: ```python theme={null} from bootstrap.process import EMBEDDED_PROFILE, configure_process configure_process(EMBEDDED_PROFILE) ``` ## Daily engineering recipes ### Recipe 1: Querying repository and workflow metrics Use `session.chat()` to query configured developer tools and observability sources (e.g. GitHub star velocity, PR backlog, deployment status): ```python theme={null} from bootstrap.process import EMBEDDED_PROFILE, configure_process from core.agent_harness import AgentSession # 1. Register tools and adapter plugins once per process configure_process(EMBEDDED_PROFILE) # 2. Start an in-process session connected to configured integrations session = AgentSession.start() # 3. Query repository trends or metrics (requires GitHub integration configured) result = session.chat("what's our GitHub star velocity over the last 7 days?") action_ok = ( result.action_result.handled and not result.action_result.has_unhandled_clause and result.action_result.accounting_status == "completed" ) if not (result.answered or action_ok) or result.cancelled or not result.primary_response_text: raise RuntimeError(f"Turn failed: {result.primary_response_text or 'no response produced'}") print(result.primary_response_text) ``` ### Recipe 2: Automated alert triage Hand a raw alert payload from your alertmanager, webhook, or monitoring system to the agent as a chat turn — it calls your connected integrations to gather context and answers with its findings: ```python theme={null} import json from bootstrap.process import EMBEDDED_PROFILE, configure_process from core.agent_harness import AgentSession configure_process(EMBEDDED_PROFILE) session = AgentSession.start() alert_payload = { "alert_name": "HighLatency", "service": "checkout-api", "severity": "warning", "description": "p99 latency exceeded 1200ms in us-east-1", } result = session.chat(f"Triage this alert:\n{json.dumps(alert_payload)}") print(result.primary_response_text) ``` ## One API — `chat` Every surface goes through the same verb: ```python theme={null} from bootstrap.process import EMBEDDED_PROFILE, configure_process from core.agent_harness import AgentSession configure_process(EMBEDDED_PROFILE) session = AgentSession.start() result = session.chat("why is checkout-api slow?") action_ok = ( result.action_result.handled and not result.action_result.has_unhandled_clause and result.action_result.accounting_status == "completed" ) if (result.answered or action_ok) and not result.cancelled: print(result.primary_response_text) ``` Or the one-liner that wires the same boot step for you: ```python theme={null} from bootstrap.embedded import start_embedded_session session = start_embedded_session() ``` `AgentSession.start()` resolves the environment, opens a session, and attaches an agent with the standard ports — the same tools and prompts the interactive shell uses. It does **not** register adapters (`core` may not import `bootstrap`); call `configure_process` first or use `start_embedded_session`. Always verify turn success before trusting chat text: * For conversational questions and digests, the agent synthesizes an answer (`result.answered`). * For action-only turns (tools handled the request directly without an LLM call), verify `result.action_result.handled` with `not result.action_result.has_unhandled_clause` and `result.action_result.accounting_status == "completed"`. * When a turn fails (for example the LLM provider is unreachable) or is cancelled (`result.cancelled`), the failure details land in `result.primary_response_text`. ### Internal seams (not for hosts) Chat hosts terminate at `dispatch_chat_turn` → `run_turn`. Do not invent parallel public entrypoints. ## Unattended daily delivery and background loops For recurring daily workflows (such as scheduled morning digests or CI health checks), start an in-process session and forward findings to team channels: ```python theme={null} from bootstrap.process import EMBEDDED_PROFILE, configure_process from core.agent_harness import AgentSession configure_process(EMBEDDED_PROFILE) session = AgentSession.start() result = session.chat("summarize critical alerts and deployment changes from the last 24h") action_ok = ( result.action_result.handled and not result.action_result.has_unhandled_clause and result.action_result.accounting_status == "completed" ) if not (result.answered or action_ok) or result.cancelled or not result.primary_response_text: raise RuntimeError(f"Scheduled digest failed: {result.primary_response_text or 'no response produced'}") summary = result.primary_response_text # Forward summary to team notification webhooks (Slack, email, or ticketing) print("Delivering daily digest:\n", summary) ``` In the interactive shell, recurring unattended runs are managed with [`/loops`](/docs/platform/cron) — each loop sends its result to the handles OpenSRE can reach (Telegram, Slack, and the local shell inbox). ## A conversation Each `chat` call is one turn in the same session, so follow-ups see earlier context: ```python theme={null} session.chat("list unresolved Sentry issues from the last 24 hours") result = session.chat("which of those affect checkout?") ``` To resume an existing session, pass its ID: ```python theme={null} from core.agent_harness import AgentSession, SessionConfig session = AgentSession.start(SessionConfig(session_id="abc123")) ``` ## Run until a goal is complete Use `chat_until_goal` when one request may need several agent turns. Pass an explicit `SessionGoal` so the completion condition, checklist, and turn limit do not depend on the first turn inferring them: ```python theme={null} from bootstrap.process import EMBEDDED_PROFILE, configure_process from core.agent_harness import AgentSession from core.agent_harness.session_goal import SessionGoal configure_process(EMBEDDED_PROFILE) session = AgentSession.start() outcome = session.chat_until_goal( "Audit active production alert rules and summarize risky thresholds.", goal=SessionGoal( condition=( "Every active production alert rule has been checked and risky " "thresholds have been summarized." ), checklist=( "List active production alert rules", "Check each threshold and evaluation window", "Summarize risky thresholds", ), max_outer_turns=5, ), ) print(outcome.goal.status) print(outcome.turn_count) if outcome.last_result.answered: print(outcome.last_result.primary_response_text) ``` The loop stops when the goal is achieved, paused, cancelled, cleared, or its turn budget is exhausted. The result contains the final `goal`, the `last_result` from `chat`, and `turn_count`. Without `goal=`, the first action turn must attach a goal; otherwise `chat_until_goal` returns after that one turn. Use `cancel_requested` to stop between turns and `on_progress` to receive checklist updates. ## Prompt surface The agent assembles its prompt from the session — the system prompt, connected integrations, memory, the task plan, and earlier turns — so there is no prompt text to pass in. What you choose is the surface the turns run on: `interactive_shell` (the default), `headless_cli`, or `gateway`. The surface decides the surface-specific parts of the prompt, such as setup facts and whether the agent may open a menu. ```python theme={null} from core.agent_harness import AgentSession, SessionConfig session = AgentSession.start(SessionConfig(), surface="headless_cli") ``` A `prompts=` provider — `SessionConfig(prompts=…)`, or `DefaultHeadlessBuild.agent(prompts=…)` on the custom-ports path below — names the surface instead. It implements `core.agent_harness.ports.PromptContextProvider`, whose one method is `surface()`. If omitted, `core.agent_harness.spi.defaults.DefaultPromptContextProvider` is used. ## Custom output and ports `start()` buffers all output. To capture tool progress yourself (for example to stream to a websocket), build the agent and pass your own sink: ```python theme={null} from core.agent_harness import AgentSession, SessionConfig from core.agent_harness.runtime import DefaultHeadlessBuild from core.agent_harness.turns.headless_adapters import BufferOutputSink session = AgentSession(SessionConfig()) startup = session.startup() sink = BufferOutputSink() agent = DefaultHeadlessBuild(session=startup.session, output=sink).agent( prompts=my_provider, # optional — same port as SessionConfig.prompts ) session.attach_agent(agent) session.chat("summarize open incidents") print(sink.lines) # rendered output lines print(sink.streamed) # streamed answer chunks ``` Any object implementing the `OutputSink` protocol (`core.agent_harness.ports.OutputSink`: `print`, `render_response_header`, `render_error`, `stream`) may replace `BufferOutputSink`. `DefaultHeadlessBuild` also takes a custom logger, console, and prompt surface; its `agent()` takes your own tool provider (`tools=`, usually a configured `DefaultToolProvider`) and gather ports — see its docstring. ## External tools Register a package before the first tool lookup: ```python theme={null} from tools.registry import register_external_tool_package import my_agent_tools register_external_tool_package(my_agent_tools) ``` Requirements: 1. Declare tools with `surfaces=("action",)` when the action loop should be able to call them. The `@tool` default is `("chat",)`. 2. Tools may be defined in the package `__init__.py` or in submodules; both are discovered after registration. # Welcome to OpenSRE Source: https://opensre.com/docs/index Install OpenSRE, run your first CI/CD demo, and scale from a single laptop to a managed gateway for your organization. [Quickstart guide →](/docs/getting-started/quickstart) [Create an account →](https://app.opensre.com) ## Choose a path Start with a CI/CD outcome, an incident question, or a rollout for your organization. Pick the one that matches your goal. > **[Measure and fix CI](/docs/cicd/overview)** > > A 30-day reliability report for a repository, fixes for failing checks, and an agent that keeps pull requests green. > > **Run the CI/CD demo →** > **[Investigate an incident](/docs/getting-started/quickstart)** > > Paste an alert or describe what is wrong. OpenSRE reads logs, metrics and deploys from your connected tools and answers in plain English. > > **Run the quickstart →** > **[Roll it out to your team](/docs/install/install-enterprise)** > > One account for the organization and a managed gateway that keeps running repairs after the laptop closes. Slack and other integrations are configured on each engineer's machine. > > **Review the enterprise setup →** ## Start with a surface The same `opensre` binary runs everywhere. Pick a surface, install it, and start your first session. **Interactive shell** Install with one command, sign in, and pick a demo or type a request. ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc opensre ``` Windows uses PowerShell; see [Install](/docs/install). The first prompt offers the CI/CD analysis of a repository on your machine, or skip it with **E**. ![The CI/CD reliability report the first demo produces](https://mintcdn.com/tracer/pusIxCBwQ1b8M9V-/images/onboarding-ci-analysis-report.png?fit=max\&auto=format\&n=pusIxCBwQ1b8M9V-\&q=85\&s=9cd0e8970caeefc6f9900ba86f71b648) **Headless CLI** One agent turn from a script or a CI job, with JSON output and exit codes. ```bash theme={null} opensre --json ask "how many PRs currently have failing CI in Tracer-Cloud?" ``` See [Headless CLI](/docs/guides/headless-cli) for `--allowed-tool` and stdin prompts. **Slack** Mention OpenSRE in a channel or DM it, and hand off DevOps chores to the team's agent. Demo option **D** in the shell connects Slack; see [Slack](/docs/integrations/messaging/slack). **Managed gateway** An organization admin provisions a hosted gateway at app.opensre.com; repairs and scheduled loops run there with the organization's credentials. See [Keep CI green with an agent](/docs/cicd/ci-repair-agent#the-managed-service) and [Deployment](/docs/guides/deployment#opensre-cloud-managed-gateway). > [!TIP] > > Follow the [Quickstart](/docs/getting-started/quickstart) to install, sign in and run your first demo in a few minutes. ## Explore the docs * [Get started](/docs/getting-started/quickstart): install on macOS, Linux, Windows or Docker, sign in, and the enterprise setup. * [CI/CD](/docs/cicd/overview): the reliability report, fixing failing checks, the repair agent, and the organization-wide health scan. * [Using the shell](/docs/getting-started/interactive-shell-commands): every slash command, how prompts map to tools, and goals that run across turns. * [Run it anywhere](/docs/guides/deployment): the gateway daemon, headless CLI, HTTP and Python APIs, and the managed gateway. * [Platform](/docs/platform/llm-providers): LLM providers, masking, sessions, memory, scheduled loops, and closed-loop learning. * [Integrations](/docs/integrations): 60+ tools across monitoring, incidents, cloud, code, chat, databases and workflows. * [Community](/docs/guides/bi-weekly-giveaway): the showcase giveaway and the pull request review flow. ## How a turn runs Say an alert fires and you paste it in: 1. OpenSRE reads the alert and decides what kind of problem it is looking at. 2. It gathers context: who owns the service, what changed recently, what is normal. 3. It checks the tools you connected for logs, metrics and traces. 4. It keeps digging and testing ideas until it can answer. 5. It answers in the shell and sends the result where you asked: Slack, Telegram, a file. Agent workflow ![How OpenSRE Works](https://mintcdn.com/tracer/imE54vEO20PEo3su/images/how_tracer_works.png?fit=max\&auto=format\&n=imE54vEO20PEo3su\&q=85\&s=a3f893b6195654e1cbcc683080a0524b) OpenSRE is open source. It works with the stack you already have, runs locally or hosted for a whole team, and shows how it reached every answer. The [FAQ](/docs/guides/faq) has the short answers. # Docker Source: https://opensre.com/docs/install/docker Run OpenSRE in Docker for a repeatable local or server setup. Use Docker when you want OpenSRE in a container — same image on your laptop or a server. For day-to-day CLI use on your machine, the normal [install](/docs/install) is simpler. ## Prerequisites Install Docker first: > **[Docker Desktop (Mac)](https://docs.docker.com/desktop/install/mac-install/)** > > Install Docker Desktop on macOS. > **[Docker Engine (Ubuntu)](https://docs.docker.com/engine/install/ubuntu/)** > > Install Docker Engine on Ubuntu. Aim for a machine with enough CPU and RAM for your LLM calls and any tools you connect (roughly 4 cores / 8GB is a comfortable start). ## 1. Build the image From a clone of the OpenSRE repo: ```bash theme={null} docker build -t opensre:latest . ``` The image contains the runtime tools. It does not pin an OpenSRE build: on start it installs the current main-channel build. Inside a running container, `opensre update -y` installs a newer main build and restarts the process onto it. Rebuild the image when Git, the GitHub CLI, Node, or Codex should change, not for an application change. The image can run in two modes via `MODE`: * **`web`** (default) — HTTP API for health checks * **`gateway`** — messaging gateway (Slack / Telegram / Discord / Buzz) plus related services ## 2. Run web mode Put your LLM keys (and any integration secrets) in a `.env` file — see `.env.example` in the repo — then: ```bash theme={null} docker run --rm -p 8000:8000 --env-file .env opensre:latest ``` Check it's up: ```bash theme={null} curl http://localhost:8000/health ``` Minimum env for an LLM (example): ```bash theme={null} LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=... ``` ## 3. Run gateway mode (optional) If you want Slack Socket Mode, Telegram, Discord, and/or Buzz on the gateway: ```bash theme={null} docker run --rm -e MODE=gateway --env-file .env opensre:latest ``` You'll need the usual LLM vars plus the matching messaging tokens. Details: [Deployment](/docs/guides/deployment) and [Messaging](/docs/integrations/messaging). ## Prefer the CLI instead? On the host (not in Docker): ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc opensre ``` See [Install](/docs/install) and [Quickstart](/docs/getting-started/quickstart). ## Next * [Deployment](/docs/guides/deployment) — hosting, gateway commands, health checks * [Environment variables](/docs/install/environment-variables) — full config list * [HTTP API](/docs/guides/api) — health and status endpoints # Environment Variables Source: https://opensre.com/docs/install/environment-variables Every OpenSRE setting you can put in the environment or a .env file. All configuration options for OpenSRE can be set via environment variables. This page provides a complete reference. ## Secret storage Environment variables are checked first. Credentials you enter through `opensre onboard` for LLM setup, or through `opensre integrations setup` for tools, are written to `.env` and to an owner-only file at `~/.opensre/credentials.json` (mode `0600`). OpenSRE does not write secrets to the OS keychain. A value in the process environment or a local `.env` file wins over the credentials file. Keep `.env` out of source control. Want credentials only from the environment, never on disk? Set `OPENSRE_DISABLE_KEYRING=1` and export everything yourself. ## LLM providers | Variable | Default | Description | | - | - | - | | `LLM_PROVIDER` | `anthropic` | Which LLM backend to use — see [LLM providers](/docs/platform/llm-providers) | | `LLM_MAX_TOKENS` | `4096` | Max tokens for LLM responses | | `OPENSRE_LLM_TRANSPORT` | `sdk` | `sdk` (native SDKs) or `litellm` | | `OPENSRE_LLM_NATIVE_STRUCTURED_OUTPUT` | | Opt-in provider-native structured outputs (`1`/`true`) | > **Anthropic** > > | Variable | Default | Description | > | - | - | - | > | `ANTHROPIC_API_KEY` | | API key for Anthropic | > | `ANTHROPIC_REASONING_MODEL` | | Model for reasoning tasks | > | `ANTHROPIC_TOOLCALL_MODEL` | | Model for tool calling | > **OpenAI** > > | Variable | Default | Description | > | - | - | - | > | `OPENAI_API_KEY` | | API key for OpenAI | > | `OPENAI_REASONING_MODEL` | | Model for reasoning tasks | > | `OPENAI_TOOLCALL_MODEL` | | Model for tool calling | > **OpenRouter** > > | Variable | Default | Description | > | - | - | - | > | `OPENROUTER_API_KEY` | | API key for OpenRouter | > | `OPENROUTER_MODEL` | | Shared model for all tasks | > | `OPENROUTER_REASONING_MODEL` | | Override model for reasoning | > | `OPENROUTER_TOOLCALL_MODEL` | | Override model for tool calling | > **TrustedRouter** > > | Variable | Default | Description | > | - | - | - | > | `TRUSTEDROUTER_API_KEY` | | API key for TrustedRouter | > | `TRUSTEDROUTER_MODEL` | | Shared model for all tasks | > | `TRUSTEDROUTER_REASONING_MODEL` | | Override model for reasoning | > | `TRUSTEDROUTER_TOOLCALL_MODEL` | | Override model for tool calling | > **DeepSeek** > > | Variable | Default | Description | > | - | - | - | > | `DEEPSEEK_API_KEY` | | API key for DeepSeek | > | `DEEPSEEK_MODEL` | | Shared model for all tasks | > | `DEEPSEEK_REASONING_MODEL` | | Override model for reasoning | > | `DEEPSEEK_TOOLCALL_MODEL` | | Override model for tool calling | > **Google Gemini** > > | Variable | Default | Description | > | - | - | - | > | `GEMINI_API_KEY` | | API key for Google Gemini | > | `GEMINI_MODEL` | | Shared model for all tasks | > | `GEMINI_REASONING_MODEL` | | Override model for reasoning | > | `GEMINI_TOOLCALL_MODEL` | | Override model for tool calling | > **NVIDIA** > > | Variable | Default | Description | > | - | - | - | > | `NVIDIA_API_KEY` | | API key for NVIDIA NIM | > | `NVIDIA_MODEL` | | Shared model for all tasks | > | `NVIDIA_REASONING_MODEL` | | Override model for reasoning | > | `NVIDIA_TOOLCALL_MODEL` | | Override model for tool calling | > **MiniMax** > > | Variable | Default | Description | > | - | - | - | > | `MINIMAX_API_KEY` | | API key for MiniMax | > | `MINIMAX_MODEL` | | Shared model for all tasks | > | `MINIMAX_REASONING_MODEL` | | Override model for reasoning | > | `MINIMAX_TOOLCALL_MODEL` | | Override model for tool calling | > **Groq** > > | Variable | Default | Description | > | - | - | - | > | `GROQ_API_KEY` | | API key for Groq | > | `GROQ_MODEL` | | Shared model for all tasks | > | `GROQ_REASONING_MODEL` | | Override model for reasoning | > | `GROQ_TOOLCALL_MODEL` | | Override model for tool calling | > **Azure OpenAI** > > | Variable | Default | Description | > | - | - | - | > | `AZURE_OPENAI_BASE_URL` | | Azure OpenAI resource URL (`https://.openai.azure.com`) | > | `AZURE_OPENAI_API_KEY` | | API key for Azure OpenAI | > | `AZURE_OPENAI_API_VERSION` | `2024-10-21` | Azure OpenAI API version | > | `AZURE_OPENAI_MODEL` | | Shared deployment name for all slots | > | `AZURE_OPENAI_REASONING_MODEL` | `gpt-5.4-mini` | Deployment for reasoning tasks | > | `AZURE_OPENAI_CLASSIFICATION_MODEL` | `gpt-5.4-mini` | Deployment for classification | > | `AZURE_OPENAI_TOOLCALL_MODEL` | `gpt-5.4-mini` | Deployment for tool calling | > **Amazon Bedrock** > > | Variable | Default | Description | > | - | - | - | > | `BEDROCK_REASONING_MODEL` | | Model for reasoning tasks | > | `BEDROCK_TOOLCALL_MODEL` | | Model for tool calling | > | `BEDROCK_CLASSIFICATION_MODEL` | `us.anthropic.claude-sonnet-4-6` | Model for classification | > **Google Vertex AI** > > | Variable | Default | Description | > | - | - | - | > | `VERTEX_AI_PROJECT` | | GCP project ID (required for Vertex) | > | `VERTEX_AI_LOCATION` | `us-central1` | Vertex AI region | > | `VERTEX_AI_REASONING_MODEL` | `gemini-2.5-pro` | Model for reasoning | > | `VERTEX_AI_TOOLCALL_MODEL` | `gemini-2.5-flash-lite` | Model for tool calling | > | `VERTEX_AI_CLASSIFICATION_MODEL` | `gemini-2.5-flash` | Model for classification | > | `VERTEX_AI_MODEL` | | Shared model override for all slots | > **Ollama (Local)** > > | Variable | Default | Description | > | - | - | - | > | `OLLAMA_MODEL` | `llama3.2` | Model name in Ollama | > | `OLLAMA_HOST` | `http://localhost:11434` | Ollama server URL | ## LLM Reasoning Effort | Variable | Default | Description | | - | - | - | | `OPENSRE_REASONING_EFFORT` | | Effort for extended-thinking models: `low`, `medium`, `high`, or `xhigh`. Unset keeps the provider default, except GPT-6.1 Sol on the native OpenAI SDK and Codex, which sends `xhigh` (extra high) | ## CLI providers > **Codex** > > | Variable | Default | Description | > | - | - | - | > | `CODEX_BIN` | | Path to Codex CLI binary | > | `CODEX_MODEL` | | Model override for Codex | > **Claude Code** > > | Variable | Default | Description | > | - | - | - | > | `CLAUDE_CODE_BIN` | | Path to Claude Code CLI binary | > | `CLAUDE_CODE_MODEL` | | Model override for Claude Code | > **Gemini CLI** > > | Variable | Default | Description | > | - | - | - | > | `GEMINI_CLI_BIN` | | Path to Gemini CLI binary | > | `GEMINI_CLI_MODEL` | | Model override for Gemini CLI | > **Antigravity** > > | Variable | Default | Description | > | - | - | - | > | `ANTIGRAVITY_CLI_BIN` | | Path to Antigravity CLI binary | > | `ANTIGRAVITY_CLI_MODEL` | | Model override for Antigravity | > | `ANTIGRAVITY_CLI_TIMEOUT_SECONDS` | | Timeout for Antigravity CLI invocations | > **OpenCode** > > | Variable | Default | Description | > | - | - | - | > | `OPENCODE_BIN` | | Path to OpenCode CLI binary | > | `OPENCODE_MODEL` | | Model override for OpenCode | > **Cursor** > > | Variable | Default | Description | > | - | - | - | > | `CURSOR_BIN` | | Path to Cursor CLI binary | > | `CURSOR_MODEL` | | Model override for Cursor | > **Kimi** > > | Variable | Default | Description | > | - | - | - | > | `KIMI_BIN` | | Path to Kimi CLI binary | > | `KIMI_MODEL` | | Model override for Kimi | > | `KIMI_API_KEY` | | API key for Kimi | > **GitHub Copilot** > > | Variable | Default | Description | > | - | - | - | > | `COPILOT_BIN` | | Path to GitHub Copilot CLI binary | > | `COPILOT_MODEL` | | Model override for Copilot | > | `COPILOT_HOME` | | Copilot config directory override | > | `COPILOT_GITHUB_TOKEN` | | GitHub token for Copilot auth | > **Grok / xAI** > > | Variable | Default | Description | > | - | - | - | > | `GROK_CLI_BIN` | | Path to Grok Build CLI binary | > | `GROK_CLI_MODEL` | | Model override for Grok Build CLI | > | `XAI_API_KEY` | | API-key auth for Grok Build CLI | > | `XAI_BASE_URL` | | Override xAI API base URL | > | `GROK_CLI_TIMEOUT_SECONDS` | `300` | Timeout for Grok CLI invocations | > **Pi** > > | Variable | Default | Description | > | - | - | - | > | `PI_BIN` | | Path to Pi CLI binary | > | `PI_MODEL` | | Model override for Pi CLI (`provider/model`) | > | `PI_AGENT_DIR` | | Pi agent config directory override | > | `PI_CONFIG_DIR` | | Pi config directory override | > **Custom endpoints** > > | Variable | Default | Description | > | - | - | - | > | `CUSTOM_OPENAI_BASE_URL` | | Base URL for an OpenAI-compatible gateway | > | `CUSTOM_OPENAI_API_KEY` | | API key for `custom-openai` | > | `CUSTOM_OPENAI_MODEL` | | Shared model for all `custom-openai` roles | > | `CUSTOM_OPENAI_TOOLCALL_MODEL` | | Optional toolcall-model override | > | `CUSTOM_ANTHROPIC_BASE_URL` | | Base URL for an Anthropic-compatible gateway | > | `CUSTOM_ANTHROPIC_API_KEY` | | API key for `custom-anthropic` | > | `CUSTOM_ANTHROPIC_MODEL` | | Shared model for all `custom-anthropic` roles | ## Output & Debugging | Variable | Default | Description | | - | - | - | | `TRACER_VERBOSE` | `0` | Verbose OpenSRE CLI/UI output (`1`, `true`, `yes`) | | `TRACER_OUTPUT_FORMAT` | | OpenSRE result output format | | `TRACER_API_URL` | | Base URL for the tracer observability integration / report delivery | | `NO_COLOR` | | Disable colored output (standard convention) | | `COLUMNS` | `80` | Terminal column width for output formatting | | `ENV` | `development` | Environment (`development`, `production`) | ## Credentials and authentication For secrets (`*_TOKEN`, `*_KEY`, `*_PASSWORD`, `*_SECRET`, and similar), OpenSRE looks in this order: 1. Your process environment 2. The owner-only file `~/.opensre/credentials.json` `opensre onboard` saves LLM credentials there so they still work after you clear `.env`. **Not stored as secrets:** webhook-style URLs such as `SLACK_WEBHOOK_URL` and `ROCKETCHAT_WEBHOOK_URL`. Those stay in env / store only — treat them as secrets and don't log them. Want credentials only from the environment, never on disk? Set `OPENSRE_DISABLE_KEYRING=1` and export everything yourself. | Variable | Default | Description | | - | - | - | | `OPENSRE_DISABLE_KEYRING` | `0` | Disable local credential storage; credentials must come from the environment | | `OPENSRE_ACCOUNT_TOKEN` | | Personal OpenSRE account token; with `OPENSRE_DISABLE_KEYRING=1`, also set `OPENSRE_APP_URL` | | `OPENSRE_APP_URL` | `https://app.opensre.com` | OpenSRE app origin; set it explicitly with an environment-only account token | | `OPENSRE_ACCOUNT_LLM_MODEL` | `gpt-6.1-sol` | Model-family hint for a non-interactive hosted gateway using the account LLM route; override only when the web app's hosted model changes family | | `OPENSRE_LLM_AUTH_METADATA_PATH` | `~/.opensre/llm-auth.json` | Non-secret LLM auth metadata path used by prompt-safe status checks | ## Telemetry & Monitoring | Variable | Default | Description | | - | - | - | | `OPENSRE_NO_TELEMETRY` | `0` | Disable all telemetry, including Sentry and product analytics (covers `$ai_generation` prompt/response events) | | `OPENSRE_ANALYTICS_DISABLED` | `0` | Disable product analytics only | | `OPENSRE_SENTRY_DISABLED` | `0` | Disable Sentry error reporting | | `OPENSRE_SENTRY_DSN` | | Override Sentry DSN | | `OPENSRE_SENTRY_LOGGING_DISABLED` | `0` | Disable Sentry log integration | | `OPENSRE_DEPLOYMENT_METHOD` | `local` | Sentry deployment tag: railway, ec2, vercel, or local | | `OPENSRE_PROMPT_LOG_DISABLED` | `0` | Disable prompt logging entirely | | `OPENSRE_PROMPT_LOG_LOCAL_DISABLED` | `0` | Disable local prompt log file | | `OPENSRE_PROMPT_LOG_REDACT` | `1` | Redact known token shapes from prompts/responses before local or PostHog logging | | `OPENSRE_PROMPT_LOG_PATH` | `~/.opensre/prompt_log.jsonl` | Path to local prompt log file | | `OPENSRE_OPERATIONS_LOG_DISABLED` | `0` | Disable the local operations log | | `OPENSRE_OPERATIONS_LOG_PATH` | `~/.opensre/operations_log.jsonl` | Path to the local operations log | | `OPENSRE_OPERATIONS_LOG_MAX_BYTES` | `5242880` | Rotate the local operations log after this many bytes | | `DO_NOT_TRACK` | `0` | Honor global do-not-track preference | | `SENTRY_TRACES_SAMPLE_RATE` | `1.0` | Trace sampling rate for Sentry | | `SENTRY_ERROR_SAMPLE_RATE` | `1.0` | Error sampling rate for Sentry | ## Paths & Directories | Variable | Default | Description | | - | - | - | | `OPENSRE_HOME` | `~/.opensre` | Host root for scheduler files, gateway state, local configuration, and sessions when no organization context root is set | | `OPENSRE_PROJECT_ENV_PATH` | `PROJECT_ROOT/.env` | Path to project-level `.env` file | | `OPENSRE_WIZARD_STORE_PATH` | `~/.opensre/opensre.json` | Path to the local wizard/provider selection store | | `OPENSRE_CONTEXT_ROOT` | | Mounted organization-scoped root for context and sessions; requires `ORGANIZATION_ID` | | `OPENSRE_INTEGRATIONS_STORE_PATH` | | Override integrations store path | ## Conversation history | Variable | Default | Description | | - | - | - | | `OPENSRE_HISTORY_TOKEN_BUDGET` | `48000` | Estimated tokens of conversation history before it is compacted | | `OPENSRE_TOOL_OUTPUT_TOKEN_LIMIT` | `10000` | Approximate tokens per tool observation; output keeps its beginning and end, using four UTF-8 bytes per token | | `OPENSRE_HISTORY_TOOL_RESULT_CHARS` | | Optional additional character cap for recorded tool results; unset keeps the tool observation budget with 20% extra space | | `OPENSRE_LLM_COMPACTION` | `1` | Set to `0` to compact history without a model-written summary | | `OPENSRE_STRUCTURED_HISTORY` | `1` | Set to `0` to send earlier turns as one text block instead of replaying them | ## Memory | Variable | Default | Description | | - | - | - | | `OPENSRE_MEMORY_DISABLED` | `0` | Disable long-term memory | | `OPENSRE_MEMORY_DIR` | | Override memory storage directory | | `OPENSRE_MEMORY_AUTOEXTRACT_DISABLED` | `0` | Disable automatic memory extraction and background upkeep | | `OPENSRE_MEMORY_GATEWAY_ENABLED` | `0` | Opt-in memory on shared gateway hosts | ## Remote sync Mirror conversation history and memory to a user-owned object store. Details: [Remote sync](/docs/install/remote-sync). | Variable | Default | Description | | - | - | - | | `OPENSRE_REMOTE_SYNC` | | Enable remote sync (`1`/`true`) | | `OPENSRE_REMOTE_SYNC_PROVIDER` | `aws` | Store backend (`aws`, `gcs`, `vercel`, `azure`, or `s3compat`) | | `OPENSRE_REMOTE_SYNC_BUCKET` | | Bucket / Blob store name (required when sync is on) | | `OPENSRE_REMOTE_SYNC_PREFIX` | `opensre` | Key prefix inside the store | | `OPENSRE_REMOTE_SYNC_REGION` | | Region override | | `OPENSRE_REMOTE_SYNC_PROFILE` | | Named credentials profile | | `OPENSRE_REMOTE_SYNC_ENDPOINT_URL` | | Custom S3 endpoint when provider is `s3compat` | | `OPENSRE_REMOTE_SYNC_EXCLUDE` | | Comma-separated globs to exclude from sync | | `OPENSRE_REMOTE_SYNC_EXCLUDE_OFF` | | Ignore configured exclusions for one run | | `BLOB_READ_WRITE_TOKEN` | | Vercel Blob token (when provider is `vercel`) | ## Masking | Variable | Default | Description | | - | - | - | | `OPENSRE_MASK_ENABLED` | `false` | Enable reversible identifier masking before external LLM calls | | `OPENSRE_MASK_KINDS` | | Comma-separated mask kinds to enable | | `OPENSRE_MASK_EXTRA_REGEX` | | JSON map of extra mask kinds to regex patterns | ## Feature Flags | Variable | Default | Description | | - | - | - | | `OSRE_HELM_INTEGRATION` | | Enable Helm integration (1/true/yes) | | `HELM_PATH` | `helm` | Path to the Helm binary | | `HELM_KUBE_CONTEXT` | | kubectl context for Helm | | `HELM_KUBECONFIG` | | kubeconfig path for Helm | | `HELM_NAMESPACE` | | Default Helm namespace | | `OPENSRE_RELEASES_API_URL` | GitHub releases API | Override releases API endpoint | ## Gateway and scheduler runtime | Variable | Default | Description | | - | - | - | | `MODE` | `web` | Container entrypoint: `web`, `gateway`, or `scheduler` | | `PORT` | `8000` | HTTP port for the web surface | | `OPENSRE_HOME` | `~/.opensre` | Host root for scheduler files, gateway state, local configuration, and sessions when `OPENSRE_CONTEXT_ROOT` is unset | | `DATABASE_URL` | | Postgres connection for gateway records shared across replicas; required for Slack Events API unless the single-replica local-dedup escape hatch is enabled | | `OPENSRE_SIZE_PROFILE` | `SMALL` | Deployment size profile; default turn caps are `SMALL=1`, `MEDIUM=2`, `LARGE=4` | | `OPENSRE_MAX_CONCURRENT_TURNS` | size-profile limit | Override the process-wide concurrent-turn cap with a positive integer | | `OPENSRE_SCHEDULER_MAX_CONCURRENT_RUNS` | `2` | Maximum scheduled callbacks executing at once; agent turns also need a process-wide turn permit | | `OPENSRE_GATEWAY_HOST_SCHEDULER` | `1` | Run the scheduler inside the gateway; set `0` when a dedicated `MODE=scheduler` service owns it | | `OPENSRE_SESSION_FILE_LOCK` | `0` | Serialize session-file writes with an OS lock; a shared mount is still required across processes | ## Integration credentials > **Observability & Monitoring** > > | Variable | Default | Description | > | - | - | - | > | `GRAFANA_INSTANCE_URL` | | Grafana instance URL | > | `GRAFANA_READ_TOKEN` | | Grafana API token | > | `GRAFANA_LOKI_DATASOURCE_UID` | | UID of Loki datasource in Grafana | > | `GRAFANA_TEMPO_DATASOURCE_UID` | | UID of Tempo datasource in Grafana | > | `GRAFANA_MIMIR_DATASOURCE_UID` | | UID of Mimir datasource in Grafana | > | `TEMPO_URL` | | Grafana Tempo / Tempo API base URL | > | `TEMPO_API_KEY` | | Tempo API key (or use username/password) | > | `TEMPO_USERNAME` | | Tempo basic-auth username | > | `TEMPO_PASSWORD` | | Tempo basic-auth password | > | `TEMPO_ORG_ID` | | Tempo org ID (multi-tenant) | > | `GRAFANA_VERIFY_SSL` | `true` | Verify Grafana SSL certificate | > | `GRAFANA_CA_BUNDLE` | | Path to a CA bundle for Grafana TLS verification (self-signed/internal CA) | > | `DD_API_KEY` | | Datadog API key | > | `DD_APP_KEY` | | Datadog application key | > | `DD_SITE` | `datadoghq.com` | Datadog site (us/eu) | > | `GROUNDCOVER_API_KEY` | | groundcover read-only service-account token (alias: `GROUNDCOVER_MCP_TOKEN`) | > | `GROUNDCOVER_MCP_URL` | `https://mcp.groundcover.com/api/mcp` | groundcover MCP endpoint | > | `GROUNDCOVER_TENANT_UUID` | | Tenant UUID (multi-workspace accounts) | > | `GROUNDCOVER_BACKEND_ID` | | Backend ID (multi-backend tenants) | > | `GROUNDCOVER_TIMEZONE` | `UTC` | Timezone for returned timestamps (`X-Timezone`) | > | `HONEYCOMB_API_KEY` | | Honeycomb API key | > | `HONEYCOMB_DATASET` | `__all__` | Honeycomb dataset slug | > | `HONEYCOMB_API_URL` | `https://api.honeycomb.io` | Honeycomb API endpoint | > | `CORALOGIX_API_KEY` | | Coralogix API key | > | `CORALOGIX_API_URL` | `https://api.coralogix.com` | Coralogix API endpoint | > | `CORALOGIX_APPLICATION_NAME` | | Application name in Coralogix | > | `CORALOGIX_SUBSYSTEM_NAME` | | Subsystem name in Coralogix | > | `SIGNOZ_URL` | | SigNoz instance URL | > | `SIGNOZ_API_KEY` | | SigNoz API key | > | `ALERTMANAGER_URL` | | Alertmanager URL | > | `ALERTMANAGER_BEARER_TOKEN` | | Bearer token for Alertmanager | > | `ALERTMANAGER_USERNAME` | | Username for Alertmanager basic auth | > | `ALERTMANAGER_PASSWORD` | | Password for Alertmanager basic auth | > | `SPLUNK_URL` | | Splunk instance URL | > | `SPLUNK_TOKEN` | | Splunk API token | > | `SPLUNK_INDEX` | `main` | Splunk index | > | `SPLUNK_VERIFY_SSL` | `true` | Verify Splunk SSL certificate | > | `SPLUNK_CA_BUNDLE` | | Path to a CA bundle for Splunk TLS verification | > | `SENTRY_URL` | `https://sentry.io` | Sentry instance URL | > | `SENTRY_ORG_SLUG` | | Sentry organization slug | > | `SENTRY_PROJECT_SLUG` | | Sentry project slug | > | `SENTRY_AUTH_TOKEN` | | Sentry auth token | > | `POSTHOG_PROJECT_ID` | | PostHog project ID | > | `POSTHOG_PERSONAL_API_KEY` | | PostHog personal API key | > | `POSTHOG_BASE_URL` | `https://us.posthog.com` | PostHog app URL (`https://eu.posthog.com` for EU) | > | `POSTHOG_TIMEOUT_SECONDS` | `15.0` | Timeout in seconds for PostHog API requests | > | `NEW_RELIC_API_KEY` | | New Relic User key (`NRAK-...`) | > | `NEW_RELIC_ACCOUNT_ID` | | New Relic account ID | > | `NEW_RELIC_API_URL` | `https://api.newrelic.com` | New Relic API URL (EU/JP endpoints supported) | > **Cloud Platforms** > > | Variable | Default | Description | > | - | - | - | > | `AWS_REGION` | `us-east-1` | AWS region | > | `AWS_ROLE_ARN` | | AWS role ARN for cross-account access | > | `AWS_EXTERNAL_ID` | | External ID for AWS cross-account role | > | `AWS_ACCESS_KEY_ID` | | AWS access key | > | `AWS_SECRET_ACCESS_KEY` | | AWS secret access key | > | `AWS_SESSION_TOKEN` | | AWS session token | > | `CLOUD_PROVIDER` | | Cloud provider hint for ambient region discovery | > | `CLOUD_REGION` | | Cloud region (falls back to `AWS_REGION`) | > | `KUBECONFIG` | | Path to kubeconfig file | > | `KUBECONFIG_CONTENT` | | Inline kubeconfig contents | > | `KUBECONFIG_CONTEXT` | | kubectl context | > | `KUBECONFIG_NAMESPACE` | `default` | Default Kubernetes namespace | > | `RDS_DB_INSTANCE_IDENTIFIER` | | AWS RDS DB instance identifier | > | `RDS_REGION` | | AWS region for RDS (falls back to `AWS_REGION`) | > | `RAILWAY_TOKEN` | | Railway API token | > | `RAILWAY_PROJECT` | | Railway project | > | `RAILWAY_SERVICE` | | Railway service | > | `RAILWAY_ENVIRONMENT` | `production` | Railway environment | > | `RAILWAY_PATH` | `railway` | Path to Railway CLI | > | `ARGOCD_BASE_URL` | | Argo CD API base URL | > | `ARGOCD_AUTH_TOKEN` | | Argo CD auth token | > | `ARGOCD_TOKEN` | | Alias for `ARGOCD_AUTH_TOKEN` | > | `ARGOCD_USERNAME` | | Argo CD username (basic auth) | > | `ARGOCD_PASSWORD` | | Argo CD password (basic auth) | > | `ARGOCD_PROJECT` | | Argo CD project | > | `ARGOCD_APP_NAMESPACE` | | Argo CD app namespace | > | `ARGOCD_VERIFY_SSL` | `true` | Verify Argo CD TLS | > **Messaging** > > | Variable | Default | Description | > | - | - | - | > | `SLACK_WEBHOOK_URL` | | Slack incoming webhook URL | > | `SLACK_BOT_TOKEN` | | Slack bot token | > | `SLACK_DEFAULT_CHAT_ID` | | Default Slack channel for scheduled delivery (bot-token path) | > | `SLACK_APP_TOKEN` | | Slack app-level token (Socket Mode) | > | `SLACK_GATEWAY_INBOUND_TRANSPORT` | `socket_mode` | How Slack delivers events: `socket_mode` or `events_api_http` | > | `SLACK_SIGNING_SECRET` | | Request-signature secret. Required for `events_api_http` | > | `SLACK_GATEWAY_HTTP_PORT` | `3000` | Port the `events_api_http` listener binds | > | `SLACK_GATEWAY_ALLOW_LOCAL_DEDUP` | | Accept process-local event dedup when `DATABASE_URL` is unset. Single replica only | > | `SLACK_GATEWAY_MAX_CONCURRENT` | size-profile limit | Parallel Slack turns before the process-wide cap is applied | > | `SLACK_ALLOWED_USERS` | | Comma-separated Slack user IDs allowed to talk to the bot | > | `SLACK_ALLOW_OPEN_WORKSPACE` | | Allow any workspace user (`1` for dogfood) | > | `SLACK_ACCESS_TOKEN` | | Slack user token | > | `SLACK_GITHUB_ISSUES_WEBHOOK_URL` | | Slack webhook for GitHub issues | > | `DISCORD_BOT_TOKEN` | | Discord bot token | > | `DISCORD_APPLICATION_ID` | | Discord application ID | > | `DISCORD_PUBLIC_KEY` | | Discord public key | > | `DISCORD_DEFAULT_CHANNEL_ID` | | Default Discord channel | > | `DISCORD_ALLOWED_USERS` | | Comma-separated Discord user IDs allowed to talk to the bot | > | `DISCORD_ALLOW_OPEN_GUILD` | | Allow any guild member (`1` for dogfood) | > | `DISCORD_GATEWAY_MAX_CONCURRENT` | size-profile limit | Parallel Discord turns before the process-wide cap is applied | > | `TELEGRAM_BOT_TOKEN` | | Telegram bot token | > | `TELEGRAM_DEFAULT_CHAT_ID` | | Default Telegram chat ID | > | `TELEGRAM_ALLOWED_USERS` | | Comma-separated Telegram user IDs allowed to talk to the bot | > | `TELEGRAM_GATEWAY_MAX_CONCURRENT` | size-profile limit | Parallel Telegram turns before the process-wide cap is applied | > | `BUZZ_PRIVATE_KEY` | | Nostr private key used by the Buzz integration and gateway | > | `BUZZ_RELAY_URL` | `http://localhost:3000` | Buzz relay URL | > | `BUZZ_DEFAULT_CHANNEL` | | Default Buzz channel UUID for delivery | > | `BUZZ_AUTH_TAG` | | Optional NIP-OA owner-attestation JSON | > | `BUZZ_PATH` | `buzz` | Path to the Buzz CLI | > | `BUZZ_ALLOWED_PUBKEYS` | | Comma-separated 64-character hex pubkeys allowed to talk to the gateway | > | `BUZZ_GATEWAY_MAX_CONCURRENT` | `4` | Parallel Buzz turns before the process-wide cap is applied | > | `ROCKETCHAT_SERVER_URL` | | Rocket.Chat server base URL | > | `ROCKETCHAT_AUTH_TOKEN` | | Rocket.Chat personal access token | > | `ROCKETCHAT_USER_ID` | | Rocket.Chat user ID for the token | > | `ROCKETCHAT_DEFAULT_CHANNEL` | | Default Rocket.Chat channel | > | `ROCKETCHAT_WEBHOOK_URL` | | Rocket.Chat incoming webhook URL | > | `TWILIO_ACCOUNT_SID` | | Twilio account SID | > | `TWILIO_AUTH_TOKEN` | | Twilio auth token | > | `TWILIO_WHATSAPP_FROM` | | WhatsApp sender number (Twilio) | > | `WHATSAPP_DEFAULT_TO` | | Default WhatsApp recipient | > | `TWILIO_SMS_FROM` | | SMS sender number (Twilio) | > | `TWILIO_SMS_MESSAGING_SERVICE_SID` | | SMS messaging service SID | > | `TWILIO_SMS_DEFAULT_TO` | | Default SMS recipient | > | `SMTP_HOST` | | SMTP server hostname | > | `SMTP_PORT` | `587` | SMTP port | > | `SMTP_SECURITY` | `starttls` | SMTP security (`starttls`, `ssl`, or `none`) | > | `SMTP_USERNAME` | | SMTP username | > | `SMTP_PASSWORD` | | SMTP password | > | `SMTP_FROM_ADDRESS` | | From address for outbound mail | > | `SMTP_DEFAULT_TO` | | Default recipient for notifications | > **Code & Collaboration** > > | Variable | Default | Description | > | - | - | - | > | `GITHUB_MCP_MODE` | `streamable-http` | GitHub MCP transport mode | > | `GITHUB_MCP_URL` | | GitHub MCP server URL | > | `GITHUB_MCP_COMMAND` | | GitHub MCP command to execute | > | `GITHUB_MCP_ARGS` | | GitHub MCP command arguments | > | `GITHUB_MCP_AUTH_TOKEN` | | Does not authorize agent GitHub tools | > | `GITHUB_MCP_TOOLSETS` | `repos,issues,pull_requests,actions,search` | Enabled GitHub MCP toolsets | > | `OPENSRE_GITHUB_MCP_REPO_PROBE_LIMIT` | `50` | Max repos to probe (5-500) | > | `GITLAB_BASE_URL` | | GitLab instance URL | > | `GITLAB_ACCESS_TOKEN` | | GitLab personal access token | > | `GITLAB_PROJECT_ID` | | GitLab project ID | > | `JIRA_BASE_URL` | | Jira instance URL | > | `JIRA_EMAIL` | | Jira user email | > | `JIRA_API_TOKEN` | | Jira API token | > | `JIRA_PROJECT_KEY` | | Default Jira project key | > | `BITBUCKET_WORKSPACE` | | Bitbucket workspace name | > | `BITBUCKET_USERNAME` | | Bitbucket username | > | `BITBUCKET_APP_PASSWORD` | | Bitbucket app password | > | `BITBUCKET_MAX_RESULTS` | `25` | Max results per API call | > | `INCIDENT_IO_BASE_URL` | | incident.io API URL | > | `INCIDENT_IO_API_KEY` | | incident.io API key | > | `PAGERDUTY_API_KEY` | | PagerDuty API key | > | `PAGERDUTY_BASE_URL` | | PagerDuty API base URL | > | `SERVICENOW_INSTANCE_URL` | | ServiceNow instance URL | > | `SERVICENOW_USERNAME` | | ServiceNow username | > | `SERVICENOW_PASSWORD` | | ServiceNow password | > | `VERCEL_API_TOKEN` | | Vercel API token | > | `VERCEL_TEAM_ID` | | Vercel team ID | > | `VERCEL_RUNTIME_LOGS_READ_TIMEOUT` | `600` | Seconds to wait per Vercel runtime-log read (minimum `30`) | > | `GOOGLE_CREDENTIALS_FILE` | | Path to Google service-account JSON for Docs export | > | `GOOGLE_DRIVE_FOLDER_ID` | | Google Drive folder for Docs export | > **Databases** > > | Variable | Default | Description | > | - | - | - | > | `MONGODB_CONNECTION_STRING` | | MongoDB connection string | > | `MONGODB_DATABASE` | | MongoDB database name | > | `MONGODB_AUTH_SOURCE` | `admin` | MongoDB auth source | > | `MONGODB_TLS` | `true` | Enable MongoDB TLS | > | `MONGODB_ATLAS_PUBLIC_KEY` | | MongoDB Atlas public API key | > | `MONGODB_ATLAS_PRIVATE_KEY` | | MongoDB Atlas private API key | > | `MONGODB_ATLAS_PROJECT_ID` | | MongoDB Atlas project ID | > | `MONGODB_ATLAS_BASE_URL` | `https://cloud.mongodb.com/api/atlas/v2` | MongoDB Atlas API base URL | > | `REDIS_HOST` | | Redis host | > | `REDIS_PORT` | `6379` | Redis port | > | `REDIS_USERNAME` | | Redis ACL username (Redis 6+) | > | `REDIS_PASSWORD` | | Redis password | > | `REDIS_DATABASE` | `0` | Redis database number | > | `REDIS_SSL` | `false` | Connect using TLS | > | `POSTGRESQL_HOST` | | PostgreSQL host | > | `POSTGRESQL_PORT` | `5432` | PostgreSQL port | > | `POSTGRESQL_DATABASE` | | PostgreSQL database name | > | `POSTGRESQL_USERNAME` | `postgres` | PostgreSQL username | > | `POSTGRESQL_PASSWORD` | | PostgreSQL password | > | `POSTGRESQL_SSL_MODE` | `prefer` | PostgreSQL SSL mode | > | `MYSQL_HOST` | | MySQL host | > | `MYSQL_PORT` | `3306` | MySQL port | > | `MYSQL_DATABASE` | | MySQL database name | > | `MYSQL_USERNAME` | `root` | MySQL username | > | `MYSQL_PASSWORD` | | MySQL password | > | `MYSQL_SSL_MODE` | `preferred` | MySQL SSL mode | > | `MARIADB_HOST` | | MariaDB host | > | `MARIADB_PORT` | `3306` | MariaDB port | > | `MARIADB_DATABASE` | | MariaDB database name | > | `MARIADB_USERNAME` | | MariaDB username | > | `MARIADB_PASSWORD` | | MariaDB password | > | `MARIADB_SSL` | `true` | Enable MariaDB SSL | > **Message Queues & Other Services** > > | Variable | Default | Description | > | - | - | - | > | `RABBITMQ_HOST` | | RabbitMQ host | > | `RABBITMQ_MANAGEMENT_PORT` | `15672` | RabbitMQ management port | > | `RABBITMQ_USERNAME` | | RabbitMQ username | > | `RABBITMQ_PASSWORD` | | RabbitMQ password | > | `RABBITMQ_VHOST` | `/` | RabbitMQ vhost | > | `RABBITMQ_SSL` | `false` | Enable RabbitMQ SSL | > | `RABBITMQ_VERIFY_SSL` | `true` | Verify RabbitMQ SSL certificate | > | `BETTERSTACK_QUERY_ENDPOINT` | | Better Stack query endpoint | > | `BETTERSTACK_USERNAME` | | Better Stack username | > | `BETTERSTACK_PASSWORD` | | Better Stack password | > | `BETTERSTACK_SOURCES` | | Better Stack sources | > | `KAFKA_BOOTSTRAP_SERVERS` | | Kafka broker addresses | > | `KAFKA_SECURITY_PROTOCOL` | `PLAINTEXT` | Kafka security protocol | > | `KAFKA_SASL_MECHANISM` | | Kafka SASL mechanism | > | `KAFKA_SASL_USERNAME` | | Kafka SASL username | > | `KAFKA_SASL_PASSWORD` | | Kafka SASL password | > | `DAGSTER_ENDPOINT` | | Dagster GraphQL endpoint | > | `DAGSTER_API_TOKEN` | | Dagster Cloud User Token (leave empty for unauthenticated local OSS) | > | `TEMPORAL_API_URL` | | Temporal Cloud / server API URL | > | `TEMPORAL_NAMESPACE` | `default` | Temporal namespace | > | `TEMPORAL_API_KEY` | | Temporal API key (optional for local) | > | `JENKINS_URL` | | Jenkins base URL | > | `JENKINS_USER` | | Jenkins username | > | `JENKINS_API_TOKEN` | | Jenkins API token | > **Data & Search** > > | Variable | Default | Description | > | - | - | - | > | `OPENSEARCH_URL` | | OpenSearch instance URL | > | `OPENSEARCH_USERNAME` | | OpenSearch username | > | `OPENSEARCH_PASSWORD` | | OpenSearch password | > | `OPENSEARCH_API_KEY` | | OpenSearch API key (alternative to username/password) | > | `OPENSEARCH_INDEX_PATTERN` | `*` | OpenSearch index pattern | > | `OPENSEARCH_MAX_RESULTS` | `100` | Max OpenSearch results | > | `OPENOBSERVE_URL` | | OpenObserve instance URL | > | `OPENOBSERVE_TOKEN` | | OpenObserve API token | > | `OPENOBSERVE_USERNAME` | | OpenObserve username | > | `OPENOBSERVE_PASSWORD` | | OpenObserve password | > | `OPENOBSERVE_ORG` | `default` | OpenObserve organization | > | `OPENOBSERVE_STREAM` | | OpenObserve stream | > | `OPENOBSERVE_MAX_RESULTS` | `100` | Max OpenObserve results | > | `VICTORIA_LOGS_URL` | | VictoriaLogs instance URL | > | `VICTORIA_LOGS_TENANT_ID` | | VictoriaLogs tenant ID | > | `CLICKHOUSE_HOST` | | ClickHouse host | > | `CLICKHOUSE_PORT` | `8123` | ClickHouse port | > | `CLICKHOUSE_DATABASE` | `default` | ClickHouse database | > | `CLICKHOUSE_USER` | `default` | ClickHouse username | > | `CLICKHOUSE_PASSWORD` | | ClickHouse password | > | `CLICKHOUSE_SECURE` | `false` | Enable ClickHouse TLS | > | `SNOWFLAKE_ACCOUNT_IDENTIFIER` | | Snowflake account identifier | > | `SNOWFLAKE_ACCOUNT` | | Snowflake account (alternative) | > | `SNOWFLAKE_TOKEN` | | Snowflake token | > | `SNOWFLAKE_USER` | | Snowflake username | > | `SNOWFLAKE_PASSWORD` | | Snowflake password | > | `SNOWFLAKE_WAREHOUSE` | | Snowflake warehouse | > | `SNOWFLAKE_ROLE` | | Snowflake role | > | `SNOWFLAKE_DATABASE` | | Snowflake database | > | `SNOWFLAKE_SCHEMA` | | Snowflake schema | > | `SNOWFLAKE_MAX_RESULTS` | `50` | Max Snowflake results | > | `AZURE_SQL_SERVER` | | Azure SQL server name | > | `AZURE_SQL_DATABASE` | | Azure SQL database name | > | `AZURE_SQL_PORT` | `1433` | Azure SQL port | > | `AZURE_SQL_USERNAME` | | Azure SQL username | > | `AZURE_SQL_PASSWORD` | | Azure SQL password | > | `AZURE_SQL_DRIVER` | `ODBC Driver 18 for SQL Server` | Azure SQL ODBC driver | > | `AZURE_SQL_ENCRYPT` | `true` | Enable Azure SQL encryption | > | `AZURE_LOG_ANALYTICS_WORKSPACE_ID` | | Azure Log Analytics workspace ID | > | `AZURE_LOG_ANALYTICS_TOKEN` | | Azure Log Analytics token | > | `AZURE_LOG_ANALYTICS_ENDPOINT` | `https://api.loganalytics.io` | Log Analytics Query API endpoint (sovereign clouds) | > | `AZURE_TENANT_ID` | | Azure tenant ID | > | `AZURE_SUBSCRIPTION_ID` | | Azure subscription ID | > | `AZURE_MAX_RESULTS` | `100` | Max Azure results | > **MCP Servers** > > | Variable | Default | Description | > | - | - | - | > | `POSTHOG_MCP_MODE` | `streamable-http` | PostHog MCP transport mode | > | `POSTHOG_MCP_URL` | `https://mcp.posthog.com/mcp` | PostHog MCP URL | > | `POSTHOG_MCP_AUTH_TOKEN` | | PostHog MCP auth token | > | `POSTHOG_MCP_PROJECT_ID` | | PostHog project scoped to MCP | > | `POSTHOG_MCP_ORGANIZATION_ID` | | PostHog organization scoped to MCP | > | `POSTHOG_MCP_FEATURES` | | Comma-separated PostHog MCP features | > | `POSTHOG_MCP_READ_ONLY` | `true` | Restrict PostHog MCP to read-only | > | `POSTHOG_MCP_COMMAND` | | PostHog MCP stdio command | > | `POSTHOG_MCP_ARGS` | | PostHog MCP stdio args | > | `SENTRY_MCP_MODE` | `streamable-http` | Sentry MCP transport mode | > | `SENTRY_MCP_URL` | `https://mcp.sentry.dev/mcp` | Sentry MCP URL | > | `SENTRY_MCP_AUTH_TOKEN` | | Sentry MCP auth token | > | `SENTRY_MCP_HOST` | | Self-hosted Sentry MCP host | > | `SENTRY_MCP_ORGANIZATION_SLUG` | | Sentry org slug for MCP | > | `SENTRY_MCP_PROJECT_SLUG` | | Sentry project slug for MCP | > | `SENTRY_MCP_SKILLS` | | Enabled Sentry MCP skills | > | `SENTRY_MCP_COMMAND` | | Sentry MCP stdio command | > | `SENTRY_MCP_ARGS` | | Sentry MCP stdio args | > **Git Integrations** > > | Variable | Default | Description | > | - | - | - | > | `GITHUB_EVENT_PATH` | | GitHub Actions event payload path | > | `GITHUB_REPOSITORY` | | GitHub repository (owner/repo) | > | `GITHUB_TOKEN` | | Does not authorize agent GitHub tools; connect GitHub in the OpenSRE app | > | `GH_TOKEN` | | Does not authorize agent GitHub tools | ## Multi-instance integrations Need more than one Datadog, Grafana, or AWS account (for example prod and staging)? Use a JSON `*_INSTANCES` variable: | Variable | What it configures | | - | - | | `GRAFANA_INSTANCES` | Multiple Grafana instances | | `DD_INSTANCES` | Multiple Datadog sites/accounts | | `GROUNDCOVER_INSTANCES` | Multiple groundcover instances | | `HONEYCOMB_INSTANCES` | Multiple Honeycomb instances | | `CORALOGIX_INSTANCES` | Multiple Coralogix instances | | `SPLUNK_INSTANCES` | Multiple Splunk instances | | `AWS_INSTANCES` | Multiple AWS accounts | | `ARGOCD_INSTANCES` | Multiple Argo CD instances | Format and examples: [Multi-instance integrations](/docs/platform/multi-instance-integrations). ## LLM classification models If you don't set these, OpenSRE falls back to the reasoning model (or the provider default): | Variable | Default | Description | | - | - | - | | `ANTHROPIC_CLASSIFICATION_MODEL` | | Anthropic classification model | | `OPENAI_CLASSIFICATION_MODEL` | | OpenAI classification model | | `OPENROUTER_CLASSIFICATION_MODEL` | | OpenRouter classification model | | `TRUSTEDROUTER_CLASSIFICATION_MODEL` | | TrustedRouter classification model | | `DEEPSEEK_CLASSIFICATION_MODEL` | | DeepSeek classification model | | `GEMINI_CLASSIFICATION_MODEL` | | Gemini classification model | | `NVIDIA_CLASSIFICATION_MODEL` | | NVIDIA classification model | | `MINIMAX_CLASSIFICATION_MODEL` | | MiniMax classification model | | `GROQ_CLASSIFICATION_MODEL` | | Groq classification model | # Install OpenSRE Source: https://opensre.com/docs/install/index Install the OpenSRE CLI on your machine and start chatting with the agent. ## Before you begin Make sure you have: * A terminal open in a code project * A Git repository (recommended for the full workflow demonstration) ## Step 1: Install and start opensre **macOS and Linux** ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc ``` Prebuilt Linux binaries need glibc 2.35 or newer (Ubuntu 22.04 or comparable); see [Linux](/docs/install/linux-local). **Windows** Open PowerShell and run: ```powershell theme={null} & ([scriptblock]::Create((irm https://install.opensre.com/install.ps1))) -dc ``` See [Windows](/docs/install/windows-local). Then start OpenSRE: ```bash theme={null} opensre ``` ![OpenSRE CLI welcome screen with the sign-in prompt](https://mintcdn.com/tracer/N6OGfOUQjx4VJof5/images/opensre-welcome.png?fit=max\&auto=format\&n=N6OGfOUQjx4VJof5\&q=85\&s=a111e922dc38f18a550747d6974f2521) For PATH troubleshooting and platform requirements, see [Install locally](/docs/install/install-local). ## Step 2: Sign in Choose **Sign in or create account** in the welcome screen. Your browser opens the OpenSRE sign-up page; sign in with email or another enabled provider. Signing in activates the hosted model, so you need no LLM key. Back in the terminal, OpenSRE confirms the session with the web app and opens the shell. ## Step 3: Pick what to do The first shell prompt asks what you want to do: ```text theme={null} What would you like to do? (A) Analyze & improve a repo (recommended) (B) Keep my CI/CD healthy automatically (C) Open the shell ``` Choosing **B** asks how that should run: ```text theme={null} Keep my CI/CD healthy automatically (A) Run continuously on this machine (recommended) (B) Run one repair in OpenSRE Cloud (C) Connect Slack ``` Choose **A** to see what OpenSRE does with a repository on your machine: it reads the last 30 days of GitHub Actions history and reports how reliable your CI is and how much developer time it costs. It needs GitHub access: connect it in the OpenSRE app through OAuth or its token-entry form. Run `opensre account login` to load that connection on this machine. If it is missing, the flow offers the app setup link. See [CI/CD analytics demo](/docs/cicd/cicd-analytics-demo). Choose **C** to go straight to the prompt. Type `/demo` to open the menu again at any time, or start with `opensre --skip-onboarding` to skip it. ## Step 4: Ask your first question Describe what's going on in plain English: ```text theme={null} why is checkout latency spiking? ``` Connect the tools OpenSRE should read when you need them, from the shell (`/integrations setup github`) or the terminal: ```bash theme={null} opensre integrations setup ``` Full walkthrough: [Quickstart](/docs/getting-started/quickstart). ## Environment-specific guides | Platform | Guide | | - | - | | macOS | [macOS](/docs/install/macos) | | Linux | [Linux (local)](/docs/install/linux-local) | | Windows | [Windows (local)](/docs/install/windows-local) | | Docker | [Docker](/docs/install/docker) | ## Stuck? * **Command not found** — open a new terminal, or add the bin directory the installer printed to your PATH * **The sign-in screen comes back after you signed in** — run `opensre account status`; it names what is wrong with the session * **Anything else** — see [Quickstart troubleshooting](/docs/getting-started/quickstart#troubleshooting) Need deploy options beyond a laptop binary? See [Deployment](/docs/guides/deployment). # Enterprise setup (OpenSRE Cloud) Source: https://opensre.com/docs/install/install-enterprise Set up OpenSRE for your organization in the hosted OpenSRE Cloud web app This guide is for teams using the hosted OpenSRE account and an optional organization-managed gateway at [app.opensre.com](https://app.opensre.com). ## 1) Create your organization account Sign up at [app.opensre.com](https://app.opensre.com) to create or join your organization workspace. The same account activates the hosted model when an engineer signs in from the CLI. ## 2) Complete workspace onboarding In the web app, complete the available workspace onboarding to: * Confirm the organization and membership * Review account usage * Provision the organization-managed gateway, if your plan includes one ## 3) Run the local CLI Each engineer installs OpenSRE and signs in with the organization account: ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc opensre ``` The browser sign-in activates the account-hosted model. GitHub and production systems are still optional integrations; account sign-in does not connect them. ## 4) Manage the organization gateway An organization admin provisions the gateway from [Agent Backend settings](https://app.opensre.com/settings/agent-backend). Once it exists, a signed-in shell user can ask OpenSRE to check its health. Admins can also ask the shell to start or stop it; both operations require approval. Stopping the gateway keeps its configuration and credentials. Remote loops and chat integrations remain stopped until an admin starts it again. See [Deployment](/docs/guides/deployment#opensre-cloud-managed-gateway). ## 5) Configure integrations locally Organization-wide hosted connectors are not available in the current CLI release. Configure the integrations an engineer or self-hosted gateway needs on that machine: ```bash theme={null} opensre integrations setup opensre integrations verify ``` Start with [Integrations overview](/docs/integrations). The signed-in shell uses the account-hosted model. A bring-your-own provider is needed only where nobody is signed in: self-hosted gateways and headless `opensre ask` jobs on CI machines, see [LLM providers](/docs/platform/llm-providers). For a completely self-hosted gateway instead of the managed service, follow [Deployment](/docs/guides/deployment#self-hosted-container). # Local and on-prem setup Source: https://opensre.com/docs/install/install-local Run OpenSRE on your laptop, a VM, or an on-prem host with the local CLI. ## Before you begin Make sure you have: * A terminal open in a code project * A Git repository (recommended for the full workflow demonstration) ## Step 1: Install and start opensre ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc ``` Then start OpenSRE: ```bash theme={null} opensre ``` ![OpenSRE CLI welcome screen with the sign-in prompt](https://mintcdn.com/tracer/N6OGfOUQjx4VJof5/images/opensre-welcome.png?fit=max\&auto=format\&n=N6OGfOUQjx4VJof5\&q=85\&s=a111e922dc38f18a550747d6974f2521) On Windows, open PowerShell and run: ```powershell theme={null} & ([scriptblock]::Create((irm https://install.opensre.com/install.ps1))) -dc ``` See [Windows](/docs/install/windows-local). You usually don't need `sudo`. The installer uses a writable directory on your PATH when it can. Otherwise, it uses `~/.local/bin` and prints how to add that to PATH. For platform requirements, see [macOS](/docs/install/macos), [Linux](/docs/install/linux-local), or [Windows](/docs/install/windows-local). For a container, see [Docker](/docs/install/docker). ## Step 2: Sign in Choose **Sign in or create account** in the welcome screen. Your browser opens the OpenSRE sign-up page; sign in with email or another enabled provider. Signing in activates the hosted model, so you need no LLM key. Back in the terminal, OpenSRE confirms the session with the web app, opens the shell and offers the [demos](/docs/getting-started/quickstart#step-3-pick-a-demo-or-skip-it); choose **E** to go straight to the prompt. If the sign-in screen comes back instead, `opensre account status` names what is wrong with the session. ## Step 3: Check your tools Inside the shell: ```bash theme={null} integrations verify ``` One tool only: ```bash theme={null} integrations verify datadog ``` From a normal terminal, prefix with `opensre` (for example `opensre integrations verify`). ## Step 4: Ask a first question Start the shell and describe what's going on in plain English: ```bash theme={null} opensre ``` Ask: *why is checkout latency spiking?* The agent calls the tools you connected and answers in the shell. ## Where to connect tools The full tool list lives under **Integrations**, not Install. Start at [Integrations overview](/docs/integrations), then open each provider page for keys, permissions, and examples. ## Related * [Quickstart](/docs/getting-started/quickstart) — install, sign in, pick a demo, and ask a question * [Deployment](/docs/guides/deployment) — gateway and hosted options beyond a single laptop # Linux Source: https://opensre.com/docs/install/linux-local Install OpenSRE on a local Linux machine and ask your first question. Install OpenSRE on Linux (laptop, desktop, or a local VM), sign in to your account, then ask the agent your first question. ## 1. Install ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc ``` You usually don't need `sudo`. If `opensre` isn't found after install, open a new terminal or add the bin directory the installer printed (often `~/.local/bin`) to your PATH. ### Binary compatibility Prebuilt Linux binaries require **glibc 2.35+** (Ubuntu 22.04+ or a comparable distribution) and do not run on Alpine Linux. Use a supported Linux distribution or the [Docker guide](/docs/install/docker). ## 2. Set things up ```bash theme={null} opensre ``` Create an OpenSRE account or sign in to activate the hosted model. The first prompt offers a demo; pick the CI/CD analysis of a repository on this machine, or choose **E** to skip it. Add tools later, when you need them: ```bash theme={null} opensre integrations setup ``` Then verify connected tools: ```bash theme={null} opensre integrations verify ``` ## 3. Ask the agent Chat with it in the shell: ```bash theme={null} opensre ``` Ask: *why is checkout latency spiking?* ## Next * [Quickstart](/docs/getting-started/quickstart) — full walkthrough with screenshots * [Install locally](/docs/install/install-local) — shared VM / on-prem notes * [Integrations overview](/docs/integrations) — tool catalog # macOS Source: https://opensre.com/docs/install/macos Install OpenSRE on a Mac and ask your first question. Install OpenSRE on macOS, sign in to your account, then ask the agent your first question. ## 1. Install ```bash theme={null} curl -fsSL https://install.opensre.com | bash -s -- -dc ``` You usually don't need `sudo`. If the terminal says `opensre: command not found`, open a new window or add the path the installer printed (often `~/.local/bin`). ## 2. Set things up ```bash theme={null} opensre ``` Create an OpenSRE account or sign in to activate the hosted model. The first prompt offers a demo; pick the CI/CD analysis of a repository on this machine, or choose **E** to skip it. Add tools later, when you need them: ```bash theme={null} opensre integrations setup ``` Then check connected tools: ```bash theme={null} opensre integrations verify ``` ## 3. Ask the agent Open the shell and describe what's going on in plain English: ```bash theme={null} opensre ``` Ask: *why is checkout latency spiking?* ## Next * [Quickstart](/docs/getting-started/quickstart) — same flow with screenshots * [Install locally](/docs/install/install-local) — PATH notes, on-prem tips * [Integrations overview](/docs/integrations) — connect more tools # Sync to your own object store Source: https://opensre.com/docs/install/remote-sync Copy sessions and memory between machines using an S3, GCS, Azure, or Vercel Blob store you own. By default OpenSRE keeps everything on your machine. Turn on remote sync to copy **conversations** and **memory** to a private store you own, so another laptop can continue where you left off. Credentials and API keys stay local. You still run integration setup on each new machine. | Copied | Not copied | | - | - | | `~/.opensre/sessions/` | Integration credentials | | `~/.opensre/memory/` | Model API keys | > [!WARNING] > > Use a **private** store. It can hold incident chats and system details. ## Setup ```bash theme={null} # Interactive (writes ~/.opensre/config.yml) opensre remote-sync setup # AWS / S3 opensre remote-sync setup --provider aws --bucket my-opensre-bucket # Google Cloud Storage (Application Default Credentials) opensre remote-sync setup --provider gcs --bucket my-opensre-bucket # Vercel Blob opensre remote-sync setup --provider vercel --bucket opensre-remote-sync export BLOB_READ_WRITE_TOKEN=vercel_blob_rw_... # Azure Blob (ambient Azure credentials) opensre remote-sync setup --provider azure --bucket my-container ``` Same flags work as `/remote-sync setup` in the interactive shell. On chat gateways, pass flags (no interactive prompts) — or use the CLI. Or set env vars yourself: ```bash theme={null} export OPENSRE_REMOTE_SYNC=1 export OPENSRE_REMOTE_SYNC_BUCKET=my-opensre-bucket # optional: # export OPENSRE_REMOTE_SYNC_PROVIDER=aws # default; also gcs, vercel, azure # export OPENSRE_REMOTE_SYNC_PREFIX=opensre # default ``` A bucket name alone does nothing — you must also set `OPENSRE_REMOTE_SYNC=1` (or enable it via `opensre remote-sync setup`). | Variable | What it does | | - | - | | `OPENSRE_REMOTE_SYNC` | `1` / `true` turns sync on | | `OPENSRE_REMOTE_SYNC_PROVIDER` | `aws` (default), `gcs`, `vercel`, `azure`, or `s3compat` | | `OPENSRE_REMOTE_SYNC_BUCKET` | Bucket, container, or Blob store name (required when on) | | `OPENSRE_REMOTE_SYNC_PREFIX` | Key prefix (default `opensre`). Same prefix = shared history | | `OPENSRE_REMOTE_SYNC_REGION` | AWS region (AWS only) | | `OPENSRE_REMOTE_SYNC_PROFILE` | AWS profile name, or Azure storage account name when `provider=azure` | | `OPENSRE_REMOTE_SYNC_ENDPOINT_URL` | Custom S3 endpoint when `provider=s3compat` (MinIO / R2 / Spaces) | | `BLOB_READ_WRITE_TOKEN` | Vercel Blob token (Vercel only) | | `OPENSRE_REMOTE_SYNC_EXCLUDE` | Comma-separated paths to skip (see below) | | `OPENSRE_REMOTE_SYNC_EXCLUDE_OFF` | `1` ignores exclusions for one run | Env vars override `~/.opensre/config.yml` for that run. `--region` / `--profile` only work with AWS and Azure (profile = storage account); setup rejects them for other providers. Every machine that should share history needs the **same provider, store, and prefix** (and for Vercel, a token for that store). ## Sync ```bash theme={null} opensre remote-sync status # what would move opensre remote-sync sync # pull, then push opensre remote-sync sync --pull-only # new machine: download only opensre remote-sync sync --push-only # upload only ``` In the shell: `/remote-sync status` and `/remote-sync sync`. To run one full sync automatically after an interactive shell session exits: ```bash theme={null} opensre --sync-on-exit ``` Run setup first. The flag applies only to that invocation; a normal `opensre` session remains manual-only. If the automatic sync fails, OpenSRE prints a warning without changing the shell's exit status. > [!WARNING] > > Remote sync is for a **personal machine**. Org-bound Slack/Discord turns are refused — those already persist under the org context root. ## Providers **AWS / S3** — private bucket. Uses your normal AWS credentials (`AWS_*`, profile, or SSO). Uploads use server-side encryption (`AES256`). Needs `s3:ListBucket`, `s3:GetObject`, and `s3:PutObject` on your prefix. Sync never deletes objects, so `s3:DeleteObject` is not required. **Google Cloud Storage** — private bucket. Set `OPENSRE_REMOTE_SYNC_PROVIDER=gcs`. Credentials come from Application Default Credentials (`gcloud auth application-default login`). Needs `storage.objects.list`, `storage.objects.get`, `storage.objects.create`, and `storage.objects.delete` (delete is required to replace an object even though sync never removes keys). A bucket-scoped `roles/storage.objectUser` binding covers all four. **Vercel Blob** — private store. Set `BLOB_READ_WRITE_TOKEN`. `OPENSRE_REMOTE_SYNC_BUCKET` is the store name/id for status. **Azure Blob Storage** — private container. Set `OPENSRE_REMOTE_SYNC_PROVIDER=azure`, `OPENSRE_REMOTE_SYNC_BUCKET` to the container name, and `OPENSRE_REMOTE_SYNC_PROFILE` to the storage account name. Auth uses ambient Azure credentials (`az login`, Managed Identity, or `AZURE_*`). The principal needs **Storage Blob Data Contributor** on the account (plain Contributor is not enough). **S3-compatible (MinIO / R2 / Spaces)** — set `OPENSRE_REMOTE_SYNC_PROVIDER=s3compat` and `OPENSRE_REMOTE_SYNC_ENDPOINT_URL` to the custom endpoint (or `AWS_ENDPOINT_URL_S3`). Path-style addressing is enabled for MinIO. Use the same ambient S3 credentials as AWS. ## Skip some paths By default every session and memory file syncs. To hold some back: ```bash theme={null} export OPENSRE_REMOTE_SYNC_EXCLUDE='*.tmp,sessions/scratch-*' ``` Or in `~/.opensre/config.yml`: ```yaml theme={null} remote_sync: enabled: true bucket: my-opensre-bucket exclude: - "*.tmp" - sessions/scratch-* ``` If the env var is set, it **replaces** the config list for that run (it does not merge). Clearing the env var to empty does **not** turn exclusions off — use this instead: ```bash theme={null} OPENSRE_REMOTE_SYNC_EXCLUDE_OFF=1 opensre remote-sync sync ``` Patterns match keys like `sessions/…` or `memory/…` (case-sensitive). `*` can cross folders. Excluded paths are skipped both ways (not uploaded, not downloaded). Patterns starting with `!` are rejected. Sync never deletes from the store. ## Conflicts Sync pulls first, then pushes. * Only on one side → copied to the other * Both changed → newer write wins * Nothing is deleted on either side Deleting a session on one laptop does **not** remove it from the store or other machines. Delete the object in the store if you want it gone everywhere. # Windows Source: https://opensre.com/docs/install/windows-local Install OpenSRE on Windows with the PowerShell installer. Open PowerShell. The installer does not need an elevated session. ## 1. Install and start OpenSRE ```powershell theme={null} & ([scriptblock]::Create((irm https://install.opensre.com/install.ps1))) -dc ``` ```powershell theme={null} opensre ``` Create an OpenSRE account or sign in to activate the hosted model. The first prompt offers a demo; pick the CI/CD analysis of a repository on this machine, or choose **E** to skip it. The installer puts `opensre.exe` in `%USERPROFILE%\.local\bin`. If `opensre` is not found, follow the PATH instructions printed by the installer or open a new terminal. ## 2. Connect tools ```bash theme={null} opensre integrations setup opensre integrations verify ``` ## 3. Ask the agent Ask in the interactive shell: *why is checkout latency spiking?* ## Next * [Quickstart](/docs/getting-started/quickstart) — install, sign in, and ask your first question * [Install locally](/docs/install/install-local) — PATH and account troubleshooting * [Integrations overview](/docs/integrations) — connect more tools # Argo CD Source: https://opensre.com/docs/integrations/cloud/argocd Connect Argo CD so OpenSRE can inspect GitOps application health, sync state, revisions, and drift ## Overview OpenSRE queries the Argo CD REST API as a read-only evidence source when you ask about GitOps deployments. It can list visible applications, inspect one application's sync and health status, and fetch sanitized server-side diff output to show deployment drift. ## Prerequisites * Argo CD API server reachable from the machine running OpenSRE * A dedicated Argo CD account or API token with read access to the applications you want OpenSRE to inspect * The Argo CD base URL, for example `https://argocd.example.com` ## Setup Argo CD is configured through environment variables or the persistent integration store. There is no dedicated `opensre integrations setup argocd` wizard today. ### Option 1: Environment variables Add one authentication method to your `.env`: ```bash theme={null} ARGOCD_BASE_URL=https://argocd.example.com # Option A: API token auth. The token may be set with or without a "Bearer " prefix. ARGOCD_AUTH_TOKEN=*** # ARGOCD_TOKEN=*** # alias also supported # Option B: username/password auth. Use instead of ARGOCD_AUTH_TOKEN. # ARGOCD_USERNAME=opensre-readonly # ARGOCD_PASSWORD=*** # Optional scoping and TLS settings ARGOCD_PROJECT=default ARGOCD_APP_NAMESPACE=argocd ARGOCD_VERIFY_SSL=true ``` | Variable | Default | Description | | - | - | - | | `ARGOCD_BASE_URL` | — | **Required.** Argo CD API base URL. Remote URLs must use `https://`; plain `http://` is accepted only for loopback or localhost development URLs. | | `ARGOCD_AUTH_TOKEN` | — | Argo CD bearer/API token. Use this or username/password, not both. | | `ARGOCD_TOKEN` | — | Alias for `ARGOCD_AUTH_TOKEN`. | | `ARGOCD_USERNAME` | — | Username for Argo CD session login. Use together with `ARGOCD_PASSWORD`. | | `ARGOCD_PASSWORD` | — | Password for Argo CD session login. Use together with `ARGOCD_USERNAME`. | | `ARGOCD_PROJECT` | — | Optional Argo CD project filter for listing and application-specific requests. | | `ARGOCD_APP_NAMESPACE` | — | Optional application namespace passed as `appNamespace` for application-specific requests. | | `ARGOCD_VERIFY_SSL` | `true` | Whether to verify TLS certificates. Set to `false` only for trusted local or lab environments. | OpenSRE rejects ambiguous auth configuration. Do not set a bearer token and username/password at the same time. ### Option 2: Persistent store You can also add Argo CD to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "argocd-prod", "service": "argocd", "status": "active", "credentials": { "base_url": "https://argocd.example.com", "bearer_token": "***", "project": "default", "app_namespace": "argocd", "verify_ssl": true } } ] } ``` The store also accepts `auth_token` or `token` as aliases for `bearer_token`. For username/password auth, omit `bearer_token` and set `username` and `password` instead. ### Option 3: Multiple Argo CD instances For multiple Argo CD instances, set `ARGOCD_INSTANCES` to a JSON array. The first valid instance is used as the default integration. ```bash theme={null} export ARGOCD_INSTANCES='[ { "name": "prod", "tags": {"env": "prod"}, "credentials": { "base_url": "https://argocd.prod.example.com", "bearer_token": "***", "project": "default" } }, { "name": "staging", "tags": {"env": "staging"}, "base_url": "https://argocd.staging.example.com", "username": "opensre-readonly", "password": "***" } ]' ``` When `ARGOCD_INSTANCES` is set, the single-instance `ARGOCD_BASE_URL` and auth variables are ignored for this service. `opensre integrations verify argocd` validates the resolved default instance. ## Credentials 1. In Argo CD, create a dedicated **read-only** account or API token for OpenSRE. 2. Grant that identity list/get access to the applications (and projects) you want investigated. 3. Set either: * `ARGOCD_AUTH_TOKEN` / `ARGOCD_TOKEN` (or store `bearer_token`), **or** * `ARGOCD_USERNAME` + `ARGOCD_PASSWORD` (or store `username` / `password`) 4. Set `ARGOCD_BASE_URL` to your API base (HTTPS for remote hosts). ## Tools | Tool / evidence | What it does | | - | - | | `argocd_application_status` | With `application_name`, returns a compact summary: sync status, health status, current revision, operation phase/message, destination, images, and recent deployment history. Without `application_name`, lists visible applications (optionally scoped by `ARGOCD_PROJECT`). | | `argocd_application_diff` | Server-side diff for one application (requires `application_name`). Returns `drift_detected`, `diff_count`, and sanitized diff records for Kubernetes objects whose live state differs from GitOps. | Once Argo CD is configured, start `opensre` and name the application in your question, e.g. *"Is checkout-api out of sync? What drifted?"* ### Local verification recipe Verified both registered tools live against a real local Argo CD install with a real GitOps application synced from a public repo. ```bash theme={null} if kind get clusters 2>/dev/null | grep -qx opensre-argocd-demo; then read -r -p "A kind cluster named opensre-argocd-demo already exists. Delete it and continue? [y/N] " confirm [ "$confirm" = "y" ] || [ "$confirm" = "Y" ] || { echo "Aborted -- rename the recipe's cluster or clean up the existing one first." >&2; exit 1; } kind delete cluster --name opensre-argocd-demo fi kind create cluster --name opensre-argocd-demo kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml \ --server-side --force-conflicts kubectl -n argocd wait --for=condition=Available deployment/argocd-server --timeout=180s kubectl -n argocd wait --for=condition=Available deployment/argocd-repo-server --timeout=180s ``` > [!WARNING] > > The stock `install.yaml` fails under a plain `kubectl apply` with `metadata.annotations: > Too long: may not be more than 262144 bytes` -- the ApplicationSet CRD is large enough > that client-side apply's `kubectl.kubernetes.io/last-applied-configuration` annotation > exceeds Kubernetes' own annotation size limit. `--server-side --force-conflicts` avoids > storing that annotation entirely. ```bash theme={null} kubectl -n argocd port-forward svc/argocd-server 8080:443 & ARGOCD_PORT_FORWARD_PID=$! trap 'kill "$ARGOCD_PORT_FORWARD_PID" 2>/dev/null' EXIT i=0 until curl -sk -o /dev/null https://localhost:8080/api/v1/session; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: port-forward never became ready" >&2; exit 1; } sleep 1 done ARGOCD_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d) ARGOCD_AUTH_TOKEN=$(curl -sk -X POST https://localhost:8080/api/v1/session \ -H "Content-Type: application/json" \ -d "{\"username\":\"admin\",\"password\":\"${ARGOCD_PWD}\"}" \ | python3 -c "import sys,json; print(json.load(sys.stdin)['token'])") export ARGOCD_BASE_URL="https://localhost:8080" export ARGOCD_AUTH_TOKEN export ARGOCD_VERIFY_SSL=false ``` Create a real Application synced from Argo CD's own public example repo: ```bash theme={null} cat > /tmp/argocd-app.yaml << 'YAML' apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: guestbook namespace: argocd spec: project: default source: repoURL: https://github.com/argoproj/argocd-example-apps.git targetRevision: HEAD path: guestbook destination: server: https://kubernetes.default.svc namespace: demo-app syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true YAML kubectl apply -f /tmp/argocd-app.yaml i=0 until kubectl -n argocd get application guestbook -o jsonpath='{.status.health.status}' 2>/dev/null | grep -q "Healthy"; do i=$((i + 1)) [ "$i" -ge 60 ] && { echo "ERROR: guestbook never became healthy" >&2; exit 1; } sleep 5 done ``` Verify: ```bash theme={null} opensre integrations verify argocd ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL argocd │ local env │ ✓ passed │ Connected to Argo CD and listed 1 application. ``` Chat sessions (unlike `opensre integrations verify`) only fall through to env vars when the store has no records at all -- any existing record, for any service, blocks env-var resolution entirely. Point `OPENSRE_INTEGRATIONS_STORE_PATH` at a path inside a fresh empty directory instead, so your real config is never read or written and the `ARGOCD_*` vars above are the only source of connection info: ```bash theme={null} export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-argocd-demo.XXXXXX)" export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json" ``` `argocd_application_diff` needs real drift to return anything interesting -- with `selfHeal: true` set above, Argo CD reverts live-cluster drift within seconds, so disable automated sync first to introduce a drift that actually sticks: ```bash theme={null} kubectl -n argocd patch application guestbook --type merge -p '{"spec":{"syncPolicy":{"automated":null}}}' kubectl -n demo-app patch deployment guestbook-ui -p '{"spec":{"replicas":3}}' ``` Now ask the agent about the drift: ```bash theme={null} opensre ``` Ask: *Is the guestbook Argo CD application out of sync? What drifted?* The agent decides which tools to call on a given turn, so `argocd_application_diff` isn't guaranteed to run every time -- on a turn where it did, it correctly reported the drift introduced above: `guestbook-ui`'s replica count changed from the desired `1` to the live `3`. Both registered tools return real data through this same chat flow -- `argocd_application_status` for sync/health status, revision, and operation phase, and `argocd_application_diff` for per-resource drift when the agent calls it. Teardown: ```bash theme={null} kill "$ARGOCD_PORT_FORWARD_PID" 2>/dev/null kind delete cluster --name opensre-argocd-demo rm -rf "$OPENSRE_DEMO_STORE_DIR" unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH ARGOCD_BASE_URL ARGOCD_AUTH_TOKEN ARGOCD_VERIFY_SSL ARGOCD_PORT_FORWARD_PID ``` ## Verify ```bash theme={null} opensre integrations verify argocd ``` Expected output: ```text theme={null} Service: argocd Status: passed Detail: Connected to Argo CD and listed 3 applications. ``` Verification performs a read-only application list call. It proves OpenSRE can reach Argo CD and list visible applications with the configured credentials; it does not write to Argo CD or sync applications. ## Troubleshooting | Symptom | Fix | | - | - | | **Status: missing** | Set `ARGOCD_BASE_URL` and exactly one auth method, or add an active `argocd` store entry. | | **Remote `http://` URL rejected** | Use `https://` for remote Argo CD. Use plain HTTP only for `localhost`, `127.0.0.1`, or `::1` development endpoints. | | **401 Unauthorized** | Check the token, or verify that username/password login can create an Argo CD session. | | **403 Forbidden** | Ensure the account can list applications and read the target application. | | **SSL error** | Fix the certificate chain or, for a trusted lab only, set `ARGOCD_VERIFY_SSL=false`. | | **No diff evidence** | The diff tool requires an application name — name the application in your question. | | **Application list succeeds but a named app fails** | Check `ARGOCD_PROJECT` and `ARGOCD_APP_NAMESPACE`, and confirm the account has access to that application. | ## Security * Use a dedicated read-only Argo CD account or token for OpenSRE. * Store credentials in `.env` or `~/.opensre/integrations.json`, not in source code. * Use `https://` for remote Argo CD URLs. Plain `http://` is accepted only for loopback or localhost development URLs. * Do not disable `ARGOCD_VERIFY_SSL` for production instances. * OpenSRE redacts bearer tokens, passwords, token-like strings, and Kubernetes `Secret` diffs before surfacing Argo CD errors or diff evidence. * The integration is read-only: it lists applications, reads application summaries, and reads server-side diff data. It does not sync, modify, or delete Argo CD resources. # AWS Source: https://opensre.com/docs/integrations/cloud/aws Connect AWS so OpenSRE can map your infrastructure and investigate cloud-related issues ## Overview OpenSRE uses AWS to map your environment: Lambda functions, EKS clusters, S3 buckets, and more. It reads infrastructure state to build context when you ask about cloud resources. Related AWS tool pages (same credentials, no separate setup): [EC2](/docs/integrations/cloud/ec2), [ELB](/docs/integrations/cloud/elb), [S3](/docs/integrations/cloud/s3), [Lambda](/docs/integrations/cloud/aws_lambda), and [CloudTrail](/docs/integrations/cloud/cloudtrail_events). ## Prerequisites * AWS account with IAM permissions * Either a role ARN (recommended) or static access keys. A role ARN is *assumed* using your ambient AWS credentials — environment keys, an `aws configure` profile, or an attached instance/task role — so one of those must exist on the machine running OpenSRE ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup aws ``` Provide a role ARN or static access keys when prompted. If you pick a role and this machine has no base AWS credentials, setup offers to switch to access keys, pause so you can run `aws configure`, or continue anyway (for EC2/ECS/Lambda with an attached role). ### Option 2: Environment variables (IAM role) ```bash theme={null} AWS_ROLE_ARN=arn:aws:iam::123456789012:role/OpenSREReadOnly AWS_EXTERNAL_ID=your-external-id # optional AWS_REGION=us-east-1 ``` ### Option 3: Environment variables (static keys) ```bash theme={null} AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY AWS_SESSION_TOKEN=... # optional, for temporary credentials AWS_REGION=us-east-1 ``` | Variable | Default | Description | | - | - | - | | `AWS_ROLE_ARN` | — | IAM role to assume (recommended) | | `AWS_EXTERNAL_ID` | — | External ID for role assumption | | `AWS_REGION` | `us-east-1` | AWS region | | `AWS_ACCESS_KEY_ID` | — | Static access key (if not using role) | | `AWS_SECRET_ACCESS_KEY` | — | Static secret key | | `AWS_SESSION_TOKEN` | — | Session token for temporary credentials | > [!NOTE] > > Either `AWS_ROLE_ARN` or `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` is required. > `AWS_ROLE_ARN` alone is not enough on a laptop: the role is assumed with whatever > credentials boto3 finds (env keys, `~/.aws/credentials`, or an instance/task role), and > that identity needs `sts:AssumeRole` on the role. For multiple AWS accounts or regions, use [multi-instance](/docs/platform/multi-instance-integrations) with `AWS_INSTANCES`. ### Option 4: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "aws-prod", "service": "aws", "status": "active", "credentials": { "role_arn": "arn:aws:iam::123456789012:role/OpenSREReadOnly", "external_id": "your-external-id", "region": "us-east-1" } } ] } ``` ## Credentials ### IAM role (recommended) 1. Create an IAM role that OpenSRE can assume (or attach to the host). 2. Attach read-only permissions (see below). 3. Set `AWS_ROLE_ARN` (and optional `AWS_EXTERNAL_ID`) or enter the ARN in `opensre integrations setup aws`. ### Static access keys 1. Create an IAM user with the same read-only permissions. 2. Create an access key and set `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (and `AWS_SESSION_TOKEN` if temporary). ### IAM permissions OpenSRE needs read-only access. Attach the AWS managed `ReadOnlyAccess` policy, or a custom policy scoped to the services you want OpenSRE to inspect. For least privilege, the minimum services used are: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:GetCallerIdentity", "ec2:Describe*", "ecs:Describe*", "ecs:List*", "eks:Describe*", "eks:List*", "lambda:List*", "lambda:Get*", "s3:ListBucket", "s3:GetObject", "logs:FilterLogEvents", "logs:GetLogEvents", "cloudwatch:GetMetricData", "cloudwatch:ListMetrics", "cloudtrail:LookupEvents", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DescribeTargetHealth" ], "Resource": "*" } ] } ``` > [!NOTE] > > `opensre integrations verify aws` assumes `AWS_ROLE_ARN` when set. Many AWS tools (S3, CloudTrail, EC2, ELB, Lambda) call AWS through the ambient credential chain (environment keys, shared profile, or instance/task role) and do **not** assume that role for the API call. Give S3/CloudTrail/EC2 permissions to the identity the OpenSRE process actually runs as. ## Tools | Area | Tools / capability | | - | - | | Core AWS | `execute_aws_operation` (manual / advanced; not auto-planned) | | EC2 | `ec2_instances_by_tag` — see [EC2](/docs/integrations/cloud/ec2) | | ELB | `get_elb_target_health` — see [ELB](/docs/integrations/cloud/elb) | | S3 | `list_s3_objects`, `inspect_s3_object`, `get_s3_object`, `check_s3_marker` — see [S3](/docs/integrations/cloud/s3) | | Lambda | `get_lambda_configuration`, `inspect_lambda_function`, `get_lambda_invocation_logs`, `get_lambda_errors` — see [Lambda](/docs/integrations/cloud/aws_lambda) | | CloudTrail | `lookup_cloudtrail_events` — see [CloudTrail](/docs/integrations/cloud/cloudtrail_events) | ## Verify ```bash theme={null} opensre integrations verify aws ``` Expected output (role example): ``` Service: aws Status: passed Detail: Authenticated via assume-role in us-east-1 as arn:aws:iam::123456789012:role/OpenSREReadOnly (account 123456789012) ``` Expected output (static keys example): ``` Service: aws Status: passed Detail: Connected to AWS STS via static-creds in us-east-1; caller identity account=123456789012 arn=arn:aws:iam::123456789012:user/opensre. ``` Inside the REPL: `/integrations verify aws` or `/verify aws`. Aliases such as `eks` also map to this check. ## Local verification recipe A short end-to-end recipe to bring AWS up locally, confirm the verifier passes, exercise `execute_aws_operation` against real data, and tear everything down. This uses temporary credentials from an existing AWS account or SSO session — no long-lived infrastructure to provision. ### Bring it up (credentials) If you authenticate with AWS SSO (`aws sso login`), the credentials are not exposed as environment variables by default. Export them into the variables OpenSRE's env resolver reads: ```powershell theme={null} # PowerShell (Windows) $creds = aws configure export-credentials --format process | ConvertFrom-Json $env:AWS_ACCESS_KEY_ID = $creds.AccessKeyId $env:AWS_SECRET_ACCESS_KEY = $creds.SecretAccessKey $env:AWS_SESSION_TOKEN = $creds.SessionToken $env:AWS_REGION = "us-east-1" ``` ```bash theme={null} # bash / zsh (macOS / Linux) eval "$(aws configure export-credentials --format env)" export AWS_REGION=us-east-1 ``` Static access keys or a role ARN work too — see [Setup](#setup) above. ### Verify ```bash theme={null} uv run opensre integrations verify aws ``` Expected: ``` SERVICE │ SOURCE │ STATUS │ DETAIL ━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ aws │ local env │ ✓ passed │ Connected to AWS STS via static-creds │ │ │ in us-east-1; caller identity │ │ │ account= arn=. ``` ### Exercise `execute_aws_operation` `execute_aws_operation` is the registered AWS tool. It is read-only: the allowlist admits `describe_*`, `get_*`, `list_*`, `head_*`, `select_*`, `batch_get_*`, `lookup_*` (e.g. `cloudtrail.lookup_events`), and the exact operations `query` and `scan` (e.g. `dynamodb.query`). Mutating operations (e.g. `terminate_instances`) are rejected with an `"Operation not allowed"` error. The agent needs an explicit service and operation to call it, so to exercise it directly, look it up in the tool registry and run it. Any region works — set `AWS_REGION`, or pass a region-scoped service; `us-east-1` is used here as the example: ```bash theme={null} uv run python -c "from tools.registry import get_registered_tool; import json; tool = get_registered_tool('execute_aws_operation'); print(json.dumps(tool.run(service='sts', operation='get_caller_identity'), indent=2, default=str))" ``` A successful call returns a payload of this shape (redacted): ```json theme={null} { "found": true, "service": "sts", "operation": "get_caller_identity", "result": { "UserId": "", "Account": "", "Arn": "arn:aws:sts:::assumed-role//" }, "metadata": {"region": "us-east-1", "parameters_provided": false} } ``` Other read-only operations confirmed against a live account (output trimmed and redacted): * `ec2` / `describe_instances` — returned a running `t2.micro` in `us-east-1a` with full metadata. * `s3` / `list_buckets` — returned the account's real buckets. * `iam` / `list_roles` with `parameters={"MaxItems": 5}` — returned real IAM roles; confirms parameters pass through to the API. * `ecs` / `list_clusters` — returned `{"clusterArns": []}`, a valid empty result (`"found": true`), not a stub or error. Because AWS spans many regions, target another region either by exporting a different `AWS_REGION` or by passing a region-scoped `service`/`parameters` combination; global services (e.g. `iam`) report `"region": "aws-global"` in the metadata. ### Teardown There is no cloud infrastructure to remove. What you clean up depends on how you configured AWS. **If you only exported credentials for this session** (the SSO / env-var path above), clear them from your shell: ```powershell theme={null} # PowerShell Remove-Item Env:AWS_ACCESS_KEY_ID, Env:AWS_SECRET_ACCESS_KEY, Env:AWS_SESSION_TOKEN, Env:AWS_REGION ``` ```bash theme={null} # bash / zsh unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_REGION ``` **If you ran `opensre integrations setup aws`**, that writes to three tiers — the integration store, the project `.env` (non-secret fields), and an owner-only local secret file at `~/.opensre/credentials.json` (secret fields such as `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN`). `opensre integrations remove aws` clears only the store entry, so clear the other two tiers too or the credentials stay resolvable: 1. Remove the store entry: ```bash theme={null} opensre integrations remove aws ``` 2. Delete the non-secret `AWS_*` lines (`AWS_ROLE_ARN`, `AWS_EXTERNAL_ID`, `AWS_REGION`, `AWS_ACCESS_KEY_ID`) from your project `.env`. 3. Delete the secret entries from the local secret store. OpenSRE stores secrets in `~/.opensre/credentials.json`, not the OS keychain. Remove the AWS keys from it: ```bash theme={null} uv run python -c "from config.secrets.store import delete_secret; [delete_secret(k) for k in ('AWS_SECRET_ACCESS_KEY', 'AWS_SESSION_TOKEN')]" ``` `delete_secret` removes each key from every local tier. Deleting `~/.opensre/credentials.json` outright also works if it holds no other integration's secrets. ## Troubleshooting | Symptom | Fix | | - | - | | **Unable to locate credentials** (role ARN setup) | You chose IAM Role ARN but nothing on this machine has base AWS credentials. The setup wizard detects this when you pick the role option and offers to switch to Access Key + Secret, or to pause while you run `aws configure` / export `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` for an identity allowed to assume the role. If you set `AWS_ROLE_ARN` by hand instead of through the wizard, provide one of those base credentials | | **AccessDenied on STS** | Ensure the caller has `sts:AssumeRole` permission on the target role | | **InvalidClientTokenId** | Check that `AWS_ACCESS_KEY_ID` is correct and the key is active | | **Could not connect to endpoint** | Check `AWS_REGION` and network connectivity | | **ExpiredTokenException** | Refresh your session token or rotate the access key | | **Verify passes but S3/CloudTrail/EC2 tools fail** | Those tools use the ambient credential chain, not only the role assumed during verify — attach the needed IAM actions to that identity | ## Security * Prefer **IAM roles** over static keys wherever possible. * Scope IAM permissions to only the AWS services OpenSRE needs to inspect. * Rotate static access keys regularly. * Enable CloudTrail so all OpenSRE API calls are auditable. * Store keys in `.env` or your secret manager — not in source control. # AWS Lambda Source: https://opensre.com/docs/integrations/cloud/aws_lambda Connect AWS Lambda so OpenSRE can inspect function configuration, invocation logs, and runtime errors. ## Overview OpenSRE integrates with AWS Lambda to inspect function configuration, retrieve recent invocation logs from CloudWatch, and investigate runtime failures during incident response. All operations are read-only. ## Prerequisites * An AWS account with AWS Lambda functions * AWS credentials configured for the runtime (see [AWS](/docs/integrations/cloud/aws)) * Permission to access Lambda and CloudWatch Logs ## Setup Lambda has no separate setup target. Configure the [AWS integration](/docs/integrations/cloud/aws): ```bash theme={null} opensre integrations setup aws ``` ### Environment variables Add the following to your `.env` (ambient credential chain used by Lambda tools): ```bash theme={null} AWS_ACCESS_KEY_ID=your-access-key AWS_SECRET_ACCESS_KEY=your-secret-key AWS_REGION=us-east-1 AWS_SESSION_TOKEN=your-session-token # optional ``` | Variable | Required | Description | | - | - | - | | `AWS_ACCESS_KEY_ID` | Yes\* | AWS access key ID | | `AWS_SECRET_ACCESS_KEY` | Yes\* | AWS secret access key | | `AWS_REGION` | No | AWS region (defaults to `us-east-1`) | | `AWS_SESSION_TOKEN` | No | Session token when using temporary credentials | \*Or use an instance/task role / shared profile instead of static keys. There are no Lambda-specific environment variables. ## Credentials Lambda tools build clients through the ambient boto3 credential chain. They do **not** assume `AWS_ROLE_ARN` for API calls — that role is used by `opensre integrations verify aws` when set. Give the running identity permission to: * Call Lambda read APIs (`lambda:List*`, `lambda:Get*`, and related describe/get actions your account uses) * Read CloudWatch Logs (`logs:FilterLogEvents`, `logs:GetLogEvents`) for invocation history If you already use the AWS managed `ReadOnlyAccess` policy on that identity, both are covered. See also the least-privilege example on the [AWS](/docs/integrations/cloud/aws) page. ## Tools | Tool | What it does | | - | - | | `get_lambda_configuration` | Inspect runtime, handler, timeout, memory, IAM role, environment variables, and deployed version | | `inspect_lambda_function` | Retrieve deployment package metadata and inspect function contents when available | | `get_lambda_invocation_logs` | Retrieve recent invocation logs from CloudWatch Logs | | `get_lambda_errors` | Investigate recent runtime failures and execution errors | ### Gotcha Lambda invocation history is retrieved from **CloudWatch Logs**. Ensure the configured AWS credentials have permission to access both **AWS Lambda** and **CloudWatch Logs**, otherwise log retrieval may fail even if Lambda access succeeds. ## Verify ```bash theme={null} opensre integrations verify aws ``` There is no separate `aws_lambda` verify target. A successful AWS verify confirms the account credentials OpenSRE uses; Lambda tools then use the ambient credential chain when called. Expected output (example): ```text theme={null} Service: aws Status: passed Detail: Connected to AWS STS via static-creds in us-east-1; caller identity account=123456789012 arn=arn:aws:iam::123456789012:user/opensre. ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Lambda tools fail after verify passes** | Verify assumes `AWS_ROLE_ARN` when set; Lambda tools use the ambient chain — attach Lambda and Logs permissions to that identity | | **Configuration works but logs/errors empty** | Grant CloudWatch Logs read access (`logs:FilterLogEvents`, `logs:GetLogEvents`) | | **Wrong region** | Set `AWS_REGION` to the region where the function is deployed | | **AccessDenied on Lambda APIs** | Attach `lambda:Get*` / `lambda:List*` (or broader read-only) to the ambient identity | ## Security * All Lambda tools are read-only. * Prefer IAM roles over long-lived static keys. * Scope permissions to the functions and log groups you want OpenSRE to inspect. * Store credentials in `.env` or your secret manager — not in source control. # AWS CloudTrail Source: https://opensre.com/docs/integrations/cloud/cloudtrail_events Trace AWS configuration-change causality — who changed what, and when — during incidents ## Overview OpenSRE uses AWS CloudTrail to answer the first question of every cloud post-mortem: **"who changed what, and when?"** When an AWS alert fires, the planner can look up recent management events — IAM changes, security-group mutations, EKS/Lambda config updates, and resource deletions — scoped to a resource, a principal, or a time window. CloudTrail lookups are read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot mutate any resources. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/integrations/cloud/aws) (role ARN recommended for verify) — CloudTrail reuses the same account credentials and region, so no extra setup is needed * IAM permission for the single read-only CloudTrail action listed below ## Setup CloudTrail is account-wide, so the tool becomes available to the planner whenever the [AWS integration](/docs/integrations/cloud/aws) is configured — there is nothing resource-specific to set up. The region comes from the AWS integration (or `AWS_REGION`, defaulting to `us-east-1`). ```bash theme={null} opensre integrations setup aws ``` ## Credentials ### IAM permissions The tool only needs one read-only CloudTrail action: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudtrail:LookupEvents" ], "Resource": "*" } ] } ``` Attach this policy to the same IAM role or user already configured for the [AWS integration](/docs/integrations/cloud/aws). If you are already using the AWS managed `ReadOnlyAccess` policy, this action is already covered. > [!NOTE] > > **Execution identity:** the AWS integration's `role_arn` / credentials gate *availability* and supply the region, but the lookup itself runs through boto3's standard credential chain (environment variables, shared config, or the host's instance role) — the configured role is **not** assumed for the call. Ensure the identity the OpenSRE process runs as can perform `cloudtrail:LookupEvents`. This matches the other AWS tools (RDS/EKS). ## Tools | Tool | AWS API call | What it returns | | - | - | - | | `lookup_cloudtrail_events` | `cloudtrail:LookupEvents` | Recent management events — event name, time, source, acting username, affected resources, AWS region, source IP, and any error code. | ### Parameters | Parameter | Default | Description | | - | - | - | | `resource_name` | — | Filter to events touching a specific resource name/ARN (the most specific filter). | | `event_source` | — | Filter by AWS service event source, e.g. `iam.amazonaws.com`, `ec2.amazonaws.com`. | | `username` | — | Filter by the acting principal / IAM username. | | `region` | `us-east-1` | AWS region to query. | | `duration_minutes` | `60` | Look-back window. CloudTrail retains 90 days of history (the upper bound). | | `max_results` | `50` | Maximum events to return (CloudTrail caps this at 50 per call). | | `next_token` | — | Pagination token from a previous truncated response; pass it to fetch the next page. | > [!NOTE] > > CloudTrail's `LookupEvents` API accepts **only one filter attribute per call**. When more than one filter is supplied, the tool sends the most specific one, in priority order: `resource_name` → `username` → `event_source`. With no filter, it returns recent account-wide events for the window. > [!NOTE] > > CloudTrail returns at most 50 events per page. When more matching events exist, the response sets **`truncated: true`** and returns a **`next_token`** — pass it back via the `next_token` parameter to fetch the next page, so a busy account or wide window never silently drops events. > [!NOTE] > > A single event can touch more resources than the transport layer returns inline (for example `CreateTags` across a fleet). When the affected-resources list for an event is trimmed, that event carries **`resources_truncated: true`**, signalling that the real blast radius is wider than the resources shown — so a large fan-out change is never silently understated. ### Use cases * Finding who modified an IAM policy, role, or security group just before an incident * Tracing configuration changes to a specific resource (by resource name) * Auditing every action taken by a principal (by username) * Reviewing recent activity from a single AWS service (by event source) * Establishing change causality at the start of a post-mortem ## Verify ```bash theme={null} opensre integrations verify aws ``` There is no separate `cloudtrail` verify target. CloudTrail availability is gated on the AWS integration; the lookup itself uses the ambient credential chain when called. ## Troubleshooting | Symptom | Fix | | - | - | | **AccessDenied on `cloudtrail:LookupEvents`** | Add the IAM policy above to the identity the OpenSRE process runs as (ambient chain), not only a role assumed during verify. | | **No events returned** | Widen `duration_minutes`, loosen the filter, or confirm you are querying the region where the activity occurred. Only management events are returned; data events are not. | | **ThrottlingException** | CloudTrail `LookupEvents` is rate-limited to two requests per second per account per region. Retry with a narrower window. | | **Tool reports the wrong region** | Set `AWS_REGION`, or check the `region` field on the configured AWS integration. | ## Security * CloudTrail lookups are read-only and cannot mutate resources. * Prefer least-privilege IAM with only `cloudtrail:LookupEvents` if you do not need broader AWS access. * Store AWS credentials in `.env` or your secret manager — not in source control. # AWS EC2 Source: https://opensre.com/docs/integrations/cloud/ec2 Discover EC2 instances by tag, tier, or VPC to map application tiers and investigate non-Kubernetes issues ## Overview OpenSRE uses AWS EC2 to discover virtual machine instances, filter them by tag (such as `tier`), instance IDs, or VPC, and group instances into application tiers (such as web or worker). This helps answer questions like "which application tier(s) plausibly drive load on this RDS?" when investigating non-Kubernetes alerts or bridging load balancers (ELB) and databases (RDS). All EC2 API calls are read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot mutate your EC2 resources. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/integrations/cloud/aws) (role ARN recommended for verify; ambient credentials for tool calls) * An EC2 environment with tagged instances, explicit instance IDs, or a VPC you want OpenSRE to inspect * IAM permissions for `ec2:DescribeInstances` ## Setup EC2 has no separate setup target. Configure the [AWS integration](/docs/integrations/cloud/aws), then ensure alerts resolve an `ec2` source with the fields the tool needs. ### Environment variables ```bash theme={null} AWS_REGION=us-east-1 ``` | Variable | Default | Description | | - | - | - | | `AWS_REGION` | `us-east-1` | AWS region the EC2 instances live in | The tool accepts parameter hints from the resolved `ec2` source: * `tiers`: List of tier names in the resolved `ec2` source (for example `["web"]` or `["web", "worker"]`). The parameter extractor maps a single item to the singular `tier` tool argument, or leaves `tier` empty when multiple tiers are in scope so instances across all tiers are enumerated and returned grouped in `by_tier`. * `instance_ids`: List of explicit EC2 instance IDs. * `vpc_id`: VPC ID filter to limit discovery to a specific network. * `region`: AWS region override (defaults to `AWS_REGION` or `us-east-1`). ## Credentials EC2 uses the same AWS account as the [AWS integration](/docs/integrations/cloud/aws). Tool calls use the ambient boto3 credential chain (environment keys, shared credentials profile, or attached instance/task role). They do **not** assume `AWS_ROLE_ARN` for the API call. ### IAM permissions The integration only needs read-only EC2 describe permissions: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstances" ], "Resource": "*" } ] } ``` Attach this policy to the identity the OpenSRE process runs as (and to the role used by `opensre integrations verify aws` if you use assume-role). If you already use the AWS managed `ReadOnlyAccess` policy, `ec2:DescribeInstances` is included. ## Tools | Tool | AWS API call | What it returns | | - | - | - | | `ec2_instances_by_tag` | `ec2:DescribeInstances` | Active EC2 instances filtered by `tier` tag, instance IDs, or `vpc_id`. Returns per-instance metadata (ID, tier, ASG, private IP, VPC, subnet, AZ, state, instance type, security groups), a `by_tier` mapping of tier name to instance IDs, `tiers_detected`, and a precomputed summary (counts by tier, total active instances, primary tier, VPCs in scope, AZs in scope). | The tool becomes available to the planner whenever `tiers`, `instance_ids`, `vpc_id`, or `_backend` is present in the resolved `ec2` sources. ### Use cases * Discovering EC2 application tiers when investigating a non-Kubernetes issue * Mapping a tier name (`web`, `worker`, etc.) to its active instance IDs * Bridging EC2 → RDS when answering "which application tier drives database load" ### Gotcha Terminated or stopped EC2 instances (`terminated`, `stopped`, `stopping`, `shutting-down`) are automatically excluded from the tool's results so they do not distort load-attribution or topology analysis. If an expected instance does not appear in the returned list, verify that the instance is in the `running` state and properly tagged with `tier` or `Tier`. Additionally, at least one filter (`tier`, `instance_ids`, or `vpc_id`) must be provided in the tool call or resolved sources, or the tool will return an unavailable status. ## Verify ```bash theme={null} opensre integrations verify aws ``` EC2 shares the AWS integration's credential check — there is no separate `ec2` verify target. A successful AWS verify confirms your AWS credentials are valid; the EC2 tool becomes available whenever `tiers`, `instance_ids`, `vpc_id`, or `_backend` is present in the resolved `ec2` source. Expected output: ``` Service: aws Status: passed Detail: Authenticated via assume-role in us-east-1 as arn:aws:iam::123456789012:role/OpenSREReadOnly (account 123456789012) ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Tool never selected** | Ensure the alert's resolved `ec2` source includes `tiers`, `instance_ids`, `vpc_id`, or `_backend` | | **Expected instance missing** | Confirm the instance is `running` and tagged with `tier` / `Tier`; stopped and terminated instances are excluded | | **AccessDenied on DescribeInstances** | Attach `ec2:DescribeInstances` to the ambient identity OpenSRE runs as | | **Wrong region** | Set `AWS_REGION` or pass `region` on the resolved source | ## Security * All EC2 API calls are read-only. * Prefer a least-privilege IAM policy with only `ec2:DescribeInstances` if you do not need broader AWS access. * Store AWS credentials in `.env` or your secret manager — not in source control. # AWS ELB Source: https://opensre.com/docs/integrations/cloud/elb Map a load balancer to its EC2 targets and surface unhealthy targets during incidents ## Overview OpenSRE uses AWS Elastic Load Balancing (ELB v2 — ALB/NLB) to bridge a public-facing alert (DNS/LB) down to the EC2 instances behind it. It maps target groups to registered targets and flags unhealthy or draining targets, pairing with the EC2 tools to identify which application tier is driving load on a downstream dependency. All ELB API calls are read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot mutate your load balancers or target groups. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/integrations/cloud/aws) (role ARN recommended for verify; ambient credentials for tool calls) * An ELB v2 (ALB or NLB) target group you want OpenSRE to investigate * IAM permissions for the two ELB describe actions listed below ## Setup ELB has no separate setup target. Configure the [AWS integration](/docs/integrations/cloud/aws), then ensure alerts resolve sources with target group or load balancer ARNs. ### Environment variables ```bash theme={null} AWS_REGION=us-east-1 ``` | Variable | Default | Description | | - | - | - | | `AWS_REGION` | `us-east-1` | AWS region the load balancer and target groups live in | The tool accepts `target_group_arns` (canonical), the singular `target_group_arn` alias, and `load_balancer_arn` as parameters. When invoked via resolved sources, it reads `target_group_arns` and `load_balancer_arns` from the resolved `ec2` source (using the first load balancer ARN). ## Credentials ELB uses the same AWS account as the [AWS integration](/docs/integrations/cloud/aws). Tool calls use the ambient boto3 credential chain (environment keys, shared credentials profile, or attached instance/task role). They do **not** assume `AWS_ROLE_ARN` for the API call. ### IAM permissions The integration only needs two read-only ELB v2 actions: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DescribeTargetHealth" ], "Resource": "*" } ] } ``` Attach this policy to the identity the OpenSRE process runs as (and to the role used by `opensre integrations verify aws` if you use assume-role). If you already use the AWS managed `ReadOnlyAccess` policy, both actions are covered. ## Tools | Tool | AWS API calls | What it returns | | - | - | - | | `get_elb_target_health` | `elasticloadbalancing:DescribeTargetGroups`, `elasticloadbalancing:DescribeTargetHealth` | Target groups in scope, healthy targets, unhealthy/draining targets, deduplicated EC2 instance IDs behind the load balancer, and a precomputed summary (total targets, healthy/unhealthy counts, healthy ratio %, unhealthy states, target group count). | The tool becomes available to the planner whenever `target_group_arns`, `target_group_arn`, or `load_balancer_arn` is present in the resolved `ec2` sources. ### Use cases * Mapping a target group ARN or load balancer ARN to the EC2 instance IDs serving traffic * Identifying unhealthy or draining targets correlated with a request-path alert * Bridging DNS → LB → EC2 when investigating a non-Kubernetes topology ### Gotcha Multi-target-group ALBs are common — a single ALB can front several listener rules, each pointing at a different target group. Pass the full `target_group_arns` list rather than a single `target_group_arn` if the load balancer has more than one target group, otherwise the tool only reports health for the one target group you named and unhealthy targets on the other groups will be silently missed. ## Verify ```bash theme={null} opensre integrations verify aws ``` ELB shares the AWS integration's connection check — there is no separate `elb` verify target. A successful AWS verify confirms credentials are valid; the ELB tool itself becomes available once a target group ARN or load balancer ARN is present in the resolved sources. ## Troubleshooting | Symptom | Fix | | - | - | | **Tool never selected** | Ensure resolved sources include `target_group_arns`, `target_group_arn`, or `load_balancer_arn` | | **Healthy targets look fine but traffic is bad** | Pass the full `target_group_arns` list for multi-TG ALBs — a single ARN can miss unhealthy groups | | **AccessDenied** | Attach `DescribeTargetGroups` and `DescribeTargetHealth` to the ambient identity OpenSRE runs as | | **Wrong region** | Set `AWS_REGION` to the region where the load balancer lives | ## Security * All ELB API calls are read-only. * Prefer a least-privilege IAM policy with only the two describe actions above if you do not need broader AWS access. * Store AWS credentials in `.env` or your secret manager — not in source control. # Helm (CLI) Source: https://opensre.com/docs/integrations/cloud/helm Connect Helm 3 so OpenSRE can list releases, inspect status and history, and read rendered values and manifests ## Overview OpenSRE runs the **Helm 3** command-line client on the machine where the agent executes. It uses **read-only** subcommands (`helm list`, `helm status`, `helm history`, `helm get values`, `helm get manifest`) with explicit `--kube-context` and `--kubeconfig` flags so tool calls target the same cluster your engineers use. > [!NOTE] > > **Helm 2 is not supported.** Verification checks `helm version` and requires a Helm **3.x** client. ## Prerequisites * **Helm 3** installed and on `PATH` (or configured via `helm_path`) * **`kubectl` access** to the cluster (kubeconfig on disk or in the default search path) ## Setup Configure Helm like other local integrations: run `opensre integrations setup helm`, use environment variables, and/or `~/.opensre/integrations.json`. ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup helm ``` ### Option 2: Environment variables Enable the integration and point it at your cluster: ```bash theme={null} # Required gate — set to 1, true, or yes OSRE_HELM_INTEGRATION=1 # Optional overrides (defaults shown where applicable) HELM_PATH=helm HELM_KUBE_CONTEXT= HELM_KUBECONFIG= # Default namespace hint when the alert does not specify one HELM_NAMESPACE= ``` | Variable | Default | Description | | - | - | - | | `OSRE_HELM_INTEGRATION` | — | **Required** to activate Helm from env. Must be `1`, `true`, or `yes` (case-insensitive). | | `HELM_PATH` | `helm` | Helm binary name or absolute path. | | `HELM_KUBE_CONTEXT` | — | Passed to every Helm invocation as `--kube-context`. | | `HELM_KUBECONFIG` | — | Passed as `--kubeconfig` (path to the kubeconfig file). | | `HELM_NAMESPACE` | — | `default_namespace` in the resolved integration; used as a fallback namespace when the alert does not supply one. | To raise the maximum size of stored manifest text (see [Advanced](#advanced)): ```bash theme={null} # Integer, minimum 1024; default in code is 600_000 characters HELM_MANIFEST_MAX_CHARS=600000 ``` ### Option 3: Persistent store Add an active `helm` record to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "helm-prod", "service": "helm", "status": "active", "credentials": { "helm_path": "helm", "kube_context": "prod-admin", "kubeconfig": "", "default_namespace": "production" } } ] } ``` **Credential field aliases** (store / API compatibility): `context` for `kube_context`, `kubeconfig_path` or `kube_config` for `kubeconfig`, and `namespace` for `default_namespace`. The setup/store path does not require `OSRE_HELM_INTEGRATION`; that env gate applies only when discovering Helm from environment variables alone. ## Credentials Helm uses your local Helm binary plus kubeconfig access to the cluster. There is no separate Helm API token. * Point `HELM_PATH` / `helm_path` at a Helm **3.x** binary. * Set `HELM_KUBECONFIG` / `kubeconfig` and `HELM_KUBE_CONTEXT` / `kube_context` when you need a specific cluster identity. * Prefer a dedicated kubeconfig or context with **least privilege** if your policy requires it. ## Tools | Tool | What it does | | - | - | | `helm_list_releases` | Lists releases (JSON from `helm list`); uses all namespaces when no namespace filter is set. | | `helm_release_status` | `helm status -o json` for one release. | | `helm_release_history` | `helm history -o json` for one release. | | `helm_get_release_values` | `helm get values -o json` (user-supplied values; JSON `null` from Helm is treated as an empty object). | | `helm_get_release_manifest` | Rendered manifest YAML (may be truncated; see Advanced). | ### Evidence keys Post-processing writes **distinct** evidence keys so parallel tools do not overwrite each other: | Evidence key | Content | | - | - | | `helm_releases` | Parsed release list | | `helm_release_status` | Status object from `helm status` | | `helm_release_history` | Revision history array | | `helm_release_values` | Values object | | `helm_release_manifest` | Rendered manifest YAML **string**; unchanged key name. When the manifest exceeds the size cap, this value is **truncated** text, not a separate key. | | `helm_manifest_truncated` | Boolean: `true` when `helm_release_manifest` was truncated because of `HELM_MANIFEST_MAX_CHARS` / the client default cap. | ### Usage in chat Once Helm is configured, start `opensre` and name the release and namespace in your question, e.g. *"What changed in the `my-api` release in `prod`? Show its history and values."* When your question omits a namespace, `default_namespace` from the config is used as the fallback. ### Advanced * **Manifest size:** Very large charts can produce multi-megabyte manifests. The client truncates manifest text by default; override with `HELM_MANIFEST_MAX_CHARS` (see Option 2). * **Local kind demo:** From a repo checkout with Docker, kind, kubectl, and Helm installed, run `./tests/e2e/kubernetes/helm/scripts/demo-helm-kind.sh` to create a sample cluster and release (see script comments for teardown). ### Local verification recipe Verified live: the demo script above produces a real cluster and release, then all 5 registered tools were confirmed against it. ```bash theme={null} ./tests/e2e/kubernetes/helm/scripts/demo-helm-kind.sh ``` This creates a `kind` cluster named `opensre-helm-demo`, installs `bitnami/nginx` as release `demo` in namespace `demo`, and prints a Helm/kubectl snapshot. ```bash theme={null} export OSRE_HELM_INTEGRATION=1 export HELM_KUBE_CONTEXT="kind-opensre-helm-demo" export HELM_NAMESPACE="demo" ``` Verify: ```bash theme={null} opensre integrations verify helm ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL helm │ local env │ ✓ passed │ Helm CLI is available and can reach the │ │ │ Kubernetes cluster. ``` Chat sessions (unlike `opensre integrations verify`) only fall through to env vars when the store has no records at all -- any existing record, for any service, blocks env-var resolution entirely. Point `OPENSRE_INTEGRATIONS_STORE_PATH` at a path inside a fresh empty directory instead, so your real config is never read or written and the `HELM_*` vars above are the only source of connection info: ```bash theme={null} export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-helm-demo.XXXXXX)" export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json" ``` Now ask the agent about the deployed release: ```bash theme={null} opensre ``` Ask: *What Helm releases are in the demo namespace, and is the demo release healthy?* All 5 registered tools return real data against this cluster -- confirmed by direct calls to each: release listing, release status, release history, release values (empty `{}` here since the demo installs with default values only -- Helm's `get values` returns only user-supplied overrides, not the chart's own defaults), and the rendered manifest. Teardown: ```bash theme={null} kind delete cluster --name opensre-helm-demo rm -rf "$OPENSRE_DEMO_STORE_DIR" unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH OSRE_HELM_INTEGRATION HELM_KUBE_CONTEXT HELM_NAMESPACE ``` ## Verify ```bash theme={null} opensre integrations verify helm ``` A passing check runs `helm version` (must report Helm 3) and a minimal `helm list -A --max 1 -o json` against your cluster to validate JSON output and reachability. ## Troubleshooting | Symptom | What to check | | - | - | | **Verify: missing** | Helm not in store / env gate `OSRE_HELM_INTEGRATION` not set / invalid JSON store entry. | | **Verify: Helm 3 required** | Upgrade to Helm 3 or point `helm_path` at a v3 binary. | | **Verify: list JSON / cluster** | kubeconfig and context; cluster reachable with `helm list -A` manually. | | **Wrong namespace / release not found** | Name the namespace in your question, or set `default_namespace` in config. | ## Security * The integration is **read-only**: it does not `install`, `upgrade`, or `uninstall` releases. * `helm get values` output can include **secrets**; treat evidence like any other sensitive kubectl/Helm output. * Prefer a dedicated kubeconfig or context with **least privilege** if your policy requires it. * Store paths and context names in `.env` or the integration store — not in source control. # Kubernetes Source: https://opensre.com/docs/integrations/cloud/kubernetes Connect any Kubernetes cluster via kubeconfig so OpenSRE can investigate pods, deployments, and events ## Overview OpenSRE connects to Kubernetes using a standard kubeconfig. Works with any cluster — GKE, AKS, EKS, on-prem, kind, or minikube — without requiring AWS credentials or cloud-specific tooling. ## Prerequisites * A kubeconfig file (typically `~/.kube/config`) or its raw YAML content * Read access to the namespaces you want to investigate (`get`/`list` on pods, deployments, events, logs) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup kubernetes ``` The wizard will ask for: 1. **kubeconfig** — paste the raw kubeconfig YAML, or provide a file path that will be read and stored inline 2. **Context** — optional; leave blank to use the kubeconfig's current context 3. **Default namespace** — defaults to `default` ### Option 2: Environment variables ```bash theme={null} # Point to a kubeconfig file KUBECONFIG=/home/user/.kube/config # Or provide the raw YAML directly (useful in CI / Docker) KUBECONFIG_CONTENT="$(cat ~/.kube/config)" # Optional overrides KUBECONFIG_CONTEXT=my-context KUBECONFIG_NAMESPACE=production ``` ### Option 3: Persistent store Store an active `kubernetes` record in `~/.opensre/integrations.json` with kubeconfig content (or path resolved into content by setup), plus optional context and namespace. ## Credentials | Field | Environment variable | Default | Description | | - | - | - | - | | `kubeconfig` | `KUBECONFIG_CONTENT` or read from `KUBECONFIG` | — | Raw kubeconfig YAML (required) | | `context` | `KUBECONFIG_CONTEXT` | current context | Kubeconfig context to activate | | `namespace` | `KUBECONFIG_NAMESPACE` | `default` | Default namespace for tool queries | The kubeconfig's service account or user needs these Kubernetes RBAC permissions: ```yaml theme={null} rules: - apiGroups: [""] resources: ["pods", "pods/log", "events", "namespaces", "nodes", "services", "configmaps", "persistentvolumeclaims"] verbs: ["get", "list"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "daemonsets", "replicasets"] verbs: ["get", "list"] - apiGroups: ["networking.k8s.io"] resources: ["ingresses"] verbs: ["get", "list"] ``` ## Tools | Tool | What it does | | - | - | | `kubernetes_list_pods` | List pods in a namespace with phase, readiness, and restart counts | | `kubernetes_get_pod_logs` | Fetch recent log lines from a pod container | | `kubernetes_list_deployments` | List deployments with desired/ready/available replica counts | | `kubernetes_get_events` | List cluster events — crash loops, OOM kills, scheduling failures | | `kubernetes_describe_pod` | Fetch full spec, status, and container states for a single pod | | `kubernetes_list_nodes` | List cluster nodes with conditions, capacity, and allocatable resources | | `kubernetes_list_services` | List services with their type, clusterIP, ports, and selector | | `kubernetes_list_statefulsets` | List StatefulSets with desired/ready/current/updated replica counts | | `kubernetes_list_daemonsets` | List DaemonSets with desired/current/ready/available counts | | `kubernetes_list_ingresses` | List Ingress resources with routing rules, host→service path mappings, and TLS config | | `kubernetes_list_configmaps` | List ConfigMaps with their key-value data | | `kubernetes_get_resource` | Fetch a single named resource by type and name (pods, deployments, statefulsets, daemonsets, services, ingresses, configmaps, replicasets, pvcs, nodes) | ### Example: investigating a crash-looping pod ``` > What's wrong with the payments-api pod in the production namespace? ``` OpenSRE will: 1. Call `kubernetes_list_pods` scoped to `production` to find pods with high restart counts 2. Call `kubernetes_get_pod_logs` to read recent container output 3. Call `kubernetes_get_events` with `involvedObject.name=payments-api` to surface Warning events 4. Correlate the evidence into a root-cause summary ### Local verification recipe Verified all 12 registered tools live against a real local `kind` cluster with a representative workload -- a Deployment, StatefulSet, DaemonSet, Service, Ingress, ConfigMap, and a deliberately crash-looping bare pod. ```bash theme={null} kind create cluster --name opensre-k8s-demo cat > /tmp/k8s-demo-workload.yaml << 'YAML' apiVersion: v1 kind: Namespace metadata: name: demo --- apiVersion: v1 kind: ConfigMap metadata: name: demo-config namespace: demo data: APP_MODE: "verify" --- apiVersion: apps/v1 kind: Deployment metadata: name: demo-web namespace: demo spec: replicas: 2 selector: matchLabels: {app: demo-web} template: metadata: labels: {app: demo-web} spec: containers: - name: nginx image: nginx:alpine --- apiVersion: apps/v1 kind: StatefulSet metadata: name: demo-cache namespace: demo spec: serviceName: demo-cache replicas: 1 selector: matchLabels: {app: demo-cache} template: metadata: labels: {app: demo-cache} spec: containers: - name: redis image: redis:alpine --- apiVersion: apps/v1 kind: DaemonSet metadata: name: demo-agent namespace: demo spec: selector: matchLabels: {app: demo-agent} template: metadata: labels: {app: demo-agent} spec: containers: - name: agent image: busybox command: ["sh", "-c", "while true; do sleep 3600; done"] --- apiVersion: v1 kind: Service metadata: name: demo-web namespace: demo spec: selector: {app: demo-web} ports: - {port: 80, targetPort: 80} --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: demo-ingress namespace: demo spec: rules: - host: demo.local http: paths: - path: / pathType: Prefix backend: service: {name: demo-web, port: {number: 80}} --- apiVersion: v1 kind: Pod metadata: name: demo-crashloop namespace: demo spec: restartPolicy: Always containers: - name: crasher image: busybox command: ["sh", "-c", "echo 'payment-service: connection refused to billing-api'; exit 1"] YAML i=0 until kubectl apply -f /tmp/k8s-demo-workload.yaml; do i=$((i + 1)) if [ "$i" -ge 10 ]; then echo "workload apply never succeeded; check: kubectl get events -n demo" >&2 break fi sleep 2 done ``` > [!WARNING] > > The bare pod's `apply` can fail on the first attempt with `error looking up service > account demo/default: serviceaccount "default" not found` -- the Namespace's default > ServiceAccount is created asynchronously by a controller shortly after the Namespace > itself, so a Pod in the same `kubectl apply` batch can race it. Retrying the whole > apply (already-created objects are simply reported `unchanged`) is simpler than adding > a separate wait step, and resolves on the very next attempt once the ServiceAccount > controller catches up. ```bash theme={null} export KUBECONFIG_CONTENT="$(cat ~/.kube/config)" export KUBECONFIG_CONTEXT="kind-opensre-k8s-demo" export KUBECONFIG_NAMESPACE="demo" ``` `opensre integrations verify` checks a saved store record before env vars, and chat sessions only fall through to env vars when the store has no records at all -- any existing record, for any service, blocks env-var resolution entirely. Point `OPENSRE_INTEGRATIONS_STORE_PATH` at a path inside a fresh empty directory before verifying, so a real saved record can't shadow the `KUBECONFIG_*` vars above, and your real config is never read or written: ```bash theme={null} export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-k8s-demo.XXXXXX)" export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json" ``` Verify: ```bash theme={null} opensre integrations verify kubernetes ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL kubernetes │ local env │ ✓ passed │ Connected to Kubernetes cluster │ │ │ namespace 'demo' accessible (1 pod(s) │ │ │ visible). ``` Now ask the agent about the crash-looping pod: ```bash theme={null} opensre ``` Ask: *Why is the demo-crashloop pod in the demo namespace crash-looping?* Against this exact local cluster the agent lists pods, describes `demo-crashloop`, pulls its events and logs, and finds the hardcoded `exit 1` in the container command. All 12 registered tools were confirmed to return real data against this cluster. Teardown: ```bash theme={null} kind delete cluster --name opensre-k8s-demo rm -rf "$OPENSRE_DEMO_STORE_DIR" unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH KUBECONFIG_CONTENT KUBECONFIG_CONTEXT KUBECONFIG_NAMESPACE ``` ## Verify ```bash theme={null} opensre integrations verify kubernetes ``` A successful verification lists the namespaces visible to the configured credentials. Inside the REPL: `/integrations verify kubernetes` or `/verify kubernetes`. ## Troubleshooting | Symptom | Fix | | - | - | | **Status: missing** | Run `opensre integrations setup kubernetes` or set `KUBECONFIG` / `KUBECONFIG_CONTENT` | | **Unauthorized / Forbidden** | Confirm the kubeconfig user/service account has the RBAC verbs listed under Credentials | | **Wrong cluster or namespace** | Set `KUBECONFIG_CONTEXT` and `KUBECONFIG_NAMESPACE`, or re-run setup | | **Cannot reach API server** | Check VPN / network path to the cluster endpoint in the kubeconfig | ## Security * Prefer a dedicated read-only service account or user for OpenSRE. * Store kubeconfig material in the integration store or a secret manager — not in source control. * Scope RBAC to the namespaces you want investigated (`get` / `list` only). # Railway Source: https://opensre.com/docs/integrations/cloud/railway Configure Railway deployment inspection ## Overview The Railway integration inspects the latest successful deployment of any configured Railway service. ## Prerequisites * Railway CLI installed (`@railway/cli`) * Either an authenticated Railway CLI session (`railway login`) or a Railway token * Optional defaults: project, service, and environment to inspect ## Setup ### Option 1: Interactive CLI ```bash theme={null} npm install -g @railway/cli railway login opensre integrations setup railway ``` The setup flow accepts an optional token and default project, service, and environment. The token is used when present; an authenticated Railway CLI can be used without one. Tool calls can override any configured default scope. ### Option 2: Environment variables ```bash theme={null} RAILWAY_TOKEN= RAILWAY_PROJECT=project-id-or-name RAILWAY_SERVICE=service-id-or-name RAILWAY_ENVIRONMENT=production RAILWAY_PATH=railway ``` | Variable | Default | Description | | - | - | - | | `RAILWAY_TOKEN` | — | Railway API/token auth when not using an interactive CLI login | | `RAILWAY_PROJECT` | — | Default project id or name | | `RAILWAY_SERVICE` | — | Default service id or name | | `RAILWAY_ENVIRONMENT` | — | Default environment (for example `production`) | | `RAILWAY_PATH` | `railway` | Path or name of the Railway CLI binary | ## Credentials 1. Install the Railway CLI and run `railway login`, **or** create a Railway token in the Railway dashboard. 2. Optionally set default project / service / environment so inspect calls need fewer parameters. 3. Store the token in `.env` as `RAILWAY_TOKEN` or enter it during `opensre integrations setup railway`. Verify requires a token **or** a complete default project + service + environment scope. ## Tools | Tool | What it does | | - | - | | `inspect_railway_deployment` | Shows the latest successful deployment and available source commit metadata for a Railway service | ### Inspect a deployment When a default scope is configured, no parameters are required. Otherwise provide the service scope: ```json theme={null} { "project": "project-id-or-name", "service": "service-id-or-name", "environment": "production" } ``` The result includes deployment ID, status, and source commit hash/message when Railway provides them. Commands always use explicit project, service, and environment flags and never write Railway link state into the workspace. ## Verify ```bash theme={null} opensre integrations verify railway ``` ## Troubleshooting | Symptom | Fix | | - | - | | **deployment\_unavailable** | Railway returned no successful deployment in the inspected history — confirm project/service/environment and that a successful deploy exists | | **Missing configuration / CLI / auth** | Install `@railway/cli`, run `railway login` or set `RAILWAY_TOKEN`, and set defaults or pass scope on each call | | **Verify fails without token** | Provide `RAILWAY_TOKEN`, or set complete `RAILWAY_PROJECT` + `RAILWAY_SERVICE` + `RAILWAY_ENVIRONMENT` with an authenticated CLI | ## Security * Railway tokens are never returned in tool output; Railway commands are non-interactive and errors are redacted. * Prefer a dedicated token scoped to the projects OpenSRE should touch. * Store tokens in `.env` or your secret manager — not in source control. # AWS S3 Source: https://opensre.com/docs/integrations/cloud/s3 Inspect S3 objects, metadata, and pipeline markers during incidents ## Overview OpenSRE uses AWS S3 to read object metadata and content when an alert points at data in a bucket — inspecting the files a pipeline produced, sampling an object to spot a schema change, pulling an audit payload, or confirming a run finished by checking for a completion marker. All S3 access is read-only and routed through the shared AWS client, so the integration cannot mutate or delete your objects. ## Prerequisites * AWS credentials with S3 read access, configured per the [AWS integration](/docs/integrations/cloud/aws) * An S3 bucket you want OpenSRE to read from * IAM permissions for the two S3 actions listed below ## Setup S3 has no credentials of its own — it uses the same AWS credentials as the rest of the [AWS integration](/docs/integrations/cloud/aws). The S3 tools build their client from the ambient AWS credential chain: environment keys, a shared credentials profile, or an attached instance/task role — plus the region. ```bash theme={null} AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY AWS_REGION=us-east-1 ``` > [!NOTE] > > `AWS_ROLE_ARN` is assumed only by `opensre integrations verify aws` — the S3 tools do **not** assume it. The credentials above must have S3 read access themselves. The bucket, key, and prefix OpenSRE reads are **not** environment variables — they come from the alert's resolved sources. Each S3 tool turns on for the planner only when the sources carry the fields it needs: | Source key | Fields | Turns on | | - | - | - | | `s3` | `bucket`, `key`, `prefix` | `list_s3_objects` (needs `bucket`), `inspect_s3_object` / `get_s3_object` (need `bucket` + `key`), `check_s3_marker` (needs `bucket` + `prefix`) | | `s3_audit` | `bucket`, `key` | `get_s3_object` — fetch an audit payload referenced by an alert | | `s3_processed` | `bucket` (`prefix` optional) | `check_s3_marker` — check a pipeline's output location | ## Credentials ### IAM permissions The integration only needs two read-only S3 actions: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject" ], "Resource": "*" } ] } ``` Attach this to the same IAM role or user the S3 tools actually use — the ambient credential chain (env keys or instance/task role), not only a role that the verifier assumes. If you already use the AWS managed `ReadOnlyAccess` policy, both actions are covered. ## Tools | Tool | IAM action | What it returns | | - | - | - | | `list_s3_objects` | `s3:ListBucket` | Object keys under an optional prefix, each with size, last-modified, ETag, and storage class (up to 100 keys). | | `inspect_s3_object` | `s3:GetObject` | Object metadata — size, content type, ETag, version ID, user metadata — plus a decoded text sample of the first few KB. | | `get_s3_object` | `s3:GetObject` | The full object body (up to 1 MB) as text, for audit payloads, config, or manifest files. | | `check_s3_marker` | `s3:ListBucket` | Whether a completion marker is present under a prefix, plus the file list — a signal that a pipeline run finished. | ### Use cases * Listing what a pipeline wrote to a bucket, or confirming an expected file is present * Sampling an object to detect a schema or format change in upstream input data * Reading an audit payload or manifest to trace an external vendor interaction * Confirming a batch job completed by checking for its `_SUCCESS` marker ## Verify S3 shares the AWS integration's verification. Verify credentials and region with: ```bash theme={null} opensre integrations verify aws ``` Expected output: ``` Service: aws Status: passed Detail: Connected to AWS STS via static-creds in us-east-1; caller identity account=123456789012 arn=arn:aws:iam::123456789012:user/opensre. ``` This confirms the credentials S3 rides on. It does not test bucket-level access — that is exercised the first time a tool runs against a bucket. ## Troubleshooting | Symptom | Fix | | - | - | | **S3 tools never run** | The alert's sources have no `s3` entry, or it lacks the required field — object-level tools need both `bucket` **and** `key`, not just `bucket`. | | **AccessDenied on `GetObject` / `ListBucket`** | Attach the IAM policy above to the credentials the S3 tools actually use — the ambient credential chain (env keys or instance/task role), not only a role that the verifier assumes. | | **`check_s3_marker` reports no marker when the run finished** | It only counts files whose key contains `_SUCCESS` or `marker` (case-insensitive). A completion file named anything else won't be detected. | | **A binary object returns metadata but no readable text** | The object isn't UTF-8 (for example Parquet or gzip); you still get size, type, and ETag, but there is no text sample to read. | ## Security * All S3 access is read-only — the integration cannot mutate or delete objects. * Prefer least-privilege IAM with only `s3:ListBucket` and `s3:GetObject` on the buckets you want OpenSRE to read. * Store AWS credentials in `.env` or your secret manager — not in source control. # AWS SQS Source: https://opensre.com/docs/integrations/cloud/sqs_queues Inspect SQS queue state — backlog depth, stuck consumers, and dead-letter wiring — during incidents OpenSRE uses AWS SQS queue attributes to answer the first question of every queue-backed incident: **"what state is this queue actually in?"** When a queue alert fires, the planner can read message depth, in-flight count, visibility timeout, and dead-letter queue wiring — the signals that distinguish a normal backlog from consumers that are stuck. Queue inspection is read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot send, consume, or delete messages. ## Why this exists Queue state lives in queue *attributes*, not in logs or metrics. A consumer that hangs without raising an exception writes no error line anywhere — so log search comes back clean while the queue quietly stops draining. The specific failure this surfaces: a message that causes a consumer to hang past its `VisibilityTimeout` becomes visible again and is redelivered to the next consumer, which also hangs. Every consumer ends up holding the same message. Nothing errors, so nothing reaches the logs, and without a `RedrivePolicy` there is no receive-count ceiling to break the cycle. Reading two attributes — `in_flight_count` pinned at the consumer count and `has_dlq: false` — identifies it immediately. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/integrations/cloud/aws) (role ARN recommended) — SQS reuses the same account credentials and region, so no extra setup is needed * IAM permissions for the two read-only SQS actions listed below ## How it works SQS inspection is account-wide, so the tool becomes available to the planner whenever the [AWS integration](/docs/integrations/cloud/aws) is configured — there is nothing queue-specific to set up. The region comes from the AWS integration (or `AWS_REGION`, defaulting to `us-east-1`). The tool calls `ListQueues` to discover queues, then `GetQueueAttributes` for each one, so a prefix filter and a queue cap keep the fan-out bounded. ## Tools | Tool | AWS API calls | What it returns | | - | - | - | | `get_sqs_queue_attributes` | `sqs:ListQueues`, `sqs:GetQueueAttributes` | Per-queue state — visible depth, in-flight count, visibility timeout, DLQ wiring, and FIFO flag. | ### Parameters | Parameter | Default | Description | | - | - | - | | `queue_name_prefix` | — | Only inspect queues whose name starts with this prefix (e.g. `payments-`). Omit to inspect every queue. | | `max_queues` | `20` | Maximum queues to inspect (1–100). Bounds the per-queue attribute fan-out. | | `region` | `us-east-1` | AWS region to query. | ### Output fields | Field | Meaning | | - | - | | `visible_count` | Messages available for delivery (`ApproximateNumberOfMessages`). | | `in_flight_count` | Messages delivered but not yet deleted (`ApproximateNumberOfMessagesNotVisible`). | | `visibility_timeout_seconds` | How long a message stays hidden after delivery before redelivery. | | `has_dlq` / `redrive_policy` | Whether a dead-letter queue is configured, and its target plus `maxReceiveCount`. | | `is_fifo` | Whether the queue is FIFO. | ### Reading the output | Signal | Normal backlog | Stuck consumers | | - | - | - | | `visible_count` | High — producers outpacing consumers | Near zero | | `in_flight_count` | Low | Pinned at the consumer/pod count | | `has_dlq` | Usually `true` | Often `false` — nothing breaks the redelivery cycle | > [!NOTE] > > A **missing** numeric attribute is reported as `null`, not `0`. An absent measurement and a genuinely empty queue are different findings, and collapsing them would let a metrics gap read as "the queue is drained". > [!NOTE] > > Attributes prefixed `Approximate` are eventually consistent and can lag by around a minute. The field names keep the `approximate` semantics in mind — use them for shape and direction, not for exact reconciliation. > [!NOTE] > > If one queue's attributes cannot be read (for example a per-queue policy denying `GetQueueAttributes`), that queue is returned with an `attributes_error` field and the remaining queues are still inspected — a single unreadable queue never sinks the whole investigation. ## IAM permissions ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sqs:ListQueues", "sqs:GetQueueAttributes" ], "Resource": "*" } ] } ``` Attach this policy to the same IAM role or user already configured for the [AWS integration](/docs/integrations/cloud/aws). If you are already using the AWS managed `ReadOnlyAccess` policy, both actions are already covered. > [!NOTE] > > **Execution identity:** the AWS integration's `role_arn` / credentials gate *availability* and supply the region, but the calls themselves run through boto3's standard credential chain (environment variables, shared config, or the host's instance role) — the configured role is **not** assumed for the call. Ensure the identity the OpenSRE process runs as can perform these actions. This matches the other AWS tools (RDS/EKS/CloudTrail). ## Troubleshooting | Symptom | Fix | | - | - | | **AccessDenied on `sqs:ListQueues`** | Add the IAM policy above to the role or user used by the AWS integration. | | **No queues returned** | Check the prefix filter and confirm you are querying the region the queues live in. `ListQueues` is region-scoped. | | **A queue shows `attributes_error`** | That queue's resource policy likely denies `sqs:GetQueueAttributes`. Other queues are unaffected. | | **Fewer queues than expected** | The `max_queues` cap was hit; the response sets `truncated: true`. Raise `max_queues` (up to 100) or narrow with `queue_name_prefix`. | | **Tool reports the wrong region** | Set `AWS_REGION`, or check the `region` field on the configured AWS integration. | # Vercel Source: https://opensre.com/docs/integrations/cloud/vercel Connect Vercel so OpenSRE can check deployment status and runtime logs during incidents ## Overview OpenSRE queries Vercel to retrieve deployment status and runtime logs — helping identify whether a recent deployment caused an incident or surface errors from serverless function executions. ## Prerequisites * Vercel account with project access * Vercel API token ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup vercel ``` Provide your API token (and optional team ID) when prompted. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} VERCEL_API_TOKEN=your-vercel-api-token VERCEL_TEAM_ID=team_xxxxxxxxx # optional, for team accounts ``` | Variable | Default | Description | | - | - | - | | `VERCEL_API_TOKEN` | — | **Required.** Vercel API token | | `VERCEL_TEAM_ID` | — | Team ID for team/organization accounts | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "vercel-prod", "service": "vercel", "status": "active", "credentials": { "api_token": "your-vercel-api-token", "team_id": "team_xxxxxxxxx" } } ] } ``` ## Credentials 1. In Vercel, go to **Account Settings** → **Tokens** 2. Click **Create Token** 3. Give it a name (for example `opensre`) and set the scope to **Full Account** or a specific team 4. Copy the token > [!NOTE] > > For team accounts, find your Team ID in **Team Settings** → **General** → **Team ID**. Set `VERCEL_TEAM_ID` (or store `team_id`) so OpenSRE can list team projects. OpenSRE calls the Vercel API at `https://api.vercel.com`. ## Tools | Tool | What it does | | - | - | | `vercel_deployment_status` | Check recent deployment status for projects OpenSRE can see | | `vercel_deployment_logs` | Retrieve runtime / deployment logs to surface serverless errors | ## Verify ```bash theme={null} opensre integrations verify vercel ``` Expected output: ``` Service: vercel Status: passed Detail: Connected to Vercel API and listed 5 project(s) ``` Inside the REPL: `/integrations verify vercel` or `/verify vercel`. ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Check that the API token is valid and not expired | | **403 Forbidden** | The token may not have access to the team — check token scope | | **Projects not found** | Ensure `VERCEL_TEAM_ID` is set if using a team account | ## Security * Create a **dedicated read-only token** for OpenSRE — do not reuse deployment tokens. * Store the token in `.env`, not in source code. * Rotate tokens periodically. # Architecture audit tool Source: https://opensre.com/docs/integrations/code/architecture-audit-tool Clone a GitHub repo, run architecture shell passes, and persist full audit observations from the interactive shell ## Overview OpenSRE provides GitHub-backed **architecture audit** tools for reviewing repository structure — import boundaries, file placement, size limits, and shim patterns. Use them from the interactive shell (`opensre`) when you want a structured architecture review before or alongside an incident investigation. ## Prerequisites * GitHub connected via [GitHub integration](/docs/integrations/code/github) in the OpenSRE app * For private repos, ensure the token has `repo` read scope * An active interactive-shell session when saving observations ## Setup There is no dedicated integration setup. Connect GitHub, then use the tools from the REPL. ```bash theme={null} opensre integrations setup github opensre integrations verify github ``` Typical workflow: 1. **Clone** the target repository into a fixed temp workspace 2. Run architecture **shell heuristic passes** (import, placement, size, shim) via the action agent 3. **Save** the full untruncated observation list to disk 4. **Cleanup** the clone when finished In the REPL, describe what you want in plain language: ```text theme={null} > clone Tracer-Cloud/opensre and run an architecture audit on it ``` Or invoke tools explicitly through the action agent. ## Credentials Uses the authorized app connection from the [GitHub integration](/docs/integrations/code/github). * Each audit clones into its own directory under the architecture workspace in the system temp directory (`opensre/workspace`) * Saved reports go under `~/.opensre/{session_id}/…` ## Tools | Tool | What it does | Side effects | | - | - | - | | `architecture_clone_repo` | Shallow-clone `owner/repo` into `opensre/workspace` under the system temp directory | Creates files on disk | | `architecture_cleanup_repo` | Delete the clone at the `workspace_root` that `architecture_clone_repo` returned in the same session (refuses any other path) | Deletes clone | | `architecture_save_observations` | Write the full observation list to `~/.opensre/{session_id}/{repo}-architecture-audit-{uuid}.md` | Writes a file | ### `architecture_clone_repo` ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre", "ref": "main" } ``` Returns `workspace_root` — the path to this audit's clone. Always pass it to `architecture_cleanup_repo` when finished. ### `architecture_save_observations` Call **after** cleanup and **before** the final summarized report. Pass the **complete** findings from all four shell passes (import, placement, size, shim) — not the short user-facing summary. ```json theme={null} { "repo_name": "opensre", "observations": "## Import pass\n- ...\n## Placement pass\n- ..." } ``` ### Gotchas * **Always cleanup.** `architecture_clone_repo` leaves a shallow clone on disk until `architecture_cleanup_repo` runs. The tool description enforces this workflow. * **Workspace is fixed.** Clones go under the architecture workspace in the system temp directory — you cannot point the clone at an arbitrary path. Clones left behind for more than a day are removed when the next audit starts. * **Save the full list.** `architecture_save_observations` expects untruncated pass output. Saving only the short report template loses audit evidence. * **Session id required.** Saving observations needs an active interactive-shell session (`~/.opensre/sessions/`). Start `opensre` before running the audit. ## Verify ```bash theme={null} opensre integrations verify github ``` Expected output includes MCP tool discovery. Architecture tools require an authorized app connection. There is no separate architecture-audit verify target. ## Troubleshooting | Symptom | Fix | | - | - | | **Clone fails / auth** | Verify GitHub credentials and `repo` scope for private repos | | **Save fails without session** | Start `opensre` so a session id exists under `~/.opensre/sessions/` | | **Leftover clone on disk** | Run `architecture_cleanup_repo` with the clone's `workspace_root` in the session that cloned it. Clones left by another session or an earlier run are removed automatically after 24 hours. | | **Short report only** | Pass the full four-pass observation list to `architecture_save_observations` | ## Security * Prefer a read-only GitHub token scoped to the repos you audit. * Always clean up clones when finished. * Related: [GitHub](/docs/integrations/code/github), [GitHub workflow tools](/docs/integrations/code/github-workflow-tools), [Shell commands](/docs/getting-started/interactive-shell-commands). # Bitbucket Source: https://opensre.com/docs/integrations/code/bitbucket Connect Bitbucket so OpenSRE can correlate recent commits and code changes with incidents ## Overview OpenSRE queries Bitbucket to retrieve recent commits, file contents, and code search results — helping trace which change in a Bitbucket-hosted repository triggered an incident. All operations are read-only. ## Prerequisites * Bitbucket Cloud account * App password with repository read access * Workspace slug (the identifier in `https://bitbucket.org//`) ## Setup Bitbucket is configured through environment variables or the persistent integration store. There is no dedicated `opensre integrations setup bitbucket` wizard today. ### Option 1: Environment variables Add to your `.env`: ```bash theme={null} BITBUCKET_WORKSPACE=your-workspace-slug BITBUCKET_USERNAME=your-username BITBUCKET_APP_PASSWORD=your-app-password # optional: # BITBUCKET_BASE_URL=https://api.bitbucket.org/2.0 ``` | Variable | Default | Description | | - | - | - | | `BITBUCKET_WORKSPACE` | — | **Required.** Bitbucket workspace slug | | `BITBUCKET_USERNAME` | — | **Required.** Bitbucket username | | `BITBUCKET_APP_PASSWORD` | — | **Required.** Bitbucket app password | | `BITBUCKET_BASE_URL` | `https://api.bitbucket.org/2.0` | Bitbucket API base URL | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "bitbucket-prod", "service": "bitbucket", "status": "active", "credentials": { "workspace": "your-workspace-slug", "username": "your-username", "app_password": "your-app-password" } } ] } ``` ## Credentials ### Creating an app password 1. In Bitbucket, go to **Personal settings** → **App passwords** 2. Click **Create app password** 3. Give it a label (for example `opensre`) 4. Enable the following permissions: **Repositories: Read** 5. Copy the generated password > [!NOTE] > > The workspace slug is the identifier in your Bitbucket URL: `https://bitbucket.org//` Auth uses HTTP Basic with username + app password. ## Tools | Tool | What it does | | - | - | | `list_bitbucket_commits` | Recent commits for a repository, optionally filtered by file path | | `get_bitbucket_file_contents` | Contents of a file at a specific revision | | `search_bitbucket_code` | Search code across the workspace or a specific repository | ## Verify ```bash theme={null} opensre integrations verify bitbucket ``` Expected output: ``` Service: bitbucket Status: passed Detail: Authenticated as Your Name; workspace: your-workspace ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Check username and app password combination | | **Workspace not found** | Verify the workspace slug — it is case-sensitive | | **403 Forbidden** | Ensure the app password has **Repositories: Read** permission | | **Code search unavailable** | Code search requires a Bitbucket Cloud Standard or Premium plan | ## Security * Use an **app password** rather than your account password — app passwords can be revoked individually. * Scope permissions to **Repositories: Read** only. * Store credentials in `.env`, not in source code. # Counting files Source: https://opensre.com/docs/integrations/code/counting-files Ask how many files of a kind a directory holds and get the project's files, not its caches ## Overview OpenSRE counts the files that belong to your project: the modules you wrote, not the compiled caches and vendored packages sitting beside them. Ask in the interactive shell (`opensre`): ``` How many test files are under tests/interactive_shell? How many Python files does surfaces/interactive_shell have? How many workflow files are in .github/workflows? ``` ## What you get back One sentence naming the count, the pattern and the directory: ``` 103 files matching test_*.py under tests/interactive_shell, not counting generated directories ``` Up to twenty matching paths come back as examples. The count itself always covers the whole tree. ## What is skipped Directories whose contents are generated, vendored, or version-control bookkeeping: `__pycache__`, `.venv`, `venv`, `site-packages`, `node_modules`, `build`, `dist`, `.git`, and the `.mypy_cache`, `.pytest_cache`, `.ruff_cache` and `.tox` caches. Symlinked directories are not followed, so a link pointing back up the tree cannot make the count run away. ## What it does not do * Counting entries inside a YAML, JSON or TOML file is a different question: see [Structured file reads](/docs/integrations/code/structured-file-tool). * Searching file contents is a shell command with `rg`. * Listing every file name: you get the count and a few examples. # GitHub Source: https://opensre.com/docs/integrations/code/github Connect GitHub so OpenSRE can read and write issues/PRs, search code, inspect commits, and correlate changes with incidents ## Overview OpenSRE connects to GitHub so the agent can work with issues, pull requests, repositories, Actions, and code — and correlate recent commits with incidents. Once GitHub is connected, the interactive-shell **action agent** uses **`github_cli`** (authenticated `gh`) for flexible reads and writes — create/list/view issues and PRs, assign, label, comment, merge, search, releases, workflow runs, and `gh api` — without a separate approval gate. Prefer this over shell `gh` / `!gh`. These requests stay on the action path (not the conversational gather/answer loop). | Area | How OpenSRE does it | | - | - | | Ad-hoc issues, PRs, repos, search, API | `github_cli` (action agent) | | Engineering digests, PR readiness, security alerts | [GitHub workflow tools](/docs/integrations/code/github-workflow-tools) | | Security and quality fixes and PRs | [GitHub security and quality fix](/docs/integrations/code/github-security-fix) | | Failing CI fixes and branch pushes | [GitHub CI fix](/docs/cicd/github-ci-fix) | | Slack → GitHub issue create/update/close | Propose + approve via workflow mutation tools | | Code, commits, files, issue search | Dedicated GitHub tools | | Failed deploys / workflow runs | [GitHub Actions tools](/docs/integrations/code/github-actions) | | Organization-owned incident runbooks | [Runbook-guided investigations](/docs/integrations/incidents/runbook-guided-investigations) | ## Prerequisites * GitHub account with repository access * An authorized GitHub connection created in the OpenSRE app * For agent fetches and pushes from an existing checkout, HTTPS GitHub remotes; SSH keys do not replace the app connection * For chat `github_cli`: the `gh` binary on `PATH` (OpenSRE supplies the token). If it is missing, follow the [GitHub CLI installation guide](https://cli.github.com/). ## Setup Connect GitHub at [app.opensre.com](https://app.opensre.com) → Integrations. Every signed-in shell and CLI loads your authorized personal and workspace connections. The onboarding demos open a setup menu until an eligible app connection is available. After connecting, choose **I've connected GitHub — continue** to refresh and resume. ### Choose among personal and workspace connections You can connect multiple GitHub accounts. A personal connection is visible only to you; an organization connection is visible to members of that workspace. List the integrations and copy the UUID from the `ID` column for the GitHub row whose `SOURCE` is `remote`: ```bash theme={null} opensre integrations list opensre --github-connection-id ``` `--github-connection-id` accepts remote connection UUIDs, not IDs from `local` rows. The choice scopes the current shell and is forwarded with managed cloud requests. If a matching connection exists but is unavailable, or several connections leave the choice ambiguous, OpenSRE disables GitHub tools instead of guessing. ### Connect in the OpenSRE app Connect GitHub in the OpenSRE app using browser sign-in or the app's token form. Personal and workspace connections are supported. A connection uploaded by an older CLI, or whose origin cannot be verified, must be reconnected in the app. ```bash theme={null} opensre integrations setup github ``` The command, `/integrations setup github`, and onboarding open or print the app setup page. Setup stays pending until OpenSRE refreshes an authorized app connection. On a signed-out machine, run `opensre account login` to load your app connections. In the setup menu, choose **I've connected GitHub — continue** after connecting. Local credential-store entries and `GITHUB_TOKEN`, `GH_TOKEN`, and `GITHUB_MCP_AUTH_TOKEN` do not authorize agent GitHub tools or their background work. ## Credentials ### Creating a personal access token 1. Open [New fine-grained personal access token](https://github.com/settings/personal-access-tokens/new). 2. Name it **OpenSRE**, choose an expiry, and select the owner and repositories OpenSRE may use. 3. Grant read access to Contents, Metadata, Issues, Pull requests, and Actions. Add write access only for operations you want OpenSRE to perform. 4. Generate and copy the token into the OpenSRE app. If your organization requires approval, approve it before reconnecting. ## Tools | Tool / area | What it does | | - | - | | `github_cli` | Authenticated `gh` for ad-hoc issues, PRs, search, API (action agent) | | `search_github_issues` | Search issues | | `list_github_commits` | List recent commits | | `search_github_code` | Search code | | `get_github_file_contents` | Read a file from a repository | | `get_github_repository` | Repository metadata | | `get_github_repository_tree` | Repository tree | | `get_github_star_history` | Star history | | `get_git_deploy_timeline` | Deploy / git timeline signals | | Actions / workflow / fix tools | See [GitHub Actions](/docs/integrations/code/github-actions), [workflow tools](/docs/integrations/code/github-workflow-tools), [CI fix](/docs/cicd/github-ci-fix), [security fix](/docs/integrations/code/github-security-fix) | ## Verify ```bash theme={null} opensre integrations verify github ``` Expected output: ``` Service: github Status: passed Detail: GitHub MCP validated for your-username; discovered 18 tools including repository source helpers ``` Inside the REPL: `/integrations verify github` or `/verify github`. Alias: `github_mcp`. ## Troubleshooting | Symptom | Fix | | - | - | | **Authentication failed** | Reconnect GitHub or review repository access in the OpenSRE app | | **Wrong workspace GitHub account** | Run `opensre integrations list`, copy the UUID from the remote GitHub row, then start the shell with `opensre --github-connection-id ` | | **Required tools missing** | Ensure toolsets include `repos` — it provides `get_file_contents`, `list_commits`, etc. | | **`github_cli` fails / gh not found** | Install [GitHub CLI](https://cli.github.com/) so `gh` is on `PATH`; review access in the OpenSRE app | | **Connection refused** | Check that GitHub and the app connection are reachable | ## Security * Prefer the **least privilege** that matches how you use OpenSRE (read-only if you only ask questions; write scopes if you want chat to create/edit issues and PRs). * Limit token scope to the repositories OpenSRE needs. * Enter tokens only in the OpenSRE app; keep them out of source code. # GitHub Actions Source: https://opensre.com/docs/integrations/code/github-actions Trace incidents back to failing GitHub Actions workflow runs, jobs, and step logs ## Overview OpenSRE’s GitHub Actions tools help you trace incidents back to the workflow run that caused them. They are designed for situations where a failed deploy, a flaky test, or a broken secret rotation explains a production problem. This is **not** a separate integration id — it uses the same GitHub MCP credentials as [GitHub](/docs/integrations/code/github). For mutating CI fixes, see [Fix GitHub CI](/docs/cicd/github-ci-fix). ## Prerequisites * GitHub connected via [GitHub integration](/docs/integrations/code/github) * MCP toolsets that include `actions` (default: `repos,issues,pull_requests,actions`) * Token access to the repositories whose workflows you want to inspect ## Setup ```bash theme={null} opensre integrations setup github opensre integrations verify github ``` Ensure the GitHub MCP toolsets include `actions`. GitHub access comes from your selected connection in the OpenSRE app. Reconnect or review repository and Actions access there when a request is rejected. ## Credentials Use the same token and scopes as [GitHub](/docs/integrations/code/github). For private repos, the token needs access to that repository. Confirm verify output mentions Actions tools. ## Tools | Tool | What it does | | - | - | | `list_github_actions_workflow_runs` | Recent workflow runs for a repository, including status and trigger | | `list_github_actions_active_runs` | Queued and in-progress runs | | `list_github_actions_run_jobs` | Jobs and step outcomes for a run | | `get_github_actions_step_log` | Log output for a failed job step | ### Examples ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre", "per_page": 10 } ``` ```json theme={null} { "owner": "Tracer-Cloud", "repo": "opensre", "run_id": 123456789, "job_id": 987654321, "step_name": "Deploy" } ``` ### Typical workflow 1. Find the workflow run that happened right before the incident. 2. Open the job list and identify the failed job or step. 3. Pull the failed step log and look for the exact deployment/test error. 4. Use the run metadata to correlate the failure with commits, pull requests, or a secret/config change. ### Example RCA usage A failed deployment workflow often shows up as: * a run with `conclusion: failure` * a job named `deploy`, `release`, or `rollout` * a step such as `Deploy`, `Apply manifests`, or `Run migrations` That context is usually enough to connect the incident to a recent workflow change. ## Verify ```bash theme={null} opensre integrations verify github ``` Confirm the verify output mentions Actions tools. Then test from the REPL: ```text theme={null} > list recent failed workflow runs for Tracer-Cloud/opensre ``` ## Troubleshooting | Symptom | Fix | | - | - | | **GitHub integration unavailable** | Reconnect GitHub or review repository access in the OpenSRE app | | **No workflow runs returned** | Confirm `owner`/`repo` match the repository; check token access to private repos | | **Step log empty** | Use `job_id` and exact `step_name` from `list_github_actions_run_jobs` | | **MCP connection failed** | Verify `GITHUB_MCP_URL` is reachable; see [GitHub troubleshooting](/docs/integrations/code/github#troubleshooting) | | **Actions tools missing** | Ensure `GITHUB_MCP_TOOLSETS` includes `actions` | ## Security * Prefer a least-privilege token scoped to the repositories you investigate. * These tools are read-only; mutating CI fixes go through [GitHub CI fix](/docs/cicd/github-ci-fix) with explicit confirmation. * Store tokens in `.env`, not in source code. # Fix GitHub security and quality findings Source: https://opensre.com/docs/integrations/code/github-security-fix Use OpenSRE to remediate Dependabot, code-scanning, and Code Quality findings, then open a pull request or push straight to the default branch. ## Overview OpenSRE can turn one GitHub security or quality finding into a local fix and, when approved, a pull request or a commit pushed straight to the default branch. It supports Dependabot alerts, code-scanning/CodeQL alerts, and GitHub Code Quality standard findings. > [!WARNING] > > This is mutating. OpenSRE asks before editing files, then asks again before committing, pushing, and opening a PR. ## Prerequisites * Local checkout whose `origin` matches the finding repository * GitHub token with the access listed under Credentials * Optional coding-agent CLI (Pi, Claude Code, Codex, or Cursor) for findings that need broader reasoning ## Setup There is no dedicated `integrations setup` target for this tool. Connect GitHub in the OpenSRE app with repository write and security-alert read access. Configure the workspace: ```bash theme={null} export CODING_WORKSPACE=/path/to/repo # optional; defaults to cwd ``` OpenSRE has built-in local fixers for some Code Quality findings, such as unused imports and unused local variables. For findings that need broader code reasoning, it automatically uses the first coding agent CLI it finds installed and logged in: Pi, Claude Code, Codex, or Cursor. No configuration is needed when one of those CLIs already works on your machine. To pin a specific agent instead of auto-detection: ```bash theme={null} export CODING_AGENT=claude-code # pi | claude-code | codex | cursor (default: auto) export CODING_MODEL=... # optional model override for that agent ``` ## Credentials | Need | GitHub access | | - | - | | Read Dependabot alerts | `security_events` on classic PATs for private repos, `public_repo` for public-only use, or Dependabot alerts read on fine-grained tokens | | Read code-scanning alerts | Code scanning alerts read | | Read Code Quality findings | `repo` on classic PATs, `public_repo` for public-only use, or Code quality read on fine-grained tokens | | Push a branch and open a PR | Contents write and pull requests write, or equivalent `repo` access | | Push straight to the default branch | Contents write, plus a bypass for any branch rule that requires pull requests (see Troubleshooting) | ## Tools | Tool | What it does | | - | - | | `fix_github_security_alert` | Remediate one Dependabot, code-scanning, or Code Quality finding; optionally open a PR | Related skill: `fixing-github-security-alerts`. ### Use it In the interactive shell: ```text theme={null} hey fix the security issues fix the security and quality issues in Tracer-Cloud/opensre and raise a PR fix the code quality findings on https://github.com/Tracer-Cloud/opensre/security/quality fix the security issues in Tracer-Cloud/opensre and raise a PR ``` For a broad repo request, OpenSRE selects one open supported finding by severity, skipping findings that already have an open OpenSRE fix PR. When nothing is left, it reports that no PR was needed. If you do not name a repo, it uses the current checkout's GitHub `origin`. For a specific alert, include the alert URL or say the alert type and number: ```text theme={null} fix Dependabot alert 12 in this repo fix the code scanning errors on https://github.com/acme/app/security/code-scanning and raise a PR fix https://github.com/acme/app/security/code-scanning/7 and open a PR fix Code Quality finding 42 in Tracer-Cloud/opensre and raise a PR fix the code scanning issues in Tracer-Cloud/opensre and push straight to main ``` OpenSRE asks before editing files, then asks again before committing, pushing, and opening the PR. ### What happens 1. OpenSRE reads the GitHub alert or Code Quality finding details. 2. It verifies the local checkout's `origin` matches the finding repository. 3. OpenSRE first tries a built-in local fixer when one safely applies. 4. If no built-in fixer applies, OpenSRE runs the auto-detected (or pinned) coding agent CLI to implement the fix in the local checkout. 5. If PR shipping is requested and approved, OpenSRE commits only the files the fix run changed, pushes an `opensre/github-security-fix-*` branch, and opens a PR into the default branch. 6. If you ask to push straight to the default branch, OpenSRE makes the fix in a fresh clone of that branch, commits it there, and pushes it without force. If someone else pushed in the meantime, it rebases the fix once and pushes again; if the fix conflicts with their change, nothing is pushed. Secret-scanning alerts are not auto-fixed. Revoke or rotate the secret first, then plan repository cleanup separately. ### Scheduled fixes on public repositories A scheduled loop that opens PRs on a public repository should pass `quality_only=true`. OpenSRE then fixes only code-scanning and Code Quality findings without a security severity, so a vulnerability fix is never published in a public PR before you can handle it privately. ## Verify There is no dedicated `integrations verify` target. Confirm readiness by: 1. An authorized GitHub connection in the OpenSRE app with the scopes above 2. Local checkout `origin` matches the target repository 3. A coding agent CLI is available when built-in fixers do not cover the finding ## Troubleshooting | Symptom | Fix | | - | - | | **Secret-scanning refused** | Expected — revoke/rotate the secret first; this tool does not auto-fix secret leaks | | **Dual confirmation prompts** | Edit confirm, then commit/push/PR confirm when `open_pr` is requested | | **Wrong repository** | Ensure checkout `origin` matches the finding repo, or name the repo in the prompt | | **No coding agent found** | Install/login Pi, Claude Code, Codex, or Cursor, or pin `CODING_AGENT` | | **Asked to push to `main` but got a PR** | A branch rule on `main` requires pull requests, so OpenSRE opened one instead. To allow direct pushes, add the OpenSRE GitHub app as a bypass actor on that rule (repository Settings → Rules), or merge the PR | | **A loop reports nothing to fix while findings stay open** | Each remaining finding already has an open OpenSRE fix PR: merge or close those PRs, or dismiss findings you will not fix | ## Security * Dual confirmation when shipping: edit first, then commit/push/PR. * PR branches use the `opensre/github-security-fix-*` prefix. * OpenSRE pushes to the default branch only when you ask it to, from a fresh clone so your local commits and uncommitted work never ride along, and never with force. * Store tokens in `.env` or your secret manager — not in source control. # GitHub workflow tools Source: https://opensre.com/docs/integrations/code/github-workflow-tools Use GitHub-backed tools for read snapshots, status reports, follow-up summaries, and approved issue mutations ## Overview OpenSRE provides GitHub workflow tools for engineering coordination. The public workflow is: 1. Read a GitHub snapshot. 2. Generate a report or community follow-up summary. 3. Render a mutation proposal for explicit Slack-sourced task requests. 4. Execute the proposal only through the runtime approval path. The workflow tools deliberately separate read-only snapshots from mutating execution. Headless or non-approved runs can produce proposals, but they cannot execute workflow mutations. For ad-hoc chat mutations outside the Slack proposal flow (create issue, assign, merge, `gh api`, etc.), use **`github_cli`** instead — see [GitHub](/docs/integrations/code/github). ## Prerequisites * GitHub connected via [GitHub integration](/docs/integrations/code/github) * Token with at least `repo` scope; add security-alert scopes for `list_github_security_alerts` ## Setup Workflow tools use the same GitHub MCP credentials as the [GitHub integration](/docs/integrations/code/github). ```bash theme={null} opensre integrations setup github opensre integrations verify github ``` | Requirement | Details | | - | - | | **Token** | Connect GitHub in the OpenSRE app | | **Scopes** | `repo` minimum; add security alert scopes for `list_github_security_alerts` | | **Alternate env** | Local environment tokens do not authorize these tools | ### Example REPL turn ```text theme={null} > generate a Slack-ready morning check-in for Tracer-Cloud/opensre · gathering via GitHub · list work items… ``` ## Credentials Connect or reconnect GitHub in the OpenSRE app. Security alert endpoints require token scopes that GitHub enforces separately. If the token cannot read one alert class, the tool returns an error for that alert type while still returning any alert classes it can read. ## Tools | Tool | Role | Side effects | | - | - | - | | `list_github_work_items` | Reads issues and classifies work as `taken`, `up_for_grabs`, or `unassigned`. | None | | `summarize_github_pr_status` | Reads PR detail endpoints, authoritative mergeability, check runs, and blocking reasons. | None | | `list_github_security_alerts` | Reads Dependabot, secret-scanning, and code-scanning alerts when token scope allows. | None | | `generate_work_status_report` | Produces a Slack-ready status report from work items and PR status. | None | | `summarize_community_followups` | Reads repository issue comments, then summarizes unanswered questions, agenda items, and suggested replies. | None | | `propose_github_issue_mutation_from_slack` | Builds a deterministic proposal for creating, updating, or closing a GitHub issue from an explicit Slack request. | None | | `execute_github_issue_mutation` | Executes an approved proposal. | Mutating; requires runtime approval | ### Workflow #### 1. Read snapshot Use read tools first: * `list_github_work_items` * `summarize_github_pr_status` * `list_github_security_alerts` when security status is relevant `summarize_github_pr_status` does not trust list-endpoint mergeability. It fetches each PR detail endpoint so `mergeable` and `mergeable_state` are authoritative. Unknown mergeability is reported as `unknown`, not as ready to merge. #### 2. Report or summarize Use `generate_work_status_report` for morning check-ins, Slack updates, blockers, owners, and next actions. If the report tool performs its own GitHub reads and a required read fails, it returns: * `available: false` * `incomplete: true` * an `errors` list It must not produce a false "no blockers" report from partial data. Use `summarize_community_followups` for contributor questions, community meeting agenda items, and suggested replies. It reads repository issue comments directly with pagination instead of performing one request per issue. #### 3. Propose mutation Only when the user explicitly asks to turn a Slack request into a GitHub task, call `propose_github_issue_mutation_from_slack`. Good proposal triggers: * "Add this Slack request to the project task list." * "Create a GitHub task from this thread." * "Update issue 42 with this Slack context." * "Close task 51; PR #2973 shipped." The proposal includes: * operation: `create`, `update`, or `close` * target issue, when applicable * rendered GitHub payload * source Slack link * deterministic `proposal_id` * idempotency marker for retry detection #### 4. Execute mutations `execute_github_issue_mutation` is the only **workflow** mutating tool. It has no `confirm` argument. Ad-hoc GitHub writes in chat go through `github_cli`. Mutation behavior: * **create**: searches for the idempotency marker first; creates only if no existing issue is found. * **update**: fetches the issue, adds the Slack follow-up as a comment unless the proposal marker is already present, and patches title/labels/assignees only when those fields are explicitly present. It never replaces the issue body. * **close**: fetches the issue, adds a closing comment unless the proposal marker is already present, then patches `state=closed` and `state_reason=completed`. It never replaces the issue body. > [!WARNING] > > Do not expose `execute_github_issue_mutation` outside the runtime approval path. Headless runs may render proposals, but should not execute mutations directly. ### Example prompts * "Which PRs are mergeable, blocked, or unknown in `Tracer-Cloud/opensre`?" * "What issues are taken vs up for grabs?" * "Generate a Slack-ready morning check-in from current GitHub work." * "List unanswered community questions from recent issue comments." * "Propose a GitHub issue from this Slack request and keep the source link." * "Execute this approved GitHub issue proposal." ## Verify ```bash theme={null} opensre integrations verify github ``` There is no separate `github-workflow` verify target. Workflow tools become available when GitHub credentials validate. ## Troubleshooting | Symptom | Fix | | - | - | | **Tools unavailable** | Connect GitHub in the OpenSRE app, then refresh the connection | | **Security alerts error for one class** | Token lacks that alert scope — other alert classes may still return | | **False "no blockers" concern** | If report reads fail, expect `available: false` / `incomplete: true` rather than an empty green report | | **Mutation not executing** | Headless/non-approved runs can propose only — execute needs the runtime approval path | | **Want ad-hoc issue create outside Slack flow** | Use `github_cli` — see [GitHub](/docs/integrations/code/github) | ## Security * Keep read tools and mutating execute on separate approval paths. * Prefer least-privilege tokens; add security scopes only when you need alert listing. * Store tokens in `.env`, not in source control. # GitLab Source: https://opensre.com/docs/integrations/code/gitlab Connect GitLab so OpenSRE can read merge requests, commits, pipelines, and files ## Overview OpenSRE's GitLab integration gives the agent read access to your merge requests, commits, pipelines, and files, so it can answer questions about recent changes and failing pipelines. ## Prerequisites * GitLab.com or a self-hosted GitLab instance reachable from OpenSRE * A **Personal Access Token** or **Project Access Token** * Scopes: `read_api` (required); `api` only if you enable MR write-back ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup gitlab ``` When prompted, enter: * **GitLab base URL** — leave as `https://gitlab.com/api/v4` for GitLab.com, or change to your self-hosted instance URL (for example `https://gitlab.example.com/api/v4`) * **GitLab access token** — from Credentials below Setup validates your token by calling the GitLab API and writes credentials to your environment / store. ### Option 2: Environment variables | Variable | Description | | - | - | | `GITLAB_ACCESS_TOKEN` | Token used for all GitLab API calls | | `GITLAB_BASE_URL` | API base URL (defaults to `https://gitlab.com/api/v4`) | | `GITLAB_PROJECT_ID` | Optional default project scope helper | | `GITLAB_REF` | Optional default ref helper | | `GITLAB_FILE_PATH` | Optional default file-path helper | ```bash theme={null} GITLAB_ACCESS_TOKEN=glpat-... GITLAB_BASE_URL=https://gitlab.com/api/v4 ``` GitLab tools use this configured integration for the base URL and token. Do not pass `gitlab_url` or `gitlab_token` as tool arguments; those values are resolved from the integration configuration. A self-hosted instance host such as `https://gitlab.example.com` is normalized to `https://gitlab.example.com/api/v4`. ## Credentials OpenSRE supports both **Personal Access Tokens** and **Project Access Tokens**. A Project Access Token is recommended for production — it is scoped to a single project and does not depend on a user account. ### Personal Access Token (quickest for local use) 1. In GitLab, go to your avatar → **Edit profile** → **Access Tokens**. 2. Click **Add new token**. 3. Give it a name (for example `opensre`) and set an expiry date. 4. Select the following scopes: * `read_api` — required for reading MRs, commits, pipelines, and files * `api` — required only if you enable MR write-back (posting findings as MR notes) 5. Click **Create personal access token** and copy the value immediately — it is shown only once. ### Project Access Token (recommended for server deployments) 1. Open your GitLab project → **Settings** → **Access Tokens**. 2. Click **Add new token**. 3. Give it a name, set a role of **Reporter** (or **Developer** if write-back is needed), and select the same scopes as above. 4. Click **Create project access token** and copy the value. > [!NOTE] > > If your GitLab instance is self-hosted, make sure your OpenSRE server can reach it over the network before proceeding. ## Tools | Tool | What it reads | | - | - | | `list_gitlab_mrs` | MR title, description, author, reviewers, labels, diff stats | | `list_gitlab_commits` | Commit messages, authors, and changed files in a branch or MR | | `list_gitlab_pipelines` | Pipeline status, failed jobs, and job logs | | `get_gitlab_file` | File contents at a specific ref for diff analysis | ### Use GitLab tools in the interactive shell After connecting GitLab, include a GitLab project or file URL in your request. OpenSRE infers the project scope from the current message and keeps it for follow-up questions in the same session. ```text theme={null} Read https://gitlab.com/group/project/-/blob/main/runbooks/api.md and summarize the recovery steps. ``` Project URLs enable commits, merge request, and pipeline tools: ```text theme={null} Check recent changes and failed pipelines for https://gitlab.com/group/project. ``` OpenSRE also checks recent conversation, `GITLAB_PROJECT_ID`, `GITLAB_REF`, `GITLAB_FILE_PATH`, and the current repository's GitLab origin remote when the current message does not contain a URL. ## Verify ```bash theme={null} opensre integrations verify gitlab ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Validation fails with 401** | Your token is invalid or has expired. Regenerate it in GitLab and re-run setup. | | **Validation fails with 403** | Ensure `read_api` is selected. | | **Self-hosted instance not reachable** | Verify that `GITLAB_BASE_URL` ends in `/api/v4` and that OpenSRE can reach the host. Test with `curl -H "Authorization: Bearer " https://your-gitlab-host/api/v4/user`. | ## Security * Prefer a **Project Access Token** with **Reporter** for read-only access. * Store tokens in `.env` or your secret manager — not in source control. # Jenkins Source: https://opensre.com/docs/integrations/code/jenkins Connect Jenkins so OpenSRE can correlate failed builds and deployments with incidents ## Overview Correlate failed builds and deployments with incidents. OpenSRE reads your [Jenkins](https://www.jenkins.io) server over its REST API to answer: *"was there a recent build or deployment that coincides with this alert?"* ## Prerequisites * Jenkins server reachable from OpenSRE * Jenkins username with API access * A Jenkins **API token** (not your account password) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup jenkins ``` You will be prompted for the Jenkins URL, username, and API token. Related commands: | Command | What it does | | - | - | | `opensre integrations verify jenkins` | Check connectivity to the Jenkins server | | `opensre integrations show jenkins` | Show the configured Jenkins connection | | `opensre integrations remove jenkins` | Remove the stored Jenkins credentials | ### Option 2: Environment variables ```bash theme={null} export JENKINS_URL="http://localhost:8080" export JENKINS_USER="your-username" # required — Basic auth is username:token export JENKINS_API_TOKEN="" ``` | Variable | Description | | - | - | | `JENKINS_URL` | **Required.** Jenkins base URL | | `JENKINS_USER` | **Required.** Jenkins username for Basic auth | | `JENKINS_API_TOKEN` | **Required.** API token (not password) | Folder-organized jobs are supported — pass the full path, for example `team/payment-service`. ## Credentials ### Create a Jenkins API token In Jenkins: click your **username** (top-right) → **Security** → **API Token** → **Add new Token** → **Generate**. Copy the token — Jenkins shows it only once. API access uses HTTP Basic auth with your **username** and this **token** (not your password). ### Quick local test with Docker Bring up a disposable Jenkins with a pre-configured admin account (no setup wizard): ```bash theme={null} mkdir -p /tmp/jenkins-init cat > /tmp/jenkins-init/basic-security.groovy << 'EOF' import jenkins.model.* import hudson.security.* def instance = Jenkins.get() def realm = new HudsonPrivateSecurityRealm(false) realm.createAccount("admin", "admin") instance.setSecurityRealm(realm) def strategy = new FullControlOnceLoggedInAuthorizationStrategy() strategy.setAllowAnonymousRead(false) instance.setAuthorizationStrategy(strategy) instance.save() EOF docker run -d --name jenkins-dev -p 127.0.0.1:8090:8080 \ -e JAVA_OPTS="-Djenkins.install.runSetupWizard=false" \ -v /tmp/jenkins-init/basic-security.groovy:/usr/share/jenkins/ref/init.groovy.d/basic-security.groovy \ jenkins/jenkins:lts-jdk17 ``` Wait for it to come up, then create and run a job so there is a real build to inspect: ```bash theme={null} i=0 until curl -sf -o /dev/null -u admin:admin http://localhost:8090/crumbIssuer/api/json; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: Jenkins never became ready" >&2; exit 1; } sleep 2 done COOKIE_JAR=/tmp/jenkins-init/cookies.txt CRUMB=$(curl -s -c "$COOKIE_JAR" -u admin:admin http://localhost:8090/crumbIssuer/api/json | python3 -c "import json,sys;print(json.load(sys.stdin)['crumb'])") curl -s -b "$COOKIE_JAR" -u admin:admin -H "Jenkins-Crumb: $CRUMB" -H "Content-Type: application/xml" \ --data-binary 'echo deploying payment-service; exit 1' \ "http://localhost:8090/createItem?name=payment-service-deploy" curl -s -b "$COOKIE_JAR" -u admin:admin -H "Jenkins-Crumb: $CRUMB" -X POST \ "http://localhost:8090/job/payment-service-deploy/build" # The build runs async — wait for it to leave the queue and finish before inspecting it i=0 until curl -s -u admin:admin "http://localhost:8090/job/payment-service-deploy/lastBuild/api/json" \ | python3 -c "import json,sys; sys.exit(0 if json.load(sys.stdin).get('result') else 1)" 2>/dev/null; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: build never finished" >&2; exit 1; } sleep 2 done ``` ```bash theme={null} export JENKINS_URL="http://localhost:8090" export JENKINS_USER="admin" export JENKINS_API_TOKEN="admin" # local demo only — real deployments must use a token, not a password export OPENSRE_INTEGRATIONS_STORE_PATH="$(mktemp /tmp/opensre-jenkins-demo.XXXXXX)" ``` `admin`/`admin` is only valid because this is a disposable local instance with anonymous read disabled — Jenkins accepts a real password over Basic auth the same way it accepts a token when no token is configured. Never do this against a real server. The empty temporary integration store prevents saved integrations from overriding the demo environment variables. Verify: ```bash theme={null} opensre integrations verify jenkins ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL jenkins │ local env │ ✓ passed │ Jenkins connectivity successful at │ │ │ http://localhost:8090 (node: built-in) ``` The exported variables also make Jenkins available in chat; no separate setup step is required. Now ask the agent about the failing build: ```bash theme={null} opensre ``` Ask: *Why did the latest build of the payment-service-deploy Jenkins job fail?* Against this exact local build the agent calls `list_jenkins_jobs`, `list_jenkins_running_builds`, `get_jenkins_pipeline_stages`, and `get_jenkins_build_log`, and finds the real cause: the job's only 'Execute shell' step is a placeholder that runs `echo deploying payment-service` followed by `exit 1`, so every build fails deterministically. `list_jenkins_builds` is the one registered tool not needed here -- `list_jenkins_jobs` already returned this job's last-build number and status, so the agent went straight to `get_jenkins_pipeline_stages`/`get_jenkins_build_log` for that specific build instead of a redundant builds listing. Teardown: ```bash theme={null} docker rm -f jenkins-dev rm -rf /tmp/jenkins-init rm -f "$OPENSRE_INTEGRATIONS_STORE_PATH" unset JENKINS_URL JENKINS_USER JENKINS_API_TOKEN OPENSRE_INTEGRATIONS_STORE_PATH ``` ## Tools | Tool | What it does | | - | - | | `list_jenkins_builds` | Recent builds for a job with status (SUCCESS / FAILURE / RUNNING / ABORTED) and timestamp | | `get_jenkins_build_log` | Console log for a specific build — the failing step or error | | `get_jenkins_pipeline_stages` | Per-stage status and duration for a Pipeline build (empty for freestyle jobs) | | `list_jenkins_jobs` | All jobs with their last-build status | | `list_jenkins_running_builds` | Builds currently in progress across all jobs | ### Example question Name the affected job so the agent can pull its recent builds and logs: ```text theme={null} Error rate spiked shortly after a deploy — did the payment-service-deploy Jenkins job fail recently? ``` The agent lists recent builds for the job, spots the failed one near the time you mention, fetches its console log, and surfaces the failing step in its answer. ### API reference | Purpose | Endpoint | | - | - | | Connectivity check | `GET {JENKINS_URL}/api/json` | | Recent builds | `GET {JENKINS_URL}/job//api/json?tree=builds[number,result,timestamp,duration,url,building]` | | Build console log | `GET {JENKINS_URL}/job///consoleText` | | Pipeline stages | `GET {JENKINS_URL}/job///wfapi/describe` (Pipeline Stage View) | | Job list | `GET {JENKINS_URL}/api/json?tree=jobs[name,url,color,lastBuild[...]]` | ## Verify ```bash theme={null} opensre integrations verify jenkins ``` A successful check reports the server it reached, for example: ``` Jenkins connectivity successful at http://localhost:8080 (node: built-in) ``` ## Troubleshooting | Symptom | Fix | | - | - | | `Jenkins base URL is required` | Set `JENKINS_URL` or run `opensre integrations setup jenkins`. | | `Jenkins API token is required` | Generate an API token (User → Security → API Token) and configure it. | | `HTTP 401` on verify | Check the username and regenerate the API token; passwords are not accepted. | | `HTTP 403` on verify | The user lacks Overall/Read permission, or CSRF/crumb settings block the token. | | No builds returned | Confirm the job name is correct and has at least one build. | | Pipeline stages empty | Freestyle jobs have no Stage View data — use console logs instead. | ## Security * Use an API token, never your Jenkins password. * Prefer a dedicated Jenkins user with read access to the jobs OpenSRE should inspect. * Store credentials in `.env` or your secret manager — not in source control. # Pi coding tasks Source: https://opensre.com/docs/integrations/code/pi-coding Let OpenSRE hand a coding task to the Pi agent, which edits the working tree and returns a diff. ## Overview The **Pi coding tool** lets OpenSRE submit a coding task to the [Pi](https://pi.dev) agent. Pi edits files in a workspace to implement the change and returns a summary plus the git diff. It **does not commit, push, or open a pull request** — it only edits the working tree so you can review the diff. This is different from using Pi as your LLM provider (`LLM_PROVIDER=pi`, see [LLM providers](/docs/platform/llm-providers)). There Pi is the reasoning engine; here Pi is the thing that actually changes code. > [!WARNING] > > This is a **mutating** tool — it changes files on disk. It is **disabled by default**: it only becomes available to the agent when `PI_CODING_ENABLED=1`. Enable it only when you want OpenSRE to make code changes. It never commits, pushes, or opens a PR, so you always review the diff. ## Prerequisites * Pi CLI installed and authenticated (same binary/credentials as the Pi provider) * Explicit opt-in via `PI_CODING_ENABLED=1` ## Setup There is no `opensre integrations setup` target for Pi coding. Enable it with environment variables. ### Enable it 1. Install and authenticate the Pi CLI: ```bash theme={null} npm i -g @earendil-works/pi-coding-agent # then authenticate: run `pi` and use /login, or export a provider key such as GEMINI_API_KEY ``` 2. Turn the tool on: ```bash theme={null} export PI_CODING_ENABLED=1 export PI_CODING_MODEL=anthropic/claude-haiku-4-5 # optional ``` 3. Confirm Pi is ready: ```bash theme={null} uv run opensre doctor ``` | Env var | What it does | | - | - | | `PI_CODING_ENABLED` | Opt-in switch. Set to `1` to make the tool available. Off by default. | | `PI_CODING_MODEL` | Optional Pi model in `provider/model` form (for example `anthropic/claude-haiku-4-5`). | | `PI_CODING_WORKSPACE` | Default repository path Pi edits. Defaults to the current directory. | | `PI_CODING_TIMEOUT_SECONDS` | Per-task timeout (default 600, clamped 60–1800). | | `PI_BIN` | Optional explicit path to the `pi` binary. | ## Credentials Pi coding shares the `pi` binary and provider credentials with the LLM-provider Pi path. Authenticate once with `/login` inside `pi`, or export the provider API key your Pi model needs. OpenSRE does not store a separate Pi coding token. ## Tools | Tool | What it does | | - | - | | `pi_coding_task` | Run Pi headlessly on a coding task; return summary, changed files, and diff | ### How it works When the tool runs, it: 1. runs `pi` in headless mode inside the workspace with your task plus a fixed set of rules (follow `AGENTS.md`, do not commit or push, do not run destructive git commands, preserve unrelated changes, summarize what changed), 2. lets Pi edit the files, 3. returns `success`, a `summary`, the list of `changed_files`, and the `diff` (truncated if large). You review the diff and decide whether to keep, commit, or revert the changes. ### Example Once enabled, the tool is available to the agent. Describe a scoped coding task, for example: ``` add input validation to parse_config() in config/loader.py ``` If the agent calls the tool, Pi edits `PI_CODING_WORKSPACE`, and you get back a summary and the diff to review. Nothing is committed. (The agent decides whether to call the tool; for a deterministic run, invoke `pi_coding_task` directly — see the tests.) ## Verify There is no dedicated `integrations verify` target. Confirm readiness with: ```bash theme={null} uv run opensre doctor ``` In-tool readiness also checks the Pi binary and auth via `verify_pi_coding()`. If `PI_CODING_ENABLED` is unset, or the Pi CLI is missing or unauthenticated, the tool stays unavailable and returns a clear message instead of running. ## Troubleshooting | Symptom | Fix | | - | - | | **Tool never offered** | Set `PI_CODING_ENABLED=1` | | **Pi missing / unauthenticated** | Install `@earendil-works/pi-coding-agent` and complete `/login` or export a provider key | | **Wrong workspace** | Set `PI_CODING_WORKSPACE` to the repo path | | **Timeouts** | Raise `PI_CODING_TIMEOUT_SECONDS` (max 1800) | ## Security * Disabled by default — enable deliberately. * Edits the working tree only — never commits, pushes, or opens a PR. * Review every diff before you keep changes. # Resolve merge conflicts Source: https://opensre.com/docs/integrations/code/resolve-merge-conflicts Ask OpenSRE to resolve the conflicts of a git merge with a coding agent, review each hunk side by side, then commit and push. ## Overview When a `git merge` stops on conflicts, ask OpenSRE to resolve them. It hands the conflicted files to your installed coding agent, checks that no conflict marker or unmerged path is left, shows you each conflict side by side, then commits the merge and pushes the branch to the one it tracks, so the pull request updates. Files the agent cannot settle are named back to you and the merge stays open, so you decide them together and run it again. > [!WARNING] > > This is a **mutating** tool: it edits the conflicted files in your repository, > creates the merge commit, and pushes your branch. It never aborts a merge and > never pushes to a protected branch such as `main`. Whether it asks before the > commit and push follows your `/auto` level (see below). ## Prerequisites * A coding agent CLI installed and logged in (Pi, Claude Code, Codex, or Cursor; `CODING_AGENT=auto` picks the first ready one) * The repository checked out locally, with the merge already started or a branch name to merge ## Use it Start the merge yourself, or let OpenSRE start it, then ask in `opensre`: ```text theme={null} resolve the merge conflicts merge main into this branch and resolve the conflicts resolve the conflicts, keep our version of config.py ``` Any decision you give ("keep our version of config.py") is passed to the coding agent word for word. ### What happens 1. OpenSRE reads the unmerged paths of the merge in progress in the current directory. With no merge in progress, it merges the branch you named first. 2. It prints a one-line overview per conflict right away (the first line of each side), so you see what is disputed before anything runs. 3. The coding agent resolves every file on its own, as Claude Code or Cursor would. While it works, each step it takes is printed under the running tool ("Inspecting the merge", "Reading migrations.py", "Editing migrations.py", "Running tests: tests/scheduler", "Checking lint and types"), so you can see what it is doing and stop it with ESC at any point. A decision you stated in your request ("take main's version of config.py") is applied by git instead, and "decide file by file" opens a menu per file (keep ours, take theirs, combine, or type what you want). Lockfiles are regenerated from the resolved manifest, never merged by hand. 4. OpenSRE verifies that no conflict marker or unmerged path remains and prints a verdict per conflict: kept ours, took theirs, or combined, with the merged code shown only for combined hunks. A file the agent could not settle comes back as a menu showing both sides, and the merge waits. 5. It asks to commit the merge and push the branch, following your `/auto` level, then does both. Unrelated changes you had in the working tree stay uncommitted. While a merge is in progress the shell refuses `git commit`, `push`, `checkout`, `reset` and `merge` typed by the assistant, so the merge only changes through this flow. 6. When the branch has an open pull request on GitHub, it waits for that pull request's checks (up to 15 minutes) and reports whether they passed, which ones failed, or whether another commit replaced the merge. Pass `wait_for_checks: false` in your request to skip the wait. Without a GitHub origin, a GitHub token, or an open pull request, the checks are reported as not watched. ### Approval and unattended runs The commit and push are one approval, recorded as the shell's `/auto` level: | `/auto` level | Behaviour | | - | - | | `high` (default) | Commits and pushes without asking | | `med`, `low`, `off` | Asks `[y/N]` after showing the side-by-side view; `n` leaves the resolved files in the working tree, uncommitted | | Unattended run (scheduled skill, no terminal) | Commits and pushes, like `high` | Run `/auto med` when you want to review before anything leaves your machine. After a decline, ask again ("commit and push the merge") once you are happy; the coding agent is not run a second time. | Env var | What it does | | - | - | | `CODING_AGENT` | Which coding agent to run (`auto`, `pi`, `claude-code`, `codex`, `cursor`) | | `CODING_MODEL` | Optional model override for the coding agent | | `CODING_WORKSPACE` | Repository to work in. Defaults to the current directory | | `CODING_TIMEOUT_SECONDS` | Per-run timeout (default 600, clamped 60 to 1800) | ## Troubleshooting | Symptom | Fix | | - | - | | "No merge is in progress" | Run `git merge ` first, or name the branch in your request | | "none is ready" | Install and log in to a coding agent CLI | | "still need a person's decision" | Tell OpenSRE how to settle the named files and ask again | | "abandoned the merge" | The agent reset the merge; start it again with `git merge ` | | "did not push it" | The merge is committed locally; push the branch yourself (`git push`) | | "checks failed" | The merge is pushed; open the named checks on the pull request, or ask OpenSRE to fix its CI | | "checks were not watched" | Connect GitHub in the OpenSRE app, install `gh`, and open a pull request for the branch | ## Security * The merge commit is authored and committed as the OpenSRE Agent GitHub account, so the pull request shows who made it. * The push goes to the branch your checkout tracks, never to `main`, `master`, `develop`, or `trunk`. * Review the side-by-side view before approving when your `/auto` level asks. # Structured file reads Source: https://opensre.com/docs/integrations/code/structured-file-tool Ask how many jobs a workflow defines or what a config section contains, and get the answer from a parser ## Overview Questions like "how many jobs does this workflow define" have an exact answer, and it is easy to get wrong from the shell: a search for indented keys also matches `steps`, `env` and `permissions`, and reading the first 200 lines of a long file misses everything after them. OpenSRE reads YAML, JSON and TOML files with a parser instead, so the count is the count. Ask in the interactive shell (`opensre`): ``` How many top-level jobs does .github/workflows/ci.yml define? What triggers .github/workflows/codeql.yml? How many dependencies are declared in pyproject.toml? ``` ## What you get back One sentence naming the count and the entry names: ``` 7 entries under `jobs` in .github/workflows/ci.yml: quality-static, quality-typecheck, test, coverage-report, session-store-locked, package-preflight, ci-gate ``` Point at part of a file with a dotted path, such as `project.dependencies` or `jobs.test`. Leave it out for the top level of the file. ## What it does not do It reports names and counts, never a value. A configuration file may hold a token, so nothing from inside the file is echoed into the transcript. When you need a value, read it with a shell command. It reads `.yml`, `.yaml`, `.json` and `.toml`. Anything else, including prose and code, is a job for a shell command. ## Workflow files and the word `on` YAML treats a bare `on`, `off`, `yes` and `no` as true or false. A GitHub workflow's triggers live under `on:`, so a naive read reports the key as `True` and can merge two differently spelled keys into one. OpenSRE keeps keys as the file spells them, so `on` is both listed and reachable by name. ## When the answer is not there * A path that does not parse says which format failed, without the parser's internals. * A key that is not in the file says so rather than guessing a number. * Counting files in a directory is not this: use a shell command. # Google Docs Source: https://opensre.com/docs/integrations/collaboration/google-docs Connect Google Docs so OpenSRE can write incident reports to your Google Drive ## Overview Ask OpenSRE to write incident findings to Google Drive as formatted Google Docs — creating a persistent, shareable record of an incident linked to your team's existing Drive folder. ## Environment recipe Google Docs is a hosted Google API. There is no local Docker stack. You need a Google Cloud project, a service account JSON key, and a Drive folder that account can edit. ### Bring up 1. In [Google Cloud Console](https://console.cloud.google.com/), create or pick a project. 2. Enable the **Google Drive API** and the **Google Docs API**. 3. Create a service account (for example `opensre-docs`). Skip role assignment. 4. Add a JSON key and download it to a path **outside** this repository. 5. In Google Drive, create a folder for OpenSRE reports (or pick an existing one). 6. Share that folder with the service account email (`opensre-docs@.iam.gserviceaccount.com`) as **Editor**. 7. Copy the folder ID from the Drive URL: `https://drive.google.com/drive/folders/`. ### Credentials | Variable | Required | What it is | | - | - | - | | `GOOGLE_CREDENTIALS_FILE` | yes | Absolute path to the service account JSON key | | `GOOGLE_DRIVE_FOLDER_ID` | yes | Drive folder ID where reports are created | Auth uses a **service account** only (Docs + Drive API scopes) — not end-user OAuth. ### Tear down ```bash theme={null} opensre integrations remove google_docs ``` Then: 1. Unshare the Drive folder from the service account (or delete the folder if it was created only for OpenSRE). 2. In Google Cloud Console, delete the service account key (and the service account if nothing else uses it). 3. Delete the JSON file from disk. ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup google_docs ``` Provide the credentials file path and folder ID when prompted. Setup verifies Drive access before saving. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} GOOGLE_CREDENTIALS_FILE=/path/to/service-account.json GOOGLE_DRIVE_FOLDER_ID=your-google-drive-folder-id ``` ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "google-docs-prod", "service": "google_docs", "status": "active", "credentials": { "credentials_file": "/path/to/service-account.json", "folder_id": "1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgVE2upms" } } ] } ``` ## Tools | Tool | What it does | | - | - | | `create_google_docs_incident_report` | Creates a Google Doc in the configured Drive folder and inserts the incident / postmortem content. Can optionally share the doc (`reader` / `writer` / `owner`). | ## Verify ```bash theme={null} opensre integrations verify google_docs ``` Expected output: ``` Service: google_docs Status: passed Detail: Connected to Drive folder 1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgVE2upms (3 items in folder) ``` Verification probes Drive access to the configured folder. ## Troubleshooting | Symptom | Fix | | - | - | | **Credentials file not found** | Check the path in `GOOGLE_CREDENTIALS_FILE` — use an absolute path | | **403 Forbidden** | The service account hasn't been added to the Drive folder with Editor access | | **Drive API not enabled** | Enable the **Google Drive API** and **Google Docs API** in your Google Cloud project | | **Folder not found** | Confirm the folder ID and that the service account has access | ## Security * Keep the service account JSON file outside of your repository — add it to `.gitignore`. * Grant the service account access **only** to the specific Drive folder it needs. * Rotate the service account key periodically via Google Cloud Console. # Jira Source: https://opensre.com/docs/integrations/collaboration/jira Connect Jira so OpenSRE can create and update incident tickets automatically ## Overview OpenSRE connects to Jira to create incident tickets, update existing issues, and add the agent's findings as comments — keeping your team's workflow in sync. ## Prerequisites * Jira Cloud account (Jira Server/Data Center also supported) * API token and project access ## Setup There is no dedicated `opensre integrations setup jira` command today. Configure Jira through environment variables or the persistent store. ### Option 1: Environment variables ```bash theme={null} JIRA_BASE_URL=https://your-org.atlassian.net JIRA_EMAIL=you@example.com JIRA_API_TOKEN=your-api-token JIRA_PROJECT_KEY=OPS ``` | Variable | Description | | - | - | | `JIRA_BASE_URL` | **Required.** Your Jira instance URL (for example `https://your-org.atlassian.net`) | | `JIRA_EMAIL` | **Required.** Email address associated with the API token | | `JIRA_API_TOKEN` | **Required.** Jira API token | | `JIRA_PROJECT_KEY` | Default project key for new issues (for example `OPS`, `SRE`) | ### Option 2: Persistent store Add to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "jira-prod", "service": "jira", "status": "active", "credentials": { "base_url": "https://your-org.atlassian.net", "email": "you@example.com", "api_token": "your-api-token", "project_key": "OPS" } } ] } ``` | Field | Description | | - | - | | `base_url` | **Required.** Your Jira instance URL | | `email` | **Required.** Email address associated with the API token | | `api_token` | **Required.** Jira API token | | `project_key` | **Required** for create flows. Default project key for new issues | ## Credentials ### Creating a Jira API token 1. Go to [id.atlassian.com/manage-profile/security/api-tokens](https://id.atlassian.com/manage-profile/security/api-tokens) 2. Click **Create API token** 3. Give it a label (for example `opensre`) 4. Copy the token > [!NOTE] > > For Jira Data Center/Server, use a personal access token from **Profile** → **Personal Access Tokens** instead. Auth uses REST API v3 with Basic auth (email + API token). Issue descriptions use Atlassian Document Format (ADF) on create. ## Tools | Tool | What it does | | - | - | | `jira_create_issue` | Create an issue in the configured (or specified) project | | `jira_add_comment` | Add a comment with investigation findings | | `jira_issue_detail` | Fetch details for an existing issue | | `jira_search_issues` | Search issues with JQL | `project_key` is required for create (from config or the tool call). ## Verify ```bash theme={null} opensre integrations verify jira ``` Expected output on success: ``` Service: jira Status: passed Detail: Configured for Jira at https://your-org.atlassian.net. ``` CLI verify is a **configuration-presence** check (`base_url`, `email`, and `api_token` are all set) rather than a live API call. The onboard wizard performs a live validation when you configure interactively. ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Check email and API token combination | | **404 Not Found** | Verify `base_url` (include `https://`) and `project_key` | | **403 Forbidden** | The user may lack permission to create issues in the project | | **Create fails without project** | Set `JIRA_PROJECT_KEY` / store `project_key`, or pass a project key on the tool call | ## Security * Use a **dedicated Jira user** for OpenSRE with only the permissions it needs (create and comment on issues). * Prefer storing credentials in `~/.opensre/integrations.json` or a secret manager — not in source control. * Rotate the API token periodically. # Trello Source: https://opensre.com/docs/integrations/collaboration/trello Connect Trello so OpenSRE can create cards on boards during incident follow-up ## Overview When incidents happen, keeping everyone organized matters. OpenSRE can talk to the Trello API with an API key and token so you can create incident cards on a board and list you choose. > [!NOTE] > > Trello is configured via environment variables or the persistent store. There is currently no `opensre integrations setup trello` wizard, no `opensre integrations verify trello` target, and no registered tool package — card creation is available through the Trello client helpers (`create_trello_card`) when you wire credentials yourself. ## Prerequisites * A Trello account with access to the boards you want OpenSRE to use * A Trello API key and token * Optional board and list IDs for automatic card creation ## Setup ### Option 1: Environment variables Add these to your `.env`: ```bash theme={null} TRELLO_API_KEY=your_api_key TRELLO_TOKEN=your_token TRELLO_BOARD_ID=board_id_optional TRELLO_LIST_ID=list_id_optional TRELLO_BASE_URL=https://api.trello.com/1 ``` | Variable | Default | Description | | - | - | - | | `TRELLO_API_KEY` | — | **Required.** Trello API key | | `TRELLO_TOKEN` | — | **Required.** Trello API token | | `TRELLO_BOARD_ID` | *(empty)* | Optional default board ID | | `TRELLO_LIST_ID` | *(empty)* | Optional default list ID for card creation | | `TRELLO_BASE_URL` | `https://api.trello.com/1` | Trello API base URL | ### Option 2: Persistent store You can also save your Trello configuration to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "trello-prod", "service": "trello", "status": "active", "credentials": { "api_key": "your_api_key", "token": "your_token", "board_id": "board_id_optional", "list_id": "list_id_optional" } } ] } ``` ## Credentials ### Getting your API key and token 1. Visit [https://trello.com/app-key](https://trello.com/app-key) 2. Copy your Trello API key 3. Generate a token from the same page 4. Authorize access when prompted 5. Store both values securely ### Optional: Finding board and list IDs If you'd like OpenSRE to create cards in a specific location, you'll need the board and list IDs. #### Using the API ```bash theme={null} curl "https://api.trello.com/1/members/me/boards?key=${TRELLO_API_KEY}&token=${TRELLO_TOKEN}" ``` #### Manual method 1. Open the board in Trello 2. Append `.json` to the board URL 3. Search the response for the board ID 4. Locate the desired list and copy its ID ### Token longevity Trello tokens can be created with different expiration periods. When generating a token, choose an expiration period that aligns with your organization's security requirements. For long-lived OpenSRE integrations, consider using an appropriately scoped token and rotating it regularly according to your security policies. ### Required permissions Your Trello token should have: * **read** — Allows OpenSRE to view boards, lists, and cards * **write** — Allows OpenSRE to create or update cards during incident follow-up ## Tools There is no registered Trello tool package today. The integration library exposes: | Helper | What it does | | - | - | | `validate_trello_connection` | Connectivity probe (`GET /members/me`) | | `create_trello_card` | Create a card (`POST /cards`) — needs a `list_id` (argument or `TRELLO_LIST_ID`) | Set `TRELLO_LIST_ID` to the list where new cards should appear. If omitted, card creation requires the list ID at call time. `board_id` is stored for reference; card create uses the list ID. Once credentials are set, you can: * Create Trello cards from the agent's findings (via the client helper) * Send incident titles and descriptions to a target list * Track follow-up tasks on your incident board ## Verify Trello does **not** currently support `opensre integrations verify trello`. Confirm credentials by calling the members probe (or creating a test card): ```bash theme={null} curl "https://api.trello.com/1/members/me?key=${TRELLO_API_KEY}&token=${TRELLO_TOKEN}" ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 / unauthorized** | Regenerate the API key and token from [https://trello.com/app-key](https://trello.com/app-key) | | **Card create fails without list** | Set `TRELLO_LIST_ID` or pass `list_id` to `create_trello_card` | | **Board not visible** | Confirm the token has access to that board | | **No Trello tools available** | Expected today — there is no registered Trello tool package yet | ## Security * Scope the token to **read** and **write** only as needed. * Prefer a dedicated Trello account or token for OpenSRE. * Rotate tokens regularly according to your security policies. * Store credentials in `.env` or your secret manager — not in source control. # Azure SQL Source: https://opensre.com/docs/integrations/databases/azure-sql Connect Azure SQL so OpenSRE can diagnose database issues and query performance ## Overview OpenSRE connects to Azure SQL to diagnose database alerts — checking server health, finding slow queries, monitoring resource usage, and analyzing wait stats to pinpoint bottlenecks. Uses ODBC (`ODBC Driver 18 for SQL Server` by default). ## Prerequisites * An Azure SQL Database instance * Network connectivity from your OpenSRE environment to your Azure SQL server (firewall / VNet) * SQL authentication username and password * Server hostname (for example `myserver.database.windows.net`) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup azure_sql ``` ### Option 2: Environment variables ```bash theme={null} AZURE_SQL_SERVER=myserver.database.windows.net AZURE_SQL_DATABASE=mydb AZURE_SQL_USERNAME=sqladmin AZURE_SQL_PASSWORD=your_password AZURE_SQL_ENCRYPT=true ``` | Variable | Default | Description | | - | - | - | | `AZURE_SQL_SERVER` | — | **Required.** Azure SQL server hostname | | `AZURE_SQL_DATABASE` | — | **Required.** Database name | | `AZURE_SQL_USERNAME` | — | **Required.** SQL authentication username | | `AZURE_SQL_PASSWORD` | — | **Required.** SQL authentication password | | `AZURE_SQL_ENCRYPT` | `true` | Encrypt the connection | | `AZURE_SQL_PORT` | `1433` | SQL Server port | | `AZURE_SQL_DRIVER` | `ODBC Driver 18 for SQL Server` | ODBC driver name | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "azure-sql-prod", "service": "azure_sql", "status": "active", "credentials": { "server": "myserver.database.windows.net", "database": "mydb", "username": "sqladmin", "password": "your_password", "encrypt": true } } ] } ``` ## Credentials ### Finding your connection details 1. Open the Azure Portal 2. Navigate to your SQL Database resource 3. Copy **Server name** from the overview panel (add `.database.windows.net` if needed) ### Network access: Firewall 1. In Azure Portal, go to your SQL server → **Security** → **Firewalls and virtual networks** 2. Add your OpenSRE environment's IP address 3. Or enable **Allow Azure services and resources to access this server** if running in Azure > [!TIP] > > If you're not sure of your IP, start with a permissive rule temporarily, get OpenSRE working, then lock it down. ## Tools | Tool | What it does | | - | - | | `get_azure_sql_server_status` | Service tier, resource utilization, connections, database size | | `get_azure_sql_current_queries` | Active sessions and running queries | | `get_azure_sql_slow_queries` | Top resource consumers from Query Store / DMVs | | `get_azure_sql_wait_stats` | Cumulative wait types (I/O, lock, CPU) | | `get_azure_sql_resource_stats` | CPU, memory, and I/O utilization | ## Verify ```bash theme={null} opensre integrations verify azure_sql ``` Expected output: ``` Service: azure_sql Status: passed Detail: Connected to Azure SQL Database ... ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Connection timeout** | Add your OpenSRE IP to the Azure SQL firewall rules | | **Login failed** | Confirm username/password and that SQL authentication is enabled | | **SSL/certificate error** | Keep `AZURE_SQL_ENCRYPT=true`. Install `ODBC Driver 18 for SQL Server` | | **Permission denied on DMVs** | Grant `VIEW SERVER STATE` and `VIEW DATABASE STATE` to the OpenSRE user | ## Security * Use a **dedicated read-only SQL user** for OpenSRE — avoid admin credentials. * Keep encryption enabled (`AZURE_SQL_ENCRYPT=true`) in production. * Restrict firewall rules to the OpenSRE environment's egress IP. * Store credentials in `.env` or the integration store, never in source code. # ClickHouse Source: https://opensre.com/docs/integrations/databases/clickhouse Connect ClickHouse so OpenSRE can inspect query activity and system health ## Overview OpenSRE queries ClickHouse system tables to surface query activity and system health — helping diagnose database performance issues. Uses the HTTP client (`clickhouse_connect`). All operations are read-only with result caps. ## Prerequisites * ClickHouse 21.x or later * Network access from the OpenSRE environment to your ClickHouse instance * A user with read access to system tables ## Setup There is no dedicated `opensre integrations setup clickhouse` wizard today. Configure via environment variables or the persistent store, then verify. ### Option 1: Environment variables ```bash theme={null} CLICKHOUSE_HOST=your-clickhouse-host CLICKHOUSE_PORT=8123 CLICKHOUSE_DATABASE=default CLICKHOUSE_USER=default CLICKHOUSE_PASSWORD=your-password CLICKHOUSE_SECURE=false # set to true for HTTPS ``` | Variable | Default | Description | | - | - | - | | `CLICKHOUSE_HOST` | — | **Required.** ClickHouse hostname or IP | | `CLICKHOUSE_PORT` | `8123` | HTTP(S) port (8123 for HTTP, 8443 for HTTPS) | | `CLICKHOUSE_DATABASE` | `default` | Default database | | `CLICKHOUSE_USER` | `default` | Username | | `CLICKHOUSE_PASSWORD` | — | Password | | `CLICKHOUSE_SECURE` | `false` | Use HTTPS (`true`/`false`) | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "clickhouse-prod", "service": "clickhouse", "status": "active", "credentials": { "host": "your-clickhouse-host", "port": 8443, "database": "default", "username": "opensre_readonly", "password": "your-password", "secure": true } } ] } ``` ## Credentials ### Creating a read-only user ```sql theme={null} CREATE USER opensre_readonly IDENTIFIED BY 'secure-password'; GRANT SELECT ON system.* TO opensre_readonly; GRANT SELECT ON default.* TO opensre_readonly; ``` ## Tools | Tool | What it does | | - | - | | `get_clickhouse_system_health` | Metrics for connections, queries, replica status, uptime | | `get_clickhouse_query_activity` | Recent slow / failed queries from `system.query_log` | Tools require `connection_verified` after a successful verify. ## Verify ```bash theme={null} opensre integrations verify clickhouse ``` Expected output: ``` Service: clickhouse Status: passed Detail: Connected to ClickHouse 23.8.1; database: default ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused** | Check host, port, and firewall. Use port `8443` with `CLICKHOUSE_SECURE=true` | | **Authentication failed** | Verify username and password | | **SSL error** | Set `CLICKHOUSE_SECURE=true` and use port `8443` | | **Access denied on system tables** | Grant `SELECT ON system.*` to the OpenSRE user | ## Security * Create a **dedicated read-only user** with access only to `system.*` and the databases OpenSRE needs. * Use **HTTPS** (`CLICKHOUSE_SECURE=true`) in production. * Store credentials in `.env` or your secret manager — not in source control. # AWS Elasticsearch (AWS ES) Source: https://opensre.com/docs/integrations/databases/elasticsearch Connect Amazon OpenSearch Service (AWS ES) so OpenSRE can search application logs ## Overview OpenSRE queries Amazon OpenSearch Service (formerly Amazon Elasticsearch Service, AWS ES) over its REST API to pull error logs and application events into its answers. This page covers AWS ES specifically; for self-hosted OpenSearch or Elastic Cloud, see the [OpenSearch / Elasticsearch integration](/docs/integrations/databases/opensearch). AWS ES runs the same integration as self-hosted OpenSearch: there's no separate "AWS ES" config, just an `OPENSEARCH_URL` pointed at your domain endpoint. Verify and setup use the **`opensearch`** service name. ## Prerequisites * An Amazon OpenSearch Service (AWS ES) domain * The domain endpoint URL (e.g. `https://search-mydomain-xxxxx.us-east-1.es.amazonaws.com`) * Fine-grained access control enabled on the domain, with a dedicated internal user mapped to a read-only role (the built-in `readall` role works) rather than the master user ## Setup ### Option 1: CLI setup ```bash theme={null} opensre integrations setup opensearch ``` Pick **OpenSearch / Elasticsearch**. ### Option 2: Environment variables ```bash theme={null} OPENSEARCH_URL=https://search-mydomain-xxxxx.us-east-1.es.amazonaws.com OPENSEARCH_USERNAME=opensre-reader OPENSEARCH_PASSWORD=opensre-reader-password ``` | Variable | Default | Description | | - | - | - | | `OPENSEARCH_URL` | — | **Required.** Your AWS ES domain endpoint | | `OPENSEARCH_USERNAME` | — | Internal user mapped to a read-only role, not the master user | | `OPENSEARCH_PASSWORD` | — | Password for that internal user | ## Credentials Create an internal user in fine-grained access control and map it to a read-only role (for example `readall`). Do **not** use the master user for OpenSRE. ## Tools | Tool | What it returns | | - | - | | `query_elasticsearch_logs` | Log lines matching a query string within a time range, plus a filtered list of lines that look like errors (matches on `error`, `exception`, `timeout`, etc.) | The tool becomes available once the `opensearch` integration is configured. Availability is gated on `sources["opensearch"]`. ## Verify ```bash theme={null} opensre integrations verify opensearch ``` There is no separate `elasticsearch` verify target. For a local recipe (Docker container, seeded index, real tool output), see [OpenSearch / Elasticsearch → Quick local test with Docker](/docs/integrations/databases/opensearch#quick-local-test-with-docker). ## Troubleshooting | Symptom | Fix | | - | - | | **403 Forbidden** | Enable fine-grained access control; unsigned IAM resource-policy domains reject anonymous requests | | **API key auth fails** | AWS ES / OpenSearch security plugin does not natively issue API keys — use Basic Auth | | **Wrong service name** | Always use `opensearch` for setup/verify | ### Gotcha This integration also supports an `OPENSEARCH_API_KEY` (used by Elastic Cloud), but AWS ES has no equivalent: the OpenSearch security plugin doesn't natively issue API keys ([opensearch-project/security#4009](https://github.com/opensearch-project/security/issues/4009)), so Basic Auth is the only option here. It does **not** sign requests with SigV4 either. ## Security * Prefer a dedicated internal read-only user, not the master user. * Store credentials out of source control. # MariaDB Source: https://opensre.com/docs/integrations/databases/mariadb Connect MariaDB so OpenSRE can diagnose database issues ## Overview OpenSRE uses MariaDB diagnostics to investigate database issues — global status, process list, slow queries, InnoDB status, and replication. Operations are read-only. ## Prerequisites * MariaDB instance reachable from OpenSRE * A user with monitoring privileges (`PROCESS`, `performance_schema` / InnoDB status as needed) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup mariadb ``` ### Option 2: Environment variables ```bash theme={null} MARIADB_HOST=your-mariadb-host MARIADB_PORT=3306 MARIADB_DATABASE=your-database MARIADB_USERNAME=opensre_readonly MARIADB_PASSWORD=your-password MARIADB_SSL=true ``` | Variable | Default | Description | | - | - | - | | `MARIADB_HOST` | — | **Required.** Hostname or IP | | `MARIADB_PORT` | `3306` | Port | | `MARIADB_DATABASE` | — | Target database | | `MARIADB_USERNAME` | — | Username | | `MARIADB_PASSWORD` | — | Password | | `MARIADB_SSL` | `true` | Boolean TLS enable (unlike MySQL's mode strings) | ### Option 3: Persistent store Add an active `mariadb` record to `~/.opensre/integrations.json` with host, port, database, username, password, and `ssl`. ## Credentials ### Recommended user setup ```sql theme={null} CREATE USER 'opensre_readonly'@'%' IDENTIFIED BY 'secure-password'; GRANT PROCESS ON *.* TO 'opensre_readonly'@'%'; GRANT SLAVE MONITOR ON *.* TO 'opensre_readonly'@'%'; GRANT SELECT ON performance_schema.* TO 'opensre_readonly'@'%'; GRANT SELECT ON your_database.* TO 'opensre_readonly'@'%'; FLUSH PRIVILEGES; ``` Four notes on these grants, each confirmed against a real server: * `information_schema` needs no explicit grant -- MariaDB exposes it automatically based on a user's other privileges. `GRANT SELECT ON information_schema.*` is rejected outright (`ERROR 1044`) and aborts the script before later grants run. * The grant on the **target database** (`MARIADB_DATABASE`) is required -- the connector selects it as the default schema on connect, and without it every tool call fails with `Access denied ... to database`. * The replication-status tool (`SHOW ALL SLAVES STATUS`) needs `SLAVE MONITOR` \-- MySQL's `REPLICATION CLIENT` name doesn't carry the same privilege on MariaDB (`SHOW GRANTS` reports it back as `BINLOG MONITOR`, a related but different privilege that does not cover replica status). Without it, that one tool fails with `Access denied; you need (at least one of) the SLAVE MONITOR privilege(s)` even though `integrations verify mariadb` and the other four tools pass. `SLAVE MONITOR` requires **MariaDB 10.5.9+** (the privilege didn't exist before then); on any older MariaDB, grant `SUPER` instead -- it has covered `SHOW SLAVE STATUS` across every MariaDB version. * Slow queries need `performance_schema`, which is enabled by default on the official `mariadb` Docker image but often disabled in production `my.cnf` -- check `SELECT @@performance_schema` if the slow-queries tool returns an empty note. ### Quick local test with Docker ```bash theme={null} docker run -d --name mariadb-dev -p 127.0.0.1:3307:3306 \ -e MARIADB_ROOT_PASSWORD=rootpass \ -e MARIADB_DATABASE=app_db \ mariadb:11 --performance-schema=ON i=0 until docker logs mariadb-dev 2>&1 | grep -q "port: 3306"; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: MariaDB never became ready" >&2; exit 1; } sleep 2 done ``` > [!WARNING] > > Like MySQL, the official MariaDB image starts a **temporary** init-only server before the > real one -- its own log line reports `port: 0`. A readiness check against the socket (e.g. > `mariadb-admin ping`) can succeed against that temporary server and return moments before it > shuts down and the real server restarts, causing the very next command to fail with `Access > denied` or a socket error. Waiting for the final server's own log line (`port: 3306`) avoids > the race. ```bash theme={null} docker exec mariadb-dev mariadb -uroot -prootpass -e " CREATE USER 'opensre_readonly'@'%' IDENTIFIED BY 'verifypass'; GRANT PROCESS ON *.* TO 'opensre_readonly'@'%'; GRANT SLAVE MONITOR ON *.* TO 'opensre_readonly'@'%'; GRANT SELECT ON performance_schema.* TO 'opensre_readonly'@'%'; GRANT SELECT ON app_db.* TO 'opensre_readonly'@'%'; FLUSH PRIVILEGES; " docker exec mariadb-dev mariadb -uroot -prootpass -D app_db -e " CREATE TABLE orders (id INT PRIMARY KEY, amount DECIMAL(10,2)); INSERT INTO orders VALUES (1, 42.50), (2, 17.00); SELECT SLEEP(2); " > /dev/null ``` ```bash theme={null} export MARIADB_HOST=127.0.0.1 export MARIADB_PORT=3307 export MARIADB_DATABASE=app_db export MARIADB_USERNAME=opensre_readonly export MARIADB_PASSWORD=verifypass export MARIADB_SSL=false ``` Verify: ```bash theme={null} opensre integrations verify mariadb ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL mariadb │ local env │ ✓ passed │ Connected to MariaDB 11.8.8-MariaDB-ubu2404 │ │ │ target database: app_db. ``` Chat sessions (unlike `opensre integrations verify`) only fall through to env vars when the store has no records at all -- any existing record, for any service, blocks env-var resolution entirely. Point `OPENSRE_INTEGRATIONS_STORE_PATH` at a path inside a fresh empty directory instead, so your real config is never read or written and the `MARIADB_*` vars above are the only source of connection info: ```bash theme={null} export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-mariadb-demo.XXXXXX)" export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json" ``` Now ask the agent about the seeded slow query: ```bash theme={null} opensre ``` Ask: *Does the app\_db MariaDB instance have any slow queries?* Against this exact local instance the agent calls all 5 registered tools — global status, process list, replication status, slow queries, and InnoDB status — and finds the one outlier: a `SELECT SLEEP(?)` statement at \~2001 ms in `performance_schema`, with every other engine, lock, and connection metric within normal bounds. Teardown: ```bash theme={null} docker rm -f mariadb-dev rm -rf "$OPENSRE_DEMO_STORE_DIR" unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH unset MARIADB_HOST MARIADB_PORT MARIADB_DATABASE MARIADB_USERNAME MARIADB_PASSWORD MARIADB_SSL ``` ## Tools | Tool | What it does | | - | - | | `get_mariadb_global_status` | Global status / health counters | | `get_mariadb_process_list` | Current process list | | `get_mariadb_slow_queries` | Slow query digests from `performance_schema` | | `get_mariadb_innodb_status` | InnoDB engine status | | `get_mariadb_replication_status` | Replication / slave status | ## Verify ```bash theme={null} opensre integrations verify mariadb ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused** | Check host, port, firewall | | **SSL errors** | Confirm `MARIADB_SSL` matches server TLS config (default is on) | | **Slow queries empty** | Enable `performance_schema` and grant SELECT | | **Replication empty** | Expected on standalone primary | ## Security * Use a dedicated read-only user. * Keep TLS enabled (`MARIADB_SSL=true`) in production. * Store credentials out of source control. # MongoDB Source: https://opensre.com/docs/integrations/databases/mongodb Connect MongoDB so OpenSRE can diagnose database issues during incidents ## Overview OpenSRE uses MongoDB diagnostics to investigate database-related alerts — checking server health, finding slow queries, monitoring replica sets, and analyzing collection statistics. ## Prerequisites * MongoDB 4.0+ (4.4+ recommended) * Network access from the OpenSRE environment to your MongoDB instance * Valid credentials (if authentication is enabled) For Atlas Admin API metrics (not a connection string), see [MongoDB Atlas](/docs/integrations/databases/mongodb-atlas). ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup mongodb ``` Provide your connection string and target database when prompted. ### Option 2: Environment variables ```bash theme={null} MONGODB_CONNECTION_STRING=mongodb+srv://user:pass@cluster.example.net MONGODB_DATABASE=production MONGODB_AUTH_SOURCE=admin MONGODB_TLS=true ``` | Variable | Default | Description | | - | - | - | | `MONGODB_CONNECTION_STRING` | — | **Required.** MongoDB connection URI | | `MONGODB_DATABASE` | *(empty)* | Target database for profiler and collection stats | | `MONGODB_AUTH_SOURCE` | `admin` | Authentication database | | `MONGODB_TLS` | `true` | Use TLS for the connection | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "mongodb-prod", "service": "mongodb", "status": "active", "credentials": { "connection_string": "mongodb+srv://user:pass@cluster.example.net", "database": "production", "auth_source": "admin", "tls": true } } ] } ``` ## Credentials ### Connection string formats ```bash theme={null} # Localhost (no auth) mongodb://localhost:27017 # Username/password mongodb://user:password@host:27017/database?authSource=admin # Replica set mongodb://host1:27017,host2:27017,host3:27017/?replicaSet=rs0 # Atlas (SRV) mongodb+srv://user:password@cluster.example.net ``` > [!TIP] > > URL-encode special characters in credentials: `@` → `%40`, `:` → `%3A`, `#` → `%23` TLS is enabled by default. For custom certificates: ```bash theme={null} MONGODB_CA_CERT=/path/to/ca.pem # Custom CA certificate MONGODB_TLS_INSECURE=true # Skip validation (dev only) ``` ## Tools | Tool | What it does | | - | - | | `get_mongodb_server_status` | Version, uptime, connections, opcounters, memory | | `get_mongodb_current_ops` | Ops longer than a threshold (default 1 s) | | `get_mongodb_replica_status` | Member health, heartbeat, optime lag | | `get_mongodb_profiler_data` | Slow queries from `system.profile` (needs `MONGODB_DATABASE` + profiling) | | `get_mongodb_collection_stats` | Document count, storage size, indexes | > [!NOTE] > > Enable profiling with `db.setProfilingLevel(1)` (slow queries only) or `db.setProfilingLevel(2)` (all queries). ## Verify ```bash theme={null} opensre integrations verify mongodb ``` Expected output: ``` Service: mongodb Status: passed Detail: Connected to MongoDB 6.0.5; target database: production ``` ### Local Docker verification Start a disposable single-node replica set without external credentials: ```bash theme={null} docker run --rm --detach --name opensre-mongodb \ --publish 127.0.0.1:27018:27018 \ mongo:7 --replSet rs0 --bind_ip_all --port 27018 until docker exec opensre-mongodb mongosh --port 27018 --quiet \ --eval 'db.adminCommand({ping:1}).ok' | grep --quiet 1; do sleep 1; done docker exec opensre-mongodb mongosh --port 27018 --quiet \ --eval 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27018"}]})' until docker exec opensre-mongodb mongosh --port 27018 --quiet \ --eval 'db.hello().isWritablePrimary' | grep --quiet true; do sleep 1; done ``` Seed a collection, enable profiling, complete one slow query for profiler data, and leave another running for current-operation data: ```bash theme={null} docker exec opensre-mongodb mongosh --port 27018 --quiet opensre_verify --eval ' db.setProfilingLevel(2); db.metrics.insertMany(Array.from({length:1000}, (_,i) => ({service:"checkout", value:i, ts:new Date()}))); db.metrics.createIndex({service:1}); db.slow.insertOne({name:"profile-seed"}); db.slow.find({$where:"sleep(2000); return true"}).toArray(); ' docker exec --detach opensre-mongodb mongosh --port 27018 --quiet \ opensre_verify --eval 'db.slow.find({$where:"sleep(60000); return true"}).toArray()' ``` From a source checkout, run `uv run opensre integrations setup mongodb` and use these answers: | Prompt | Answer | | - | - | | Connection string | `mongodb://127.0.0.1:27018/?directConnection=true` | | Database name | `opensre_verify` | | Auth source | `admin` | | TLS enabled | `false` | Verify the connection: ```bash theme={null} uv run opensre integrations verify mongodb ``` The verifier should report MongoDB 7 and the `opensre_verify` database. While the slow query is still running, exercise `get_mongodb_current_ops` through an agent turn: ```bash theme={null} uv run opensre ask \ --allowed-tool get_mongodb_current_ops \ "Use get_mongodb_current_ops with threshold_ms 1000 now. Report every operation above the threshold." ``` The result should include the running `COLLSCAN` on `opensre_verify.slow`. The other MongoDB tools can now read the seeded collection, profiler entries, single healthy primary member, and server status. Stop the disposable instance when finished: ```bash theme={null} docker stop opensre-mongodb ``` This instance has no authentication or TLS. It binds only to loopback and is for local verification only. ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused** | Verify host/port, firewalls, and that MongoDB is running. For Atlas, whitelist the OpenSRE IP. | | **Authentication failed** | Confirm username/password and `authSource`. Test with `mongosh` first. | | **SSL: CERTIFICATE\_VERIFY\_FAILED** | Provide a CA cert via `MONGODB_CA_CERT` or set `MONGODB_TLS_INSECURE=true` for dev. | | **Profiling is disabled** | Run `db.setProfilingLevel(1)` on the target database. | | **Server is not part of a replica set** | Expected for standalone — replica tools return empty; other tools still work. | ## Security * Use a **read-only** MongoDB user for monitoring — avoid admin credentials. * Always enable **TLS** in production. * Store connection strings in `.env`, never in code. * Rotate credentials periodically. # MongoDB Atlas Source: https://opensre.com/docs/integrations/databases/mongodb-atlas Connect MongoDB Atlas to give OpenSRE visibility into cluster health, performance metrics, deployment status, and operational insights during incidents ## Overview OpenSRE uses the **Atlas Admin API** (not a MongoDB connection string) to inspect cluster health, performance metrics, slow-query recommendations, and operational events when alerts fire. For direct MongoDB wire-protocol diagnostics, see [MongoDB](/docs/integrations/databases/mongodb). ## Prerequisites * MongoDB Atlas API Public Key * MongoDB Atlas API Private Key * Atlas Project ID * Permissions to view cluster metrics and configuration (Project Read Only or higher) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup mongodb_atlas ``` Enter your Atlas API credentials when prompted. ### Option 2: Environment variables ```bash theme={null} MONGODB_ATLAS_PUBLIC_KEY=your_public_key MONGODB_ATLAS_PRIVATE_KEY=your_private_key MONGODB_ATLAS_PROJECT_ID=your_project_id MONGODB_ATLAS_BASE_URL=https://cloud.mongodb.com/api/atlas/v2 ``` | Variable | Default | Description | | - | - | - | | `MONGODB_ATLAS_PUBLIC_KEY` | — | **Required.** Atlas API public key | | `MONGODB_ATLAS_PRIVATE_KEY` | — | **Required.** Atlas API private key | | `MONGODB_ATLAS_PROJECT_ID` | — | **Required.** Atlas project identifier | | `MONGODB_ATLAS_BASE_URL` | `https://cloud.mongodb.com/api/atlas/v2` | Atlas Admin API base URL | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "mongodb-atlas-prod", "service": "mongodb_atlas", "status": "active", "credentials": { "public_key": "your_public_key", "private_key": "your_private_key", "project_id": "your_project_id" } } ] } ``` ## Credentials 1. Log in to MongoDB Atlas 2. Navigate to Access Manager → API Keys 3. Create a Project API Key 4. Copy the Public Key and Private Key 5. Copy the Project ID from Atlas project settings > [!NOTE] > > This integration uses the **Atlas Admin API**, not a MongoDB connection string. Create a Project API key with read access to cluster metrics and alerts. ## Tools | Tool | What it does | | - | - | | `get_mongodb_atlas_clusters` | Clusters in the project — state, version, size, topology | | `get_mongodb_atlas_cluster_metrics` | CPU, disk I/O, connections, opcounters | | `get_mongodb_atlas_alerts` | Open Atlas alerts | | `get_mongodb_atlas_cluster_events` | Recent project events (scaling, backups, maintenance) | | `get_mongodb_atlas_performance_advisor` | Slow-query / missing-index recommendations | ## Verify ```bash theme={null} opensre integrations verify mongodb_atlas ``` Expected output: ``` Service: mongodb_atlas Status: passed Detail: Connected to MongoDB Atlas project ... ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Regenerate the API key pair. Confirm public and private keys are copied correctly. | | **403 Forbidden** | Grant the API key **Project Read Only** (or higher) for the target project. | | **Invalid project ID** | Copy the Project ID from Atlas project settings — not the cluster name or connection string. | | **Verify passes but tools fail** | Confirm `MONGODB_ATLAS_PROJECT_ID` matches the project that owns the clusters under investigation. | ## Security * Create a **dedicated Atlas API key** for OpenSRE with the minimum project role required. * Rotate API keys periodically and revoke unused keys. * Never commit API keys to source control — use `.env` or the integration store. # MySQL Source: https://opensre.com/docs/integrations/databases/mysql Connect MySQL so OpenSRE can diagnose database issues ## Overview OpenSRE uses MySQL diagnostics to investigate database issues — checking server health, surfacing slow queries, monitoring replication status, and analyzing table statistics. Queries are wrapped as read-only. ## Prerequisites * MySQL 5.7+ (8.0+ recommended for full `performance_schema` support) * Network access from the OpenSRE environment to your MySQL instance * A read-only user with access to `information_schema` and `performance_schema` ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup mysql ``` ### Option 2: Environment variables ```bash theme={null} MYSQL_HOST=your-mysql-host MYSQL_PORT=3306 MYSQL_DATABASE=your-database MYSQL_USERNAME=opensre_readonly MYSQL_PASSWORD=your-password MYSQL_SSL_MODE=preferred # preferred, required, or disabled ``` | Variable | Default | Description | | - | - | - | | `MYSQL_HOST` | — | **Required.** MySQL hostname or IP | | `MYSQL_PORT` | `3306` | MySQL port | | `MYSQL_DATABASE` | — | **Required.** Target database | | `MYSQL_USERNAME` | `root` | Username | | `MYSQL_PASSWORD` | *(empty)* | Password | | `MYSQL_SSL_MODE` | `preferred` | `preferred` (no cert verify), `required`, or `disabled` | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "mysql-prod", "service": "mysql", "status": "active", "credentials": { "host": "prod-primary.mysql.example.com", "port": 3306, "database": "application_db", "username": "opensre_readonly", "password": "your-password", "ssl_mode": "preferred" } } ] } ``` ## Credentials ### Creating a read-only user ```sql theme={null} CREATE USER 'opensre_readonly'@'%' IDENTIFIED BY 'secure-password'; GRANT PROCESS ON *.* TO 'opensre_readonly'@'%'; GRANT REPLICATION CLIENT ON *.* TO 'opensre_readonly'@'%'; GRANT SELECT ON performance_schema.* TO 'opensre_readonly'@'%'; GRANT SELECT ON your_database.* TO 'opensre_readonly'@'%'; FLUSH PRIVILEGES; ``` Three notes on these grants, each confirmed against a real server: * `information_schema` needs no explicit grant — MySQL exposes it automatically based on a user's other privileges. `GRANT SELECT ON information_schema.*` is rejected outright (`ERROR 1044`) and aborts the script before later grants run. * The grant on the **target database** (`MYSQL_DATABASE`) is required — the connector selects it as the default schema on connect, and without it every tool call fails with `Access denied ... to database`. * `REPLICATION CLIENT` is required for `get_mysql_replication_status` (`SHOW REPLICA STATUS` / `SHOW SLAVE STATUS`) — without it, that one tool returns an access-denied error even though `integrations verify mysql` and the other four tools pass. > [!NOTE] > > `performance_schema` is enabled by default in MySQL 5.7+. Slow query data will not be available if it has been explicitly disabled in `my.cnf`. ### Quick local test with Docker ```bash theme={null} docker run -d --name mysql-dev -p 127.0.0.1:3307:3306 \ -e MYSQL_ROOT_PASSWORD=rootpass \ -e MYSQL_DATABASE=app_db \ mysql:8.0 ``` Wait for it to become healthy, then create the read-only user and a table to inspect: ```bash theme={null} i=0 until docker logs mysql-dev 2>&1 | grep -q "ready for connections.*port: 3306"; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: MySQL never became ready" >&2; exit 1; } sleep 2 done ``` ```bash theme={null} docker exec mysql-dev mysql -uroot -prootpass -e " CREATE USER 'opensre_readonly'@'%' IDENTIFIED BY 'secure-password'; GRANT PROCESS ON *.* TO 'opensre_readonly'@'%'; GRANT REPLICATION CLIENT ON *.* TO 'opensre_readonly'@'%'; GRANT SELECT ON performance_schema.* TO 'opensre_readonly'@'%'; GRANT SELECT ON app_db.* TO 'opensre_readonly'@'%'; FLUSH PRIVILEGES; CREATE TABLE app_db.orders (id INT PRIMARY KEY, amount DECIMAL(10,2)); INSERT INTO app_db.orders VALUES (1, 42.50), (2, 17.00); " ``` ```bash theme={null} export MYSQL_HOST=127.0.0.1 export MYSQL_PORT=3307 export MYSQL_DATABASE=app_db export MYSQL_USERNAME=opensre_readonly export MYSQL_PASSWORD=secure-password export MYSQL_SSL_MODE=disabled ``` Verify: ```bash theme={null} opensre integrations verify mysql ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL mysql │ local env │ ✓ passed │ Connected to MySQL 8.0.46 │ │ │ target database: app_db. ``` Chat sessions only treat an integration as active once the store resolution has *something* to fall through to env vars with -- an untouched `~/.opensre/integrations.json` with an unrelated integration in it blocks env-var fallback entirely. Point `OPENSRE_INTEGRATIONS_STORE_PATH` at an empty, valid store instead, so your real config is never read or written and the exported `MYSQL_*` vars above are the only source of credentials: ```bash theme={null} export OPENSRE_INTEGRATIONS_STORE_PATH=$(mktemp /tmp/opensre-mysql-demo.XXXXXX) echo '{"version": 2, "integrations": []}' > "$OPENSRE_INTEGRATIONS_STORE_PATH" ``` A literal zero-byte file does not work here -- the store loader expects valid JSON and raises on an empty read, so this writes an explicitly empty (but valid) store rather than just creating the file with `mktemp` alone. Kick off a long-running query so there is something real to investigate, then ask the agent: ```bash theme={null} docker exec -d mysql-dev mysql -uroot -prootpass -e "SELECT SLEEP(30);" opensre ``` Ask: *Is anything blocking connections on the app\_db MySQL instance?* Against this exact local instance the agent calls all 5 registered tools — current processes, server status, slow queries, table stats, and replication status — and correctly identifies the synthetic `SELECT SLEEP(30)` probe as the only long-running query, with every other health signal clean. An empty store makes the session fall through to env-var resolution for *every* integration, not just MySQL -- if your shell already resolves another integration from its own env vars, it rides along too (harmless). Teardown -- remove the demo store file and unset every demo variable, so nothing lingers in the shell to affect a later command: ```bash theme={null} rm -f "$OPENSRE_INTEGRATIONS_STORE_PATH" unset OPENSRE_INTEGRATIONS_STORE_PATH unset MYSQL_HOST MYSQL_PORT MYSQL_DATABASE MYSQL_USERNAME MYSQL_PASSWORD MYSQL_SSL_MODE docker rm -f mysql-dev ``` ## Tools | Tool | What it does | | - | - | | `get_mysql_server_status` | Version, uptime, connections, query rates, InnoDB buffer pool / deadlock counts | | `get_mysql_current_processes` | Active queries longer than a threshold (default 1s) | | `get_mysql_slow_queries` | Highest average time from `performance_schema.events_statements_summary_by_digest` | | `get_mysql_table_stats` | Row estimates, data/index size from `information_schema.TABLES` | | `get_mysql_replication_status` | Replica IO/SQL health, seconds behind source (`SHOW REPLICA STATUS` with `SHOW SLAVE STATUS` fallback) | ## Verify ```bash theme={null} opensre integrations verify mysql ``` Expected output: ``` Service: mysql Status: passed Detail: Connected to MySQL 8.0.32; target database: application_db ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused** | Check host, port, firewall, `bind-address` | | **Authentication failed** | Confirm user host (`'user'@'%'` or specific IP) | | **SSL error** | Try `MYSQL_SSL_MODE=disabled` or `required` | | **Access denied on performance\_schema** | Grant `SELECT ON performance_schema.*` | | **Access denied to database** on verify/tool calls | Grant `SELECT ON .*` — the connector selects `MYSQL_DATABASE` as its default schema on connect | | **Slow query data empty** | Confirm `performance_schema=ON` and restart MySQL | | **Replication shows empty** | Expected on primary — other tools still work | ## Security * Use a **dedicated read-only user** — avoid root credentials. * Prefer `MYSQL_SSL_MODE=required` in production. * Restrict the user to specific hosts rather than `'%'` where possible. * Store credentials in `.env` or your secret manager — not in source control. # OpenSearch / Elasticsearch Source: https://opensre.com/docs/integrations/databases/opensearch Connect OpenSearch or Elasticsearch so OpenSRE can search application logs and analytics indices ## Overview OpenSRE queries OpenSearch (or Elasticsearch) to retrieve application logs, error events, and analytics records — pulling concrete log lines into its answers alongside metrics and traces from your other observability tools. For Amazon OpenSearch Service (AWS ES) domain specifics, see also [AWS Elasticsearch](/docs/integrations/databases/elasticsearch). ## Prerequisites * OpenSearch 1.x/2.x or Elasticsearch 7.x/8.x reachable from the machine running OpenSRE * The cluster URL (e.g. `https://my-cluster.us-east-1.es.amazonaws.com`) * Credentials for one of: * HTTP Basic Auth (username and password) — typical for self-hosted OpenSearch * API key — typical for Elastic Cloud * No auth — only when the cluster has the security plugin disabled OpenSearch authenticates clients via Basic Auth by default; the security plugin does not natively issue API keys ([opensearch-project/security#4009](https://github.com/opensearch-project/security/issues/4009)). Most self-hosted clusters use Basic Auth. ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup opensearch ``` Pick **OpenSearch / Elasticsearch**. Choose auth mode: **Username + Password**, **API key**, or **None**. ### Option 2: Environment variables ```bash theme={null} OPENSEARCH_URL=https://my-cluster.example.com # Basic Auth (typical for self-hosted OpenSearch) OPENSEARCH_USERNAME=admin OPENSEARCH_PASSWORD=secret # API key (typical for Elastic Cloud — use instead of username/password) OPENSEARCH_API_KEY=your-api-key # Optional OPENSEARCH_INDEX_PATTERN=* OPENSEARCH_MAX_RESULTS= ``` | Variable | Default | Description | | - | - | - | | `OPENSEARCH_URL` | — | **Required.** Base URL of your cluster | | `OPENSEARCH_USERNAME` | — | Basic auth username | | `OPENSEARCH_PASSWORD` | — | Basic auth password | | `OPENSEARCH_API_KEY` | — | API key (`Authorization: ApiKey …`) | | `OPENSEARCH_INDEX_PATTERN` | `*` | Default index pattern | | `OPENSEARCH_MAX_RESULTS` | — | Optional result cap | Use API key **or** basic auth, not both — verify rejects both set together. If only one of username/password is set, no `Authorization` header is emitted. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "opensearch-prod", "service": "opensearch", "status": "active", "credentials": { "url": "https://my-cluster.example.com", "username": "admin", "password": "secret" } } ] } ``` ## Credentials Prefer a dedicated read-only user (or API key) scoped to the indices OpenSRE should search. For AWS ES, use fine-grained access control with an internal user — see [AWS Elasticsearch](/docs/integrations/databases/elasticsearch). ### Quick local test with Docker ```bash theme={null} docker run -d --name opensearch-dev -p 127.0.0.1:9201:9200 \ -e "discovery.type=single-node" \ -e "DISABLE_SECURITY_PLUGIN=true" \ -e "OPENSEARCH_JAVA_OPTS=-Xms256m -Xmx256m" \ opensearchproject/opensearch:2 i=0 until curl -sf -o /dev/null http://localhost:9201; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: OpenSearch never became ready" >&2; exit 1; } sleep 2 done ``` Seed the index with an error **and** a non-error log line — both are needed to match the two-record output shown below. The tools sort on `@timestamp`, so use that field name, not `timestamp`: ```bash theme={null} curl -sf -X POST "http://localhost:9201/logs-app/_doc" -H "Content-Type: application/json" -d '{ "@timestamp": "'"$(date -u +%Y-%m-%dT%H:%M:%S)"'Z", "level": "ERROR", "message": "payment-service: connection timeout while calling billing-api", "service": "payment-service" }' > /dev/null && curl -sf -X POST "http://localhost:9201/logs-app/_doc" -H "Content-Type: application/json" -d '{ "@timestamp": "'"$(date -u +%Y-%m-%dT%H:%M:%S)"'Z", "level": "INFO", "message": "payment-service: request completed", "service": "payment-service" }' > /dev/null && curl -sf -X POST "http://localhost:9201/logs-app/_refresh" > /dev/null || { echo "ERROR: seeding failed" >&2; exit 1; } ``` ```bash theme={null} export OPENSEARCH_URL="http://localhost:9201" ``` Verify: ```bash theme={null} opensre integrations verify opensearch ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL opensearch │ local env │ ✓ passed │ Configured for OpenSearch at │ │ │ http://localhost:9201. ``` Register the local instance through the supported setup flow: ```bash theme={null} opensre integrations setup opensearch ``` Now ask the agent about the seeded index: ```bash theme={null} opensre ``` Ask: *Is payment-service throwing connection timeout errors calling billing-api? Check the logs.* Against this exact seeded index the agent finds the seeded error via `query_elasticsearch_logs` (one payment-service timeout, zero billing-api logs across the window) and concludes billing-api is not serving traffic. It didn't need `query_opensearch_analytics` here — both tools share the same credentials and index, so either is reachable from the same setup. Teardown: ```bash theme={null} docker rm -f opensearch-dev ``` ## Tools | Tool | What it does | | - | - | | `query_opensearch_analytics` | Search analytics / log indices | | `query_elasticsearch_logs` | Log lines matching a query in a time range, plus error-like lines | Both become available once the `opensearch` integration is configured. ## Verify ```bash theme={null} opensre integrations verify opensearch ``` CLI verify is a **configuration-coherence** check (URL + XOR auth). Onboard may also call `GET /_cluster/health` for live feedback. ## Troubleshooting | Symptom | Fix | | - | - | | **Both API key and basic auth set** | Clear one method — verify rejects both | | **403 Forbidden on AWS ES** | Enable fine-grained access control; use internal user/password (not IAM SigV4) | | **No tools in plan** | Confirm verify passes and sources include opensearch | | **Empty results** | Check `OPENSEARCH_INDEX_PATTERN` and index permissions | | **Agent asks you to re-run setup** | No cluster URL is saved, or the cluster refused the saved credentials. Run `opensre integrations setup opensearch` again | ## Security * Prefer least-privilege read users / API keys. * Do not use the master user for OpenSRE. * Store credentials out of source control. # PostgreSQL Source: https://opensre.com/docs/integrations/databases/postgresql Connect PostgreSQL so OpenSRE can diagnose database issues ## Overview OpenSRE uses PostgreSQL diagnostics to investigate database issues — checking server health, surfacing slow queries, monitoring replication status, and analyzing table statistics. All queries are read-only SELECTs. ## Prerequisites * PostgreSQL 10+ (12+ recommended for full `pg_stat_statements` support) * Network access from the OpenSRE environment to your PostgreSQL instance * A read-only user with access to system views ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup postgresql ``` Provide your host, database, and credentials when prompted. ### Option 2: Environment variables ```bash theme={null} POSTGRESQL_HOST=your-postgresql-host POSTGRESQL_PORT=5432 POSTGRESQL_DATABASE=your-database POSTGRESQL_USERNAME=opensre_readonly POSTGRESQL_PASSWORD=your-password POSTGRESQL_SSL_MODE=prefer # prefer, require, or disable ``` | Variable | Default | Description | | - | - | - | | `POSTGRESQL_HOST` | — | **Required.** PostgreSQL hostname or IP | | `POSTGRESQL_PORT` | `5432` | PostgreSQL port | | `POSTGRESQL_DATABASE` | — | **Required.** Target database | | `POSTGRESQL_USERNAME` | `postgres` | Username | | `POSTGRESQL_PASSWORD` | *(empty)* | Password | | `POSTGRESQL_SSL_MODE` | `prefer` | libpq SSL mode: `prefer`, `require`, or `disable` | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "postgresql-prod", "service": "postgresql", "status": "active", "credentials": { "host": "prod-primary.postgres.example.com", "port": 5432, "database": "application_db", "username": "opensre_readonly", "password": "your-password", "ssl_mode": "prefer" } } ] } ``` ## Credentials ### Creating a read-only user ```sql theme={null} CREATE USER opensre_readonly WITH PASSWORD 'secure-password'; GRANT pg_monitor TO opensre_readonly; GRANT CONNECT ON DATABASE application_db TO opensre_readonly; \c application_db GRANT USAGE ON SCHEMA public TO opensre_readonly; GRANT SELECT ON ALL TABLES IN SCHEMA public TO opensre_readonly; ``` > [!NOTE] > > `pg_monitor` (PostgreSQL 10+) grants read access to monitoring views including `pg_stat_activity`, `pg_stat_replication`, and `pg_stat_statements` without superuser privileges. ### Enabling slow query tracking Add to `postgresql.conf`: ```ini theme={null} shared_preload_libraries = 'pg_stat_statements' pg_stat_statements.track = all ``` Restart PostgreSQL, then: ```sql theme={null} CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` ### Quick local test with Docker ```bash theme={null} docker run -d --name postgres-dev -p 127.0.0.1:5433:5432 \ -e POSTGRES_PASSWORD=verifypass \ -e POSTGRES_DB=app_db \ postgres:16 -c shared_preload_libraries=pg_stat_statements -c pg_stat_statements.track=all i=0 until docker exec postgres-dev pg_isready -U postgres > /dev/null 2>&1; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: PostgreSQL never became ready" >&2; exit 1; } sleep 2 done docker exec postgres-dev psql -U postgres -d app_db -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements;" docker exec postgres-dev psql -U postgres -d app_db -c " CREATE TABLE orders (id INT PRIMARY KEY, amount DECIMAL(10,2)); INSERT INTO orders VALUES (1, 42.50), (2, 17.00); " docker exec postgres-dev psql -U postgres -d app_db -c "SELECT pg_sleep(2);" > /dev/null ``` ```bash theme={null} export POSTGRESQL_HOST=127.0.0.1 export POSTGRESQL_PORT=5433 export POSTGRESQL_DATABASE=app_db export POSTGRESQL_USERNAME=postgres export POSTGRESQL_PASSWORD=verifypass export POSTGRESQL_SSL_MODE=disable ``` Verify: ```bash theme={null} opensre integrations verify postgresql ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL postgresql │ local env │ ✓ passed │ Connected to PostgreSQL 16.15 │ │ │ target database: app_db. ``` Use a temporary integration-store path so saved integrations cannot override the demo environment variables: ```bash theme={null} export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-postgresql-demo.XXXXXX)" export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json" ``` The file does not need to exist. OpenSRE treats the missing temporary store as empty and resolves PostgreSQL from the exported variables above. Now ask the agent about the slow query: ```bash theme={null} opensre ``` Ask: *Does the app\_db PostgreSQL instance have any slow queries?* Against this exact local instance the agent calls all 6 registered tools — slow queries, server status, current queries, lock status, table stats, and replication status — and correctly identifies the intentional `SELECT pg_sleep($1)` call (\~2016 ms average) as the only slow query, with every other health indicator normal. Teardown: ```bash theme={null} docker rm -f postgres-dev rm -rf "$OPENSRE_DEMO_STORE_DIR" unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH unset POSTGRESQL_HOST POSTGRESQL_PORT POSTGRESQL_DATABASE unset POSTGRESQL_USERNAME POSTGRESQL_PASSWORD POSTGRESQL_SSL_MODE ``` ## Tools | Tool | What it does | | - | - | | `get_postgresql_server_status` | Version, uptime, connection counts, commit/rollback rates, buffer cache hit ratio | | `get_postgresql_current_queries` | Active queries longer than a threshold (default 1s), with PID, user, wait event, truncated SQL | | `get_postgresql_slow_queries` | Highest mean execution time from `pg_stat_statements` (needs the extension) | | `get_postgresql_table_stats` | Per-table insert/update/delete/live/dead tuples, scan ratios, vacuum/analyze times, sizes | | `get_postgresql_lock_status` | Lock / blocking information | | `get_postgresql_replication_status` | Primary vs replica (`pg_is_in_recovery()`), WAL position, per-replica lag | ## Verify ```bash theme={null} opensre integrations verify postgresql ``` Alias: `postgres`. Expected output: ``` Service: postgresql Status: passed Detail: Connected to PostgreSQL 16.1; target database: application_db ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused** | Check host, port, firewall, `listen_addresses`, and `pg_hba.conf` | | **Authentication failed** | Confirm username/password and `pg_hba.conf` auth method (`md5`, `scram-sha-256`) | | **SSL error** | Try `POSTGRESQL_SSL_MODE=disable` or `require` | | **Permission denied on pg\_stat\_activity** | Grant `pg_monitor` to the OpenSRE user | | **pg\_stat\_statements not found** | Add to `shared_preload_libraries`, restart, then `CREATE EXTENSION` | | **Replication shows replica, not primary** | Expected — connect to the primary for lag details | ## Security * Use a **dedicated read-only user** with `pg_monitor` — avoid superuser credentials. * Enable **SSL** (`POSTGRESQL_SSL_MODE=require`) in production. * Prefer `scram-sha-256` in `pg_hba.conf`. * Store credentials in `.env` or your secret manager — not in source control. * Rotate credentials periodically. # AWS RDS Source: https://opensre.com/docs/integrations/databases/rds Investigate AWS RDS instance health and recent events during incidents ## Overview OpenSRE uses AWS RDS to investigate database instance health and surface recent operational events — failovers, maintenance windows, parameter changes, and backup activity — when you ask about a managed RDS database. All RDS API calls are read-only and routed through the shared `aws_sdk_client` allowlist, so the integration cannot mutate your RDS resources. ## Prerequisites * AWS credentials configured per the [AWS integration](/docs/integrations/cloud/aws) (role ARN recommended) * An RDS DB instance you want OpenSRE to investigate * IAM permissions for the two RDS describe actions listed below ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup rds ``` ### Option 2: Environment variables ```bash theme={null} RDS_DB_INSTANCE_IDENTIFIER=prod-orders-db AWS_REGION=us-east-1 ``` | Variable | Default | Description | | - | - | - | | `RDS_DB_INSTANCE_IDENTIFIER` | — | **Required.** The DB instance identifier OpenSRE should investigate | | `AWS_REGION` | `us-east-1` | AWS region the instance lives in | | `RDS_REGION` | `us-east-1` | Fallback used only when `AWS_REGION` is not set | Region resolution order (highest priority first): 1. `region` field on the source dict (when configured via the integrations store) 2. `AWS_REGION` environment variable 3. `RDS_REGION` environment variable 4. `us-east-1` (default) ## Credentials The integration only needs two read-only RDS actions on the same IAM role or user used for the [AWS integration](/docs/integrations/cloud/aws): ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "rds:DescribeDBInstances", "rds:DescribeEvents" ], "Resource": "*" } ] } ``` If you already use the AWS managed `ReadOnlyAccess` policy, both actions are covered. ## Tools | Tool | AWS API call | What it returns | | - | - | - | | `describe_rds_instance` | `rds:DescribeDBInstances` | Status, engine + version, instance class, Multi-AZ, endpoint, storage, AZ, backup window | | `describe_rds_events` | `rds:DescribeEvents` | Recent events (failovers, maintenance, parameter changes, backups). Defaults to last 60 minutes; bounded to 20160 minutes (14 days) | Both tools become available whenever `rds.db_instance_identifier` is present in the resolved sources. ### Use cases * Verifying RDS instance status (`available`, `modifying`, `failed`) * Detecting Multi-AZ failover events around an incident timestamp * Tracing recent maintenance, parameter group changes, or backup activity that may correlate with the incident ## Verify There is no dedicated `opensre integrations verify rds` target today. Confirm AWS credentials work (see [AWS](/docs/integrations/cloud/aws)), then start `opensre` and ask, e.g. *"What's the status of prod-orders-db? Any recent failovers?"* ## Troubleshooting | Symptom | Fix | | - | - | | **AccessDenied on `rds:DescribeDBInstances`** | Add the IAM policy above to the role or user used by the AWS integration | | **DBInstanceNotFound** | Confirm `RDS_DB_INSTANCE_IDENTIFIER` matches an instance in `AWS_REGION` | | **Tool reports the wrong region** | Check the region resolution order above for a stale store `region` or wrong `AWS_REGION` | ## Security * Use read-only IAM (`Describe*` only). * Prefer role ARN / least privilege over long-lived keys. * Store identifiers and credentials out of source control. # Redis Source: https://opensre.com/docs/integrations/databases/redis Connect Redis so OpenSRE can diagnose cache, queue, and session issues during incidents ## Overview OpenSRE uses Redis diagnostics to investigate cache and key-value store alerts — checking memory pressure and eviction rates, surfacing slow commands, monitoring replication lag, and inspecting key counts and TTLs. Works with Redis 5+ and compatible servers such as Valkey. ## Prerequisites * Redis 5.0+ (or Valkey) * Network access from the OpenSRE environment to your Redis instance * Credentials, if authentication (`requirepass` or ACLs) is enabled ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup redis ``` ### Option 2: Environment variables ```bash theme={null} REDIS_HOST=localhost REDIS_PORT=6379 REDIS_USERNAME= REDIS_PASSWORD= REDIS_DATABASE=0 REDIS_SSL=false ``` | Variable | Default | Description | | - | - | - | | `REDIS_HOST` | — | **Required.** Redis hostname or IP | | `REDIS_PORT` | `6379` | Redis port | | `REDIS_USERNAME` | *(empty)* | ACL username (Redis 6+); leave blank for password-only auth | | `REDIS_PASSWORD` | *(empty)* | Password (`requirepass` or ACL) | | `REDIS_DATABASE` | `0` | Database number to inspect | | `REDIS_SSL` | `false` | Connect using TLS | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "redis-prod", "service": "redis", "status": "active", "credentials": { "host": "cache.example.net", "port": 6379, "username": "", "password": "s3cret", "db": 0, "ssl": true } } ] } ``` ## Credentials * **Password only** (`requirepass`): set `REDIS_PASSWORD` and leave `REDIS_USERNAME` blank. * **ACL user** (Redis 6+): set both `REDIS_USERNAME` and `REDIS_PASSWORD`. * **No auth**: leave both blank (development only). Set `REDIS_SSL=true` for managed Redis / TLS endpoints (TLS is off by default). Confirm the server has TLS enabled (for example `tls-port 6379`). ## Tools | Tool | What it does | | - | - | | `get_redis_server_info` | Memory, clients, stats, eviction | | `get_redis_client_list` | Connected clients | | `get_redis_slowlog` | Slow command log | | `get_redis_replication` | Replication role / lag | | `get_redis_latency_doctor` | Latency doctor output | | `get_redis_list_depth` | List key depth | | `scan_redis_keys` | Non-blocking `SCAN` (never `KEYS`), hard-capped at 10k keys | ## Verify ```bash theme={null} opensre integrations verify redis ``` Alias: `valkey`. ### Local Docker verification Use this disposable instance to verify Redis without external credentials. The diagnostic settings make the slow log and latency tools return useful sample data; do not copy them to a production server. ```bash theme={null} docker run --rm --detach --name opensre-redis \ --publish 127.0.0.1:6379:6379 \ redis:7-alpine redis-server \ --latency-monitor-threshold 1 \ --slowlog-log-slower-than 0 \ --slowlog-max-len 128 \ --enable-debug-command yes ``` Wait for Redis, then seed two keys and one latency event: ```bash theme={null} until docker exec opensre-redis redis-cli ping; do sleep 1; done docker exec opensre-redis redis-cli RPUSH opensre:jobs job-1 job-2 job-3 docker exec opensre-redis redis-cli SET opensre:service:status degraded EX 3600 docker exec opensre-redis redis-cli DEBUG SLEEP 0.02 ``` From a source checkout, configure the local connection with `uv run opensre integrations setup redis` and use these answers: | Prompt | Answer | | - | - | | Host | `127.0.0.1` | | Port | `6379` | | Username | leave blank | | Password | leave blank | | Database number | `0` | | Use TLS | `false` | Verify the connection: ```bash theme={null} uv run opensre integrations verify redis ``` The result should report a successful connection to Redis 7 on database 0. To exercise `scan_redis_keys` through an agent turn, run: ```bash theme={null} uv run opensre ask --allowed-tool scan_redis_keys \ "Use scan_redis_keys with pattern opensre:* and sample_limit 10. Report each key's type and TTL." ``` The result should include the `opensre:jobs` list and the expiring `opensre:service:status` string. Stop the disposable instance when finished: ```bash theme={null} docker stop opensre-redis ``` This local instance has no password. It binds only to loopback, and the recipe does not write a secret to the repository. ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused** | Check host, port, firewall | | **NOAUTH / WRONGPASS** | Set password and optional ACL username | | **TLS handshake failed** | Set `REDIS_SSL=true` for TLS endpoints | | **Key scan truncated** | Expected at 10k keys — narrow the pattern | ## Security * Prefer ACL users with least privilege over `requirepass` as root. * Enable TLS for managed / production Redis. * Store credentials out of source control. # Snowflake Source: https://opensre.com/docs/integrations/databases/snowflake Connect Snowflake so OpenSRE can query data warehouse and investigate analytics issues ## Overview OpenSRE connects to Snowflake to help investigate analytics pipeline issues — querying warehouse metadata and query history to surface failed queries, long-running statements, and warehouse usage patterns. Authentication uses a **programmatic access token** (Bearer). A password field may appear in config schemas but is ignored at runtime — `SNOWFLAKE_TOKEN` is required. ## Prerequisites * A Snowflake account (trial or production) * A configured warehouse * A programmatic access token for API authentication * A database and schema to query * Network access from your OpenSRE environment to Snowflake ## Setup There is no dedicated `opensre integrations setup snowflake` wizard today. Configure via environment variables or the persistent store, then verify. ### Option 1: Environment variables ```bash theme={null} SNOWFLAKE_ACCOUNT_IDENTIFIER=xy12345.us-east-1 SNOWFLAKE_USER=opensre_user SNOWFLAKE_TOKEN=your_programmatic_access_token SNOWFLAKE_PASSWORD=your_password SNOWFLAKE_WAREHOUSE=COMPUTE_WH SNOWFLAKE_DATABASE=analytics SNOWFLAKE_SCHEMA=public SNOWFLAKE_ROLE=viewer ``` | Variable | Default | Description | | - | - | - | | `SNOWFLAKE_ACCOUNT_IDENTIFIER` | — | **Required.** Snowflake account identifier | | `SNOWFLAKE_ACCOUNT` | — | Alternative account variable | | `SNOWFLAKE_USER` | — | Snowflake username | | `SNOWFLAKE_TOKEN` | — | **Required.** Programmatic access token | | `SNOWFLAKE_PASSWORD` | — | Optional; ignored for SQL API auth today | | `SNOWFLAKE_WAREHOUSE` | — | Warehouse name (for example `COMPUTE_WH`) | | `SNOWFLAKE_DATABASE` | — | Default database | | `SNOWFLAKE_SCHEMA` | — | Default schema | | `SNOWFLAKE_ROLE` | — | Role with appropriate permissions | > [!NOTE] > > OpenSRE requires `SNOWFLAKE_ACCOUNT_IDENTIFIER` (or `SNOWFLAKE_ACCOUNT`) and `SNOWFLAKE_TOKEN` to activate the integration. Generate a programmatic access token in Snowflake under your user's security settings. ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "snowflake-prod", "service": "snowflake", "status": "active", "credentials": { "account": "xy12345.us-east-1", "user": "opensre_user", "token": "your_programmatic_access_token", "password": "your_password", "warehouse": "COMPUTE_WH", "database": "analytics", "schema": "public", "role": "viewer" } } ] } ``` ## Credentials ### Finding your account identifier 1. Open the Snowflake web interface 2. Open your account or profile settings 3. Locate your account identifier 4. Use the full value for `SNOWFLAKE_ACCOUNT_IDENTIFIER` Examples: ```text theme={null} xy12345 xy12345.us-east-1 myorg-myaccount myorg-myaccount.eu-west-1 ``` > [!NOTE] > > Use the full account identifier shown by Snowflake. Do not remove region or organization information unless your Snowflake deployment documentation explicitly instructs you to do so. ### Best practice: dedicated role ```sql theme={null} CREATE ROLE opensre_viewer; GRANT SELECT ON ALL TABLES IN DATABASE analytics TO ROLE opensre_viewer; GRANT MONITOR ON WAREHOUSE COMPUTE_WH TO ROLE opensre_viewer; CREATE USER opensre_user PASSWORD = 'strong_secure_password'; GRANT ROLE opensre_viewer TO USER opensre_user; ``` Then issue a programmatic access token for that user and use it in OpenSRE. ## Tools | Tool | What it does | | - | - | | `query_snowflake_history` | Bounded, read-only query-history evidence (failed / long-running queries, warehouse usage) | ## Verify ```bash theme={null} opensre integrations verify snowflake ``` Expected output: ``` Service: snowflake Status: passed Detail: Configured for Snowflake account xy12345.us-east-1 ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Missing token credentials** | Set `SNOWFLAKE_TOKEN`. Password alone does not activate the integration | | **Invalid account identifier** | Use the full identifier from Snowflake (including region/org suffix if shown) | | **Warehouse suspended** | Resume the warehouse or grant `OPERATE` on the warehouse to the OpenSRE role | | **Insufficient privileges** | Grant `MONITOR` on the warehouse and `SELECT` on `ACCOUNT_USAGE.QUERY_HISTORY` | ## Security * Use a dedicated Snowflake user for OpenSRE * Grant only the permissions the agent needs * Rotate credentials regularly * Monitor access through Snowflake audit logs # Supabase Source: https://opensre.com/docs/integrations/databases/supabase Connect Supabase so OpenSRE can diagnose database, API, and authentication issues during incidents ## Overview When something goes wrong with your Supabase-backed database, APIs, or authentication flows, OpenSRE can use your project configuration to investigate issues, inspect resources, and gather operational insights. ## Prerequisites * A Supabase project * Your Supabase Project URL * A Supabase Service Role Key * Permissions to access project resources ## Setup There is no dedicated `opensre integrations setup supabase` wizard today. Configure via environment variables or the persistent store, then verify. ### Option 1: Environment variables ```bash theme={null} SUPABASE_URL=https://your-project.supabase.co SUPABASE_SERVICE_KEY=your_service_role_key ``` | Variable | Description | | - | - | | `SUPABASE_URL` | Supabase project URL | | `SUPABASE_SERVICE_KEY` | Supabase service role key | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "supabase-prod", "service": "supabase", "status": "active", "credentials": { "url": "https://your-project.supabase.co", "service_key": "your_service_role_key" } } ] } ``` ## Credentials 1. Log in to Supabase 2. Open your project 3. Navigate to **Settings** → **API** 4. Copy the **Project URL** 5. Copy the **Service Role Key** (not the anon key) > [!NOTE] > > The Service Role Key has elevated privileges. Store it securely and never expose it in client-side applications or public repositories. ## Tools | Tool | What it does | | - | - | | `get_supabase_service_health` | Health of PostgREST, Auth, Storage, and other services | | `get_supabase_storage_buckets` | Storage buckets and config (public/private, size limits, MIME types) | ## Verify ```bash theme={null} opensre integrations verify supabase ``` Expected output: ``` Service: supabase Status: passed Detail: Connected to Supabase project ... ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Confirm the **Service Role Key** (not the anon key) is set in `SUPABASE_SERVICE_KEY` | | **Invalid project URL** | Use the Project URL from Settings → API (`https://your-project.supabase.co`) | | **Health check shows degraded service** | Check the Supabase status page and project dashboard for the affected service | | **Storage bucket not found** | Verify the bucket name the agent queried matches an existing bucket | ## Security * Use the **service role key** only on the server side — never in client apps or public repos. * Rotate the service role key if it is ever exposed. * Prefer a dedicated Supabase project or restricted key for OpenSRE when possible. # Alertmanager Source: https://opensre.com/docs/integrations/incidents/alertmanager Connect Alertmanager so OpenSRE can surface firing alerts and active silences ## Overview OpenSRE queries Alertmanager for firing, silenced, and inhibited alerts so the agent can correlate the alert you ask about with other concurrent signals. ## Prerequisites * Alertmanager v0.20+ reachable from the machine running OpenSRE * The Alertmanager URL (for example `http://alertmanager.monitoring.svc:9093`) * Credentials if your instance uses authentication (bearer token **or** basic auth — not both) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup alertmanager ``` The wizard asks for: 1. **Alertmanager URL** — base URL of your Alertmanager instance 2. **Authentication method** — one of: * **None** — unauthenticated instances on an internal network * **Bearer token** — reverse proxy that accepts a token * **Basic auth** — username and password ### Option 2: Environment variables ```bash theme={null} ALERTMANAGER_URL=http://alertmanager.monitoring.svc:9093 # Bearer token auth (optional) ALERTMANAGER_BEARER_TOKEN=your-token # Basic auth (optional) ALERTMANAGER_USERNAME=admin ALERTMANAGER_PASSWORD=secret ``` | Variable | Default | Description | | - | - | - | | `ALERTMANAGER_URL` | — | **Required.** Base URL of your Alertmanager instance | | `ALERTMANAGER_BEARER_TOKEN` | — | Bearer token for reverse-proxy auth | | `ALERTMANAGER_USERNAME` | — | Basic auth username | | `ALERTMANAGER_PASSWORD` | — | Basic auth password | Use **at most one** auth method. Setting both a bearer token and basic auth credentials is rejected. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "alertmanager-prod", "service": "alertmanager", "status": "active", "credentials": { "base_url": "http://alertmanager.monitoring.svc:9093", "bearer_token": "", "username": "", "password": "" } } ] } ``` ## Credentials Alertmanager itself often has no built-in API key UI. Credentials usually come from the reverse proxy or basic auth in front of the instance. Prefer a **read-only** token or user — OpenSRE only reads alerts and silences. ## Tools | Tool | What it does | | - | - | | `alertmanager_alerts` | Queries `/api/v2/alerts` for firing, silenced, and inhibited alerts | | `alertmanager_silences` | Queries `/api/v2/silences` for active silences | ### `alertmanager_alerts` Typical uses: * Discover other alerts firing at the same time as the triggering alert * Check whether the triggering alert is already silenced or inhibited * Understand blast radius from active alert labels * Correlate Prometheus alerts (OOM, latency, error-rate) into one timeline Name the alert in your question — the agent filters on the `alertname` label so results stay scoped to the incident. ### `alertmanager_silences` Typical uses: * See whether a noisy alert was silenced on purpose * Surface maintenance windows that overlap the incident * Avoid false root-cause conclusions from suppressed alerts ## Verify ```bash theme={null} opensre integrations verify alertmanager ``` Expected output: ``` Service: alertmanager Status: passed Detail: Connected to Alertmanager at http://alertmanager.monitoring.svc:9093; cluster status: ready. ``` ### Local Docker verification Start a disposable Alertmanager without external credentials: ```bash theme={null} docker run --rm --detach --name opensre-alertmanager \ --publish 127.0.0.1:9093:9093 \ prom/alertmanager:v0.28.1 || exit 1 for attempt in $(seq 1 30); do curl --fail --silent http://127.0.0.1:9093/-/ready && break if [ "$attempt" -eq 30 ]; then docker logs opensre-alertmanager exit 1 fi sleep 1 done ``` Seed one firing alert and one active silence so both tools return real data: ```bash theme={null} curl --fail --silent --request POST \ --header "Content-Type: application/json" \ --data '[{"labels":{"alertname":"OpenSREHighLatency","service":"checkout","severity":"critical"},"annotations":{"summary":"Checkout p99 latency exceeds 2 seconds"}}]' \ http://127.0.0.1:9093/api/v2/alerts starts_at=$(uv run python -c 'from datetime import UTC, datetime; print(datetime.now(UTC).isoformat())') ends_at=$(uv run python -c 'from datetime import UTC, datetime, timedelta; print((datetime.now(UTC) + timedelta(hours=1)).isoformat())') curl --fail --silent --request POST \ --header "Content-Type: application/json" \ --data "{\"matchers\":[{\"name\":\"alertname\",\"value\":\"OpenSREMaintenance\",\"isRegex\":false}],\"startsAt\":\"${starts_at}\",\"endsAt\":\"${ends_at}\",\"createdBy\":\"opensre-local-verification\",\"comment\":\"Disposable local verification silence\"}" \ http://127.0.0.1:9093/api/v2/silences ``` From a source checkout, run `uv run opensre integrations setup alertmanager`. Enter `http://127.0.0.1:9093` as the URL and select **None** for authentication, then verify the connection: ```bash theme={null} uv run opensre integrations verify alertmanager ``` The verifier should report cluster status `ready`. Exercise both tools through an agent turn: ```bash theme={null} uv run opensre ask \ --allowed-tool alertmanager_alerts \ --allowed-tool alertmanager_silences \ "Report the firing Alertmanager alert and the active silence." ``` The result should include the `OpenSREHighLatency` alert and the `OpenSREMaintenance` silence. Stop the disposable instance when finished: ```bash theme={null} docker stop opensre-alertmanager ``` This unauthenticated instance binds only to loopback and is for local verification only. ## Troubleshooting | Symptom | Fix | | - | - | | **Status: missing** | Set `ALERTMANAGER_URL` or run `opensre integrations setup alertmanager` | | **Connection refused** | Confirm the URL is reachable from this host; check firewall rules | | **401 Unauthorized** | Supply a bearer token **or** basic auth credentials (not both) | | **Both auth methods set** | Clear either `ALERTMANAGER_BEARER_TOKEN` or the basic-auth pair | | **SSL error** | Trust the CA cert, or use an `http://` URL for internal instances | | **Empty alert list** | Normal when nothing is firing — check the Alertmanager UI | ## Security * Prefer a **read-only** reverse-proxy token — OpenSRE never writes to Alertmanager. * Store credentials in `.env` or your secret manager — not in source control. * For internal Kubernetes deployments, prefer network isolation over exposing credentials. # Fix a Sentry issue Source: https://opensre.com/docs/integrations/incidents/fix-sentry-issue Paste a Sentry issue URL and have the Pi coding agent propose a fix — as a reviewable diff or a pull request. ## Overview The **Sentry issue-fix tool** lets you point OpenSRE at a [Sentry](https://sentry.io) issue and have the [Pi](https://pi.dev) coding agent propose a fix. OpenSRE fetches the issue context, runs Pi in your current repository, and returns a summary plus the git diff. By default it **only edits the working tree** so you can review the diff. When you ask it to `open_pr` (and enable the ship switch), it also commits the fix to a fresh branch, pushes it, and opens a **pull request** — it **never pushes to your base/`main` branch**. This is a **tool page**, not a full integration setup page. It depends on the [Sentry](/docs/integrations/incidents/sentry) integration plus local Pi and opt-in env gates. > [!WARNING] > > This is a **mutating** tool — it changes files on disk and can open a PR. It is > **disabled by default**: the fix step needs `PI_ISSUE_FIX_ENABLED=1`, Sentry > configured, and the Pi CLI installed. Opening a PR needs a **second** switch, > `PI_ISSUE_FIX_SHIP_ENABLED=1`, plus a GitHub token. Enable each deliberately. ## Prerequisites * Sentry configured (org + auth token) — see [Sentry](/docs/integrations/incidents/sentry) * Pi coding agent CLI installed and authenticated * `PI_ISSUE_FIX_ENABLED=1` to allow the fix step * For PRs: `PI_ISSUE_FIX_SHIP_ENABLED=1` and a GitHub connection in the OpenSRE app with `repo` / `pull_request` write access ## Setup 1. Configure Sentry and install Pi: ```bash theme={null} export SENTRY_ORG_SLUG=your-org export SENTRY_AUTH_TOKEN=... # token with issue read access npm i -g @earendil-works/pi-coding-agent ``` 2. Turn the tool on: ```bash theme={null} export PI_ISSUE_FIX_ENABLED=1 export PI_CODING_MODEL=anthropic/claude-haiku-4-5 # optional ``` 3. (Optional) Allow OpenSRE to open a pull request: ```bash theme={null} export PI_ISSUE_FIX_SHIP_ENABLED=1 opensre integrations setup github # connect in the app with PR-create access ``` | Env var | What it does | | - | - | | `PI_ISSUE_FIX_ENABLED` | Opt-in for the fix step. Set to `1` to enable. Off by default. | | `PI_ISSUE_FIX_SHIP_ENABLED` | Second opt-in required to open a PR (`open_pr`). Off by default. | | GitHub app connection | Requires permission to open pull requests in the repository. | | `SENTRY_ORG_SLUG` / `SENTRY_AUTH_TOKEN` | Sentry org + token (`SENTRY_URL` for self-hosted). | | `PI_CODING_MODEL` | Pi model used for the fix (shared with the Pi coding tool). | | `PI_CODING_WORKSPACE` | Repository Pi edits. Defaults to the current directory. | | `PI_CODING_TIMEOUT_SECONDS` | Per-run timeout (default 600, clamped 60–1800). | ## Credentials | System | What you need | | - | - | | Sentry | Org slug + auth token with issue read access | | Pi | Installed/authenticated Pi CLI | | GitHub (PR only) | Token with permission to create PRs on the target repo | Your Sentry token is **never** sent to Pi. OpenSRE builds a short masked task (title, error, culprit, location) for the coding agent. ## Tools | Tool / action | What it does | | - | - | | `fix_sentry_issue` | Fetch issue context, run Pi, return summary + diff (optional PR) | | Shell action `fix_sentry_issue_start` | Interactive-shell path for natural-language fix requests | | Parameter | What it does | | - | - | | `sentry_url` | The Sentry issue URL to fix (**required**) | | `open_pr` | When `true`, commit to a fresh branch and open a PR. Defaults to `false` (diff only) | ### Using it from the interactive shell Once enabled, ask in plain language in `opensre`: ``` fix this sentry issue https://your-org.sentry.io/issues/12345/ and open a pull request ``` Say "open a pull request" (or "ship it") to open a PR; leave it out to get the diff only. Use a *fix* verb — "diagnose" / "analyze" requests are answered with the read-only Sentry tools instead. ## Verify 1. `opensre integrations verify sentry` — Sentry REST config works 2. Pi CLI available on `PATH` 3. Env gates set (`PI_ISSUE_FIX_ENABLED=1`, and ship env if you need PRs) ## Troubleshooting | Symptom | Fix | | - | - | | Tool disabled | Set `PI_ISSUE_FIX_ENABLED=1` | | PR not opened | Set `PI_ISSUE_FIX_SHIP_ENABLED=1` and connect GitHub in the OpenSRE app | | Unsupported URL | Use a supported Sentry issue URL shape (see Extras) | | Sentry unconfigured | Configure [Sentry](/docs/integrations/incidents/sentry) first | | Pi missing | Install `@earendil-works/pi-coding-agent` and authenticate | | Fix OK but PR fails | You still get `diff` + `changed_files` plus `error_kind` to ship manually | ## Security * Dual opt-in by design: fix and ship are separate switches, both off by default. * OpenSRE **never** commits to or force-pushes your base/`main` branch. * Sentry tokens stay on the OpenSRE side; Pi receives masked issue context only. * Review the diff before merging any PR. ## Extras ### How it works 1. You paste a Sentry issue URL and ask OpenSRE to fix it. 2. OpenSRE resolves the issue and builds a short, **masked** task for Pi. 3. Pi edits the current repository to implement the fix. 4. You get `success`, a `summary`, `changed_files`, and the `diff`. 5. If you asked for `open_pr` (and shipping is enabled), OpenSRE commits to a fresh `opensre/sentry-fix--` branch, pushes it, opens a PR into your base branch, and returns `branch_name`, `pr_url`, and `pr_number`. ### Opening a pull request * The fix always lands on a new namespaced branch and is proposed via PR. * The PR body links the Sentry issue and lists changed files. * If the fix succeeds but the PR cannot be opened, you still get the diff. ### Supported URLs * `https://.sentry.io/issues//` * `https://sentry.io/organizations//issues//` * Self-hosted `https://sentry..com/organizations//issues//` ### Notes * Without `open_pr`, nothing is committed or pushed. * Failures return a clear `error_kind` instead of silently doing the wrong thing. # incident.io Source: https://opensre.com/docs/integrations/incidents/incident_io Connect incident.io so OpenSRE can read incident context, updates, and metadata ## Overview OpenSRE can use incident.io as incident context during RCA. The integration can list live incidents, fetch full metadata, read incident updates, and append final findings to the incident summary through the supported incident edit API. ## Prerequisites * incident.io account * API key with read access to incidents and incident updates * Write permission if you want OpenSRE to append findings to incident summaries ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup incident_io ``` Aliases also work: `incident.io`, `incidentio`. ### Option 2: Environment variables ```bash theme={null} INCIDENT_IO_API_KEY=your-api-key INCIDENT_IO_BASE_URL=https://api.incident.io # optional ``` | Variable | Default | Description | | - | - | - | | `INCIDENT_IO_API_KEY` | — | **Required.** incident.io API key | | `INCIDENT_IO_BASE_URL` | `https://api.incident.io` | Optional API URL override for tests or private routing | ## Credentials Create an API key in the incident.io dashboard with read access to incidents and incident updates. Add write permission only if you plan to use `action="append_summary"`. ## Tools | Tool | What it does | | - | - | | `incident_io_incidents` | List, get, read updates/context, or append a summary | Actions on `incident_io_incidents`: | Action | Behavior | | - | - | | `list` | List live incidents when no incident ID is present | | `get` | Fetch incident metadata (`GET /v2/incidents/{id}`) | | `updates` | Read incident updates | | `context` | Combine metadata + updates for RCA context | | `append_summary` | Append findings to the incident summary (write-back) | When an alert includes an incident.io incident ID or URL, OpenSRE can use `action="context"` to load metadata and updates for that incident. ## Verify ```bash theme={null} opensre integrations verify incident_io ``` Verification performs a minimal `GET /v2/incidents` request (page size 1). Expected detail: ``` Connected to incident.io; API key accepted. ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Check the API key and its scopes | | **No incident context** | Confirm the alert includes an incident.io ID or URL, or use `action="list"` | | **Write-back fails** | Confirm the key can edit incidents; OpenSRE does not create native timeline events | ## Security * Prefer a read-only key unless you need `append_summary`. * Store `INCIDENT_IO_API_KEY` in `.env` or your secret manager — not in source control. ## Extras ### Write-back details OpenSRE does not create native timeline events. Write-back uses: ```text theme={null} POST /v2/incidents/{id}/actions/edit ``` Use `action="append_summary"` only after the RCA has useful findings or next steps ready to publish. # PagerDuty Source: https://opensre.com/docs/integrations/incidents/pagerduty Connect PagerDuty so OpenSRE can read incidents, on-call schedules, and service topology ## Overview OpenSRE queries PagerDuty for active incidents, timelines, on-call responders, and service / escalation-policy configuration so the agent can correlate incident management data with infrastructure events. ## Prerequisites * PagerDuty account * REST API key with read access ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup pagerduty ``` Provide your API key when prompted. ### Option 2: Environment variables ```bash theme={null} PAGERDUTY_API_KEY=your-api-key PAGERDUTY_BASE_URL=https://api.pagerduty.com # optional ``` | Variable | Default | Description | | - | - | - | | `PAGERDUTY_API_KEY` | — | **Required.** PagerDuty REST API v2 key | | `PAGERDUTY_BASE_URL` | `https://api.pagerduty.com` | Override only for EU or custom instances | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "pagerduty-prod", "service": "pagerduty", "status": "active", "credentials": { "api_key": "your-api-key" } } ] } ``` ## Credentials 1. In PagerDuty, go to **Integrations** → **Developer Tools** → **API Access Keys** 2. Click **Create New API Key** 3. Add a description (for example `opensre read-only`) 4. Copy the key immediately — it is shown only once > [!NOTE] > > PagerDuty API keys are account-level. Use a read-only key for least privilege. ## Tools | Tool | Description | | - | - | | `pagerduty_incidents` | List and search incidents (status, urgency, service, time range) | | `pagerduty_incident_detail` | Full incident details and activity timeline (log entries) | | `pagerduty_oncall` | Current on-call responders by escalation policy | | `pagerduty_services` | Services with escalation policies, integrations, and alert routing | When investigating an incident, OpenSRE may use these tools to: * Find incidents around the alert time window * Identify who was on-call and how quickly incidents were acknowledged * Map services and escalation policies involved * Correlate PagerDuty timelines with Grafana / Datadog metrics ## Verify ```bash theme={null} opensre integrations verify pagerduty ``` Expected output: ``` Service: pagerduty Status: passed Detail: Connected to PagerDuty; API key accepted. ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Check the API key — use a valid REST API v2 key | | **403 Forbidden** | The key may lack required permissions; recreate with full read access | | **429 Rate Limited** | PagerDuty limits apply (about 960 requests/min); reduce query frequency | | **No incidents returned** | Check time-range filters — default returns recent incidents only | ## Security * Use a **read-only API key** — OpenSRE does not need to modify PagerDuty resources. * Store the API key in `.env` or your secret manager — not in source control. * Rotate API keys periodically via PagerDuty Developer Tools. # Runbook-guided investigations Source: https://opensre.com/docs/integrations/incidents/runbook-guided-investigations Configure trusted GitHub Markdown runbooks that steer OpenSRE incident investigations OpenSRE can load an organization-owned runbook before choosing diagnostic steps. The same behavior is available in the interactive shell and gateway chats, including Discord. The first release supports GitHub-hosted Markdown. Its source contract is provider-neutral so later providers can use the same selection and provenance rules. ## V1 behavior OpenSRE selects a runbook in this order: 1. An explicit runbook URL supplied by the user or returned by an alert tool. 2. An exact manifest match using `alertname`, `service`, and labels. The selected document is fetched through the existing verified GitHub integration and pinned to the commit returned by GitHub. OpenSRE includes that immutable revision in the investigation evidence. Runbooks steer the investigation; they do not bypass safety policy. OpenSRE treats their contents as guidance, verifies claims with live tools, and keeps the normal approval path for mutations. ## Configure a source Connect and verify GitHub first: ```bash theme={null} opensre integrations setup github opensre integrations verify github ``` Register the repository. A manifest is optional when you only use explicit URLs. ```bash theme={null} opensre runbooks add github \ --name platform-runbooks \ --repo acme/operations \ --ref main \ --manifest .opensre/runbooks.yaml opensre runbooks verify platform-runbooks opensre runbooks list ``` Use `opensre runbooks remove platform-runbooks` to remove it. The source settings are stored with other local OpenSRE settings, while credentials remain owned by the GitHub integration. ## Add a manifest The V1 manifest is strict YAML. Document paths must be repository-relative Markdown files. Every entry needs an `alertname` or `service` match. ```yaml theme={null} version: 1 runbooks: - id: checkout-high-latency title: Checkout high latency document: runbooks/checkout-high-latency.md match: alertname: CheckoutHighLatency service: checkout labels: severity: critical ``` Matching is exact and deterministic. Alert name plus labels outranks alert name alone, which outranks service alone. More matching labels win within the same tier. OpenSRE reports equally specific matches as ambiguous instead of guessing. ## Try the bundled demo After this version is available on the repository's default branch, register its public demo runbook: ```bash theme={null} opensre runbooks add github \ --name opensre-demo \ --repo Tracer-Cloud/opensre \ --ref main \ --manifest docs/snippets/runbooks/demo-manifest.yaml opensre runbooks verify opensre-demo ``` Start `opensre` and ask: ```text theme={null} Investigate CheckoutHighLatency for service checkout with severity=critical. Use the configured runbook and tell me which checks you can verify. ``` The runbook should resolve to `checkout-high-latency`, show a commit-pinned GitHub URL, and guide the diagnostic sequence. With no monitoring integrations connected, OpenSRE should clearly report which live checks it could not perform instead of inventing data. To test explicit URL precedence, ask with this URL: ```text theme={null} Investigate using https://github.com/Tracer-Cloud/opensre/blob/main/docs/snippets/runbooks/checkout-high-latency.md ``` For Discord, run the gateway with GitHub and Discord connected, then send the same prompt in a DM, mention, or active thread. No Discord-specific runbook configuration is needed. ## Verify a production rollout Test these cases before enabling a source for responders: * A valid explicit URL loads only from the configured repository and ref. * An untrusted repository, unsafe path, or non-Markdown document is rejected. * One exact manifest match loads the expected document and commit revision. * Equally specific entries return candidate IDs and do not choose one. * Missing GitHub access or an invalid manifest produces a safe, actionable failure. * The final investigation distinguishes runbook guidance from observed evidence. V1 does not crawl directories, semantically search documents, execute embedded commands, or support GitLab, local files, object stores, or knowledge bases. Those providers and discovery modes can be added behind the same source contract in later releases. # Sentry Source: https://opensre.com/docs/integrations/incidents/sentry Connect Sentry so OpenSRE can surface error trends and issue details as evidence ## Overview OpenSRE queries Sentry for recent issues, error events, and stack traces so the agent can correlate application errors with infrastructure alerts. This page covers the **REST Sentry integration**. For the broader MCP surface (Seer, traces, replays, and more), see [Sentry (MCP)](/docs/integrations/incidents/sentry-mcp). ## Prerequisites * Sentry account with at least one organization * Auth token with `event:read` scope (Issues lookup) * For uptime watching: also `alerts:read` (or `org:read`) so `opensre sentry uptime` can list monitors ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup sentry ``` Provide your organization slug and auth token when prompted. ### Option 2: Environment variables ```bash theme={null} SENTRY_ORG_SLUG=your-organization-slug SENTRY_AUTH_TOKEN=sntrys_your_token SENTRY_URL=https://sentry.io # optional, for self-hosted Sentry SENTRY_PROJECT_SLUG=my-project # optional, to scope to one project SENTRY_STATS_PERIOD=24h # optional, issue search time window ``` | Variable | Default | Description | | - | - | - | | `SENTRY_ORG_SLUG` | — | **Required.** Your Sentry organization slug | | `SENTRY_AUTH_TOKEN` | — | **Required.** Auth token with `event:read` (add `alerts:read` for uptime watch) | | `SENTRY_URL` | `https://sentry.io` | Override for self-hosted Sentry | | `SENTRY_PROJECT_SLUG` | — | Scope queries to a specific project | | `SENTRY_STATS_PERIOD` | `24h` | Time window for issue searches (for example `24h`, `14d`, `90d`) | > [!NOTE] > > A search returns up to 100 issues per query (Sentry's maximum page size) within > the `SENTRY_STATS_PERIOD` window. Widen the window (for example > `SENTRY_STATS_PERIOD=14d`) to surface older issues. If you expect more issues > than appear, the cause is usually the time window or a `SENTRY_PROJECT_SLUG` > scope — not a hard cap of one page forever. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "sentry-prod", "service": "sentry", "status": "active", "credentials": { "base_url": "https://sentry.io", "organization_slug": "your-org", "auth_token": "sntrys_your_token", "project_slug": "my-project" } } ] } ``` ## Credentials **Recommended: Organization Token** 1. In Sentry, go to **Settings** → **Developer Settings** → **Organization Tokens** 2. Click **Create New Token** 3. Enable the `event:read` scope (and `alerts:read` if you use uptime watch) 4. Copy the token **Alternative: Internal Integration** For broader access, create an Internal Integration under **Settings** → **Developer Settings** → **Internal Integrations**. > [!NOTE] > > The organization slug appears in your Sentry URL: > `https://sentry.io/organizations//` ## Tools | Tool | What it does | | - | - | | `search_sentry_issues` | Search issues in the configured org / project window | | `get_sentry_issue_details` | Fetch details for one issue | | `list_sentry_issue_events` | List error events for an issue | | `list_sentry_uptime_alerts` | List uptime monitor health | | `get_sentry_uptime_digest` | Roll up recent DOWN / RECOVERED transitions for digests | ## Verify ```bash theme={null} opensre integrations verify sentry ``` Expected output: ``` Service: sentry Status: passed Detail: Sentry validated for org your-org; 30 issue(s) in the last 7 days. ``` The count reflects issues seen in the last 7 days (capped at 100, shown as `100+` when it saturates). Use `search_sentry_issues` when investigating an issue to enumerate issues over a custom window. ## Troubleshooting | Symptom | Fix | | - | - | | **403 Forbidden** | Ensure the token has `event:read`; for uptime watch also `alerts:read` | | **Organization not found** | Verify `SENTRY_ORG_SLUG` matches the slug in your Sentry URL | | **Connection refused** | Check `SENTRY_URL` for self-hosted instances | | **No issues returned** | Normal if none exist in the time window — check `SENTRY_PROJECT_SLUG` | | **Fewer issues than the Sentry UI shows** | Widen `SENTRY_STATS_PERIOD` (for example `14d`) and confirm project scope | ## Security * Use an **Organization Token** with only `event:read` (plus `alerts:read` if needed) — not an admin token. * Store the token in `.env` or your secret manager — not in source control. * Rotate tokens periodically. ## Extras ### Morning digest (scheduled) Deliver a daily unresolved-issues summary to Slack, Telegram, or Rocket.Chat. This uses the headless **summarizing-sentry-issues** skill path (tools + LLM via the gateway), not generic `opensre cron` kinds. When an uptime watch schedule is running, the skill also calls `get_sentry_uptime_digest` to include downtime/recovery context. | Command | What it does | | - | - | | `/sentry digest schedule add …` | Same as CLI — schedule from the REPL | | `opensre sentry digest run` | Run once and print the digest to stdout | | `opensre sentry digest schedule add` | Schedule daily delivery (cron + provider + chat) | | `opensre sentry digest schedule list` | List Sentry digest schedules | | `opensre sentry digest schedule run TASK_ID` | Run a scheduled digest immediately | | `opensre sentry digest schedule remove TASK_ID` | Remove a schedule | Example — weekdays at 08:00 London time to Telegram: ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * mon-fri" \ --tz Europe/London \ --provider telegram \ --chat-id "-1001234567890" ``` Example — Slack channel (`C…` member/channel id): ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * mon-fri" \ --tz Europe/London \ --provider slack \ --chat-id C0123ABCD ``` Example — Rocket.Chat (requires token credentials — see [Rocket.Chat](/docs/integrations/messaging/rocketchat); an incoming webhook alone cannot target an explicit `--chat-id`): ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * mon-fri" \ --tz Europe/London \ --provider rocketchat \ --chat-id "#alerts" ``` Optional `--project my-service` scopes the digest to one Sentry project. When an **uptime watch** schedule is running, the digest includes an **Uptime / downtime (last 24h)** section. If no uptime watch history exists, the digest stays Issues-only. The gateway daemon picks up scheduled digests when it is running. If the LLM is unavailable, the run fails with an error. ### Uptime watch (downtime notifications) Poll Sentry **uptime monitors** (Alerts / Uptime — not the Issues digest) and ping Slack, Telegram, or Rocket.Chat only when a monitor transitions to **down** or **recovered**. Quiet polls deliver nothing. | Command | What it does | | - | - | | `opensre sentry uptime check` | List current uptime monitor health once | | `opensre sentry uptime watch add` | Schedule polling (default every 5 minutes) | | `opensre sentry uptime watch list` | List uptime watch schedules | | `opensre sentry uptime watch run TASK_ID` | Run a watch tick now (may notify) | | `opensre sentry uptime watch remove TASK_ID` | Remove a watch schedule | Example — poll every 5 minutes and ping Slack on transitions: ```bash theme={null} opensre sentry uptime watch add \ --cron "*/5 * * * *" \ --tz UTC \ --provider slack \ --chat-id C0123ABCD ``` Requires `alerts:read` (or equivalent) on the Sentry token. `list_sentry_uptime_alerts` exposes the same monitor list in chat. Schedule an uptime watch alongside the morning digest so `summarizing-sentry-issues` can include yesterday's downtime summary via `get_sentry_uptime_digest`. Follow-ups (not in v1): inbound Sentry webhooks and auto-remediation attempts. ### Telemetry knobs (OpenSRE error reporting) These control OpenSRE's own Sentry error reporting — separate from the Issues integration above. | Variable | Default | Description | | - | - | - | | `OPENSRE_SENTRY_DSN` | — | Override the bundled Sentry DSN | | `OPENSRE_SENTRY_DISABLED` | `0` | Set to `1` to disable Sentry entirely | | `OPENSRE_SENTRY_LOGGING_DISABLED` | `0` | Set to `1` to disable automatic forwarding of `logger.error` / `logger.exception` without affecting `capture_exception` | Send one test event: ```bash theme={null} opensre debug sentry ``` For a custom or self-hosted project: ```bash theme={null} OPENSRE_SENTRY_DSN=https://public-key@example.ingest.sentry.io/123 opensre debug sentry ``` ```text theme={null} Sentry DSN host: example.ingest.sentry.io Sentry event ID: Sentry flush sent: yes ``` The event is synthetic and tagged `debug=true`. If telemetry is disabled, the command exits non-zero without sending anything. # Sentry (MCP) Source: https://opensre.com/docs/integrations/incidents/sentry-mcp Connect Sentry's hosted MCP server so OpenSRE can query issues, events, traces, releases, and Seer root-cause analysis ## Overview OpenSRE connects to Sentry's hosted [Model Context Protocol (MCP)](https://mcp.sentry.dev) server. That exposes Sentry products — issues, events, traces, replays, releases, monitors, and Seer AI root-cause analysis — as tools the agent can call. This is distinct from the [Sentry issue integration](/docs/integrations/incidents/sentry), which is a narrower REST client for issue and event lookup. Use MCP when you want the agent to explore Sentry data directly, including Seer-assisted debugging. "Hosted MCP" here means **Sentry's** `mcp.sentry.dev` endpoint — not OpenSRE Cloud. ## Prerequisites * A Sentry account (hosted SaaS, or self-hosted via the `SENTRY_MCP_HOST` override) * A Sentry **user auth token**. Required scopes depend on the skills you use: * `org:read` — read-only inspection of issues, events, traces, releases, monitors (default) * `event:write`, `team:write`, `project:write` — for triage and project-management skills ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup sentry_mcp ``` Select **Sentry (MCP)** when prompted, then paste your user auth token. Setup uses the hosted Streamable HTTP transport; keep the default URL unless you run self-hosted Sentry. To run a local server instead, set `SENTRY_MCP_MODE=stdio` via environment variables. ### Option 2: Environment variables ```bash theme={null} SENTRY_MCP_MODE=streamable-http SENTRY_MCP_URL=https://mcp.sentry.dev/mcp SENTRY_MCP_AUTH_TOKEN=your_sentry_user_auth_token SENTRY_MCP_HOST= # optional, self-hosted Sentry host SENTRY_MCP_ORGANIZATION_SLUG= # optional, scope to one organization SENTRY_MCP_PROJECT_SLUG= # optional, scope to one project SENTRY_MCP_SKILLS= # optional, comma-separated skill filter ``` | Variable | Default | Description | | - | - | - | | `SENTRY_MCP_AUTH_TOKEN` | — | **Required** (hosted). Sentry user auth token | | `SENTRY_MCP_URL` | `https://mcp.sentry.dev/mcp` | MCP server URL | | `SENTRY_MCP_MODE` | `streamable-http` | Transport: `streamable-http`, `sse`, or `stdio` | | `SENTRY_MCP_HOST` | — | Self-hosted Sentry hostname (for example `sentry.example.com`) | | `SENTRY_MCP_ORGANIZATION_SLUG` | — | Scope tools to a specific organization | | `SENTRY_MCP_PROJECT_SLUG` | — | Scope tools to a specific project | | `SENTRY_MCP_SKILLS` | — | Comma-separated skill filter (`inspect`, `seer`, `triage`, `project-management`) | | `SENTRY_MCP_COMMAND` | — | Command to launch a local MCP server (`stdio` mode only) | | `SENTRY_MCP_ARGS` | — | Arguments for the local MCP command (`stdio` mode only) | To run a local Sentry MCP server instead of the hosted endpoint: ```bash theme={null} SENTRY_MCP_MODE=stdio SENTRY_MCP_COMMAND=npx SENTRY_MCP_ARGS=@sentry/mcp-server@latest SENTRY_MCP_AUTH_TOKEN=your_sentry_user_auth_token ``` > [!NOTE] > > The `search_events` and `search_issues` tools translate natural-language queries > and require an `OPENAI_API_KEY` in the MCP server's environment. The rest of the > MCP server works without it. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "sentry-mcp-prod", "service": "sentry_mcp", "status": "active", "credentials": { "url": "https://mcp.sentry.dev/mcp", "mode": "streamable-http", "auth_token": "your_sentry_user_auth_token", "organization_slug": "my-org" } } ] } ``` ## Credentials Create a Sentry **user auth token** in account settings with at least `org:read` for read-only investigation. Add write scopes only if you enable triage or project-management skills. ## Tools | Tool | What it does | | - | - | | `list_sentry_tools` | List tools the connected Sentry MCP server exposes | | `call_sentry_tool` | Call a named Sentry MCP tool (issue lookup, trace fetch, Seer analysis, and more) | Typical flow: call `list_sentry_tools` first, then `call_sentry_tool` with the chosen name and arguments. ## Verify ```bash theme={null} opensre integrations verify sentry_mcp ``` A successful check connects to the MCP server and reports how many tools it discovered (detail shape: `Sentry MCP connected via {mode} ({endpoint}); discovered N tool(s).`). ## Troubleshooting | Symptom | Fix | | - | - | | Missing or invalid token | Confirm the user auth token has at least `org:read` | | Cannot reach hosted MCP | Allow outbound HTTPS to `mcp.sentry.dev` | | Self-hosted tools missing | Set `SENTRY_MCP_HOST` (and URL/mode as needed) | | `search_events` / `search_issues` fail | Set `OPENAI_API_KEY` in the MCP server environment | ## Security * Prefer `org:read` only unless you need write skills. * Store `SENTRY_MCP_AUTH_TOKEN` in `.env` or your secret manager — not in source control. * Scope with `SENTRY_MCP_ORGANIZATION_SLUG` / `SENTRY_MCP_PROJECT_SLUG` when possible. # ServiceNow Source: https://opensre.com/docs/integrations/incidents/servicenow Connect a ServiceNow instance so OpenSRE can check its configuration and credentials ## Overview OpenSRE connects to ServiceNow so the CLI and interactive shell can answer configuration questions about your instance — for example `opensre integrations verify servicenow` and shell questions like *"Is ServiceNow configured?"*. ServiceNow today is a **config and verify** integration. It does not register agent query tools. ## Prerequisites * A ServiceNow instance URL (a free [developer instance](https://developer.servicenow.com) works) * A user with read access to the `sys_user` table (used for the live connectivity check) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup servicenow ``` ### Option 2: Environment variables ```bash theme={null} export SERVICENOW_INSTANCE_URL=https://dev12345.service-now.com export SERVICENOW_USERNAME=admin export SERVICENOW_PASSWORD=your-password ``` `SERVICENOW_PASSWORD` is also resolved from `~/.opensre/credentials.json` when the environment variable is not set (the wizard stores it there). ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "servicenow-prod", "service": "servicenow", "status": "active", "credentials": { "instance_url": "https://dev12345.service-now.com", "username": "admin", "password": "your-password" } } ] } ``` | Field | Description | | - | - | | `instance_url` | **Required.** Instance URL (for example `https://dev12345.service-now.com`). Must use `https://` — plain `http://` is accepted only for localhost/loopback. | | `username` | **Required.** ServiceNow username for HTTP Basic authentication | | `password` | **Required.** Password for the user | ## Credentials Use a dedicated ServiceNow user with the minimum roles needed for a one-row read of `sys_user` (for example `itil`). Prefer `opensre integrations setup servicenow` so the password stays in `~/.opensre/credentials.json` instead of `.env`. ## Tools None. ServiceNow is used for configuration and verification only. The interactive shell can still answer *"Is ServiceNow configured?"* by running the verifier. ## Verify ```bash theme={null} opensre integrations verify servicenow ``` Verify is a **live** check: it validates the URL shape (HTTPS or loopback) and calls `GET …/api/now/table/sys_user?sysparm_limit=1`. Expected detail on success: ``` ServiceNow connected as at https://dev12345.service-now.com. ``` You can also ask the interactive shell: ``` > Is ServiceNow configured? ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Status `missing`** | Set `instance_url`, `username`, and `password` via any setup option above | | **401 Unauthorized** | Check the username and password combination | | **403 Forbidden** | User authenticated but cannot read `sys_user` — grant a role with table read access (for example `itil`) | | **404 Not Found** | Verify the instance URL (include `https://`, no path) | | **Developer instance unreachable** | Free developer instances hibernate after inactivity — wake it from the developer portal and retry | ## Security * Use a **dedicated ServiceNow user** with minimum read roles. * Prefer `opensre integrations setup servicenow` so the password stays in `~/.opensre/credentials.json`. * Rotate the password periodically and remove the integration when it is no longer needed. # Integrations overview Source: https://opensre.com/docs/integrations/index Connect observability, cloud, data, and messaging tools for the agent. OpenSRE answers production questions by querying the same tools your team already uses. Connect an integration so OpenSRE can pull logs and metrics, check recent deploys, and deliver findings to your channels. There are 60+ supported integrations. Connect your stack so OpenSRE can pull logs, check recent deploys, and deliver findings where your team already works. ## Two setup paths | Path | Best for | How to configure | | - | - | - | | Local CLI | Self-hosted `opensre` on your laptop or server | `opensre integrations setup`, plus env vars | | Hosted (OpenSRE Cloud) | Org-wide connectors shared across users | Connect at [app.opensre.com](https://app.opensre.com) → Integrations | Either path works on its own. When you are signed in and a service is connected in both places, the OpenSRE app connection wins. ## Local CLI ```bash theme={null} opensre integrations setup ``` You can also set environment variables; OpenSRE picks them up on the next run. See each integration’s page for the required variables, or [Environment variables](/docs/install/environment-variables). ### Verify ```bash theme={null} opensre integrations verify opensre integrations verify datadog ``` In the interactive shell: `/integrations list`, `/integrations verify `, `/verify`, and `/health`. To register more than one instance of the same provider (for example prod and staging Grafana), see [Multi-instance integrations](/docs/platform/multi-instance-integrations). ## Catalog Browse by category in the sidebar, or use the links below. Model providers are configured separately under [LLM providers](/docs/platform/llm-providers). | Category | Integrations | | - | - | | **Monitoring** | [Azure Monitor](/docs/integrations/monitoring/azure-monitor), [Better Stack](/docs/integrations/monitoring/betterstack), [Coralogix](/docs/integrations/monitoring/coralogix), [Datadog](/docs/integrations/monitoring/datadog), [Grafana](/docs/integrations/monitoring/grafana), [Groundcover](/docs/integrations/monitoring/groundcover), [Honeycomb](/docs/integrations/monitoring/honeycomb), [New Relic](/docs/integrations/monitoring/new_relic), [OpenObserve](/docs/integrations/monitoring/openobserve), [PostHog](/docs/integrations/monitoring/posthog), [PostHog MCP](/docs/integrations/monitoring/posthog-mcp), [SignOz](/docs/integrations/monitoring/signoz), [Splunk](/docs/integrations/monitoring/splunk), [Tempo](/docs/integrations/monitoring/tempo), [Victoria Logs](/docs/integrations/monitoring/victoria_logs) | | **Incidents** | [Alertmanager](/docs/integrations/incidents/alertmanager), [Incident.io](/docs/integrations/incidents/incident_io), [PagerDuty](/docs/integrations/incidents/pagerduty), [Sentry](/docs/integrations/incidents/sentry), [Sentry MCP](/docs/integrations/incidents/sentry-mcp), [ServiceNow](/docs/integrations/incidents/servicenow) | | **Cloud** | [Argo CD](/docs/integrations/cloud/argocd), [AWS](/docs/integrations/cloud/aws), [Helm](/docs/integrations/cloud/helm), [Kubernetes](/docs/integrations/cloud/kubernetes), [Railway](/docs/integrations/cloud/railway), [SQS](/docs/integrations/cloud/sqs_queues), [Vercel](/docs/integrations/cloud/vercel) | | **Code and CI** | [Bitbucket](/docs/integrations/code/bitbucket), [GitHub](/docs/integrations/code/github), [GitHub Actions](/docs/integrations/code/github-actions), [GitLab](/docs/integrations/code/gitlab), [Jenkins](/docs/integrations/code/jenkins) | | **Collaboration** | [Google Docs](/docs/integrations/collaboration/google-docs), [Jira](/docs/integrations/collaboration/jira), [Trello](/docs/integrations/collaboration/trello) | | **Messaging** | [Slack](/docs/integrations/messaging/slack), [Discord](/docs/integrations/messaging/discord), [Telegram](/docs/integrations/messaging/telegram), [Rocket.Chat](/docs/integrations/messaging/rocketchat), [Buzz](/docs/integrations/messaging/buzz), [WhatsApp](/docs/integrations/messaging/whatsapp), [Twilio SMS](/docs/integrations/messaging/twilio-sms), [SMTP](/docs/integrations/messaging/smtp) | | **Databases** | [Azure SQL](/docs/integrations/databases/azure-sql), [ClickHouse](/docs/integrations/databases/clickhouse), [Elasticsearch](/docs/integrations/databases/elasticsearch), [MariaDB](/docs/integrations/databases/mariadb), [MongoDB](/docs/integrations/databases/mongodb), [MongoDB Atlas](/docs/integrations/databases/mongodb-atlas), [MySQL](/docs/integrations/databases/mysql), [OpenSearch](/docs/integrations/databases/opensearch), [PostgreSQL](/docs/integrations/databases/postgresql), [RDS](/docs/integrations/databases/rds), [Redis](/docs/integrations/databases/redis), [Snowflake](/docs/integrations/databases/snowflake), [Supabase](/docs/integrations/databases/supabase) | | **Workflows** | [Airflow](/docs/integrations/workflows/airflow), [Dagster](/docs/integrations/workflows/dagster), [Kafka](/docs/integrations/workflows/kafka), [Prefect](/docs/integrations/workflows/prefect), [RabbitMQ](/docs/integrations/workflows/rabbitmq), [Temporal](/docs/integrations/workflows/temporal) | ## Hosted (OpenSRE Cloud) Connect services once at [app.opensre.com](https://app.opensre.com) → Integrations and every signed-in surface uses them: the gateway, the interactive shell, and the `opensre` CLI on any machine where you are signed in. No local setup is needed on each laptop. If the same service is also configured locally, the OpenSRE app connection takes precedence. To use a local credential instead, disconnect the service in the web app. `opensre integrations list` shows where each connection comes from (`remote` or `local`). Signed-out or offline machines fall back to local integrations only. # Buzz Source: https://opensre.com/docs/integrations/messaging/buzz Chat with the agent and send messages to a Buzz (block/buzz) channel. ## Overview OpenSRE's Buzz integration posts messages to a channel in [Buzz](https://github.com/block/buzz), Block's self-hostable, Nostr-based workspace where humans and AI agents share rooms. Start the interactive shell with `opensre` (no subcommand). Slash commands below are run from that REPL. > [!NOTE] > > Steps 1–3 give you **outbound delivery** for agent-requested messages. To `@mention` the agent from a Buzz channel and get replies, also do [Step 4](#step-4-enable-two-way-chat). ## Prerequisites * A running Buzz relay (self-hosted) and its URL, e.g. `https://buzz.example.com` (defaults to `http://localhost:3000` for local dev). * The **`buzz` CLI** on `PATH`. It is not published to any package registry — build it from the [block/buzz](https://github.com/block/buzz) repo: ```bash theme={null} cargo install --path crates/buzz-cli ``` If you can't put it on `PATH`, point OpenSRE at the binary with `BUZZ_PATH=/path/to/buzz` or the `buzz_path` field during setup. * An agent identity (a Nostr keypair). Generate one with the relay's admin tool: ```bash theme={null} buzz-admin generate-key ``` This prints a public key (`npub...`) and a private key (`nsec...` or hex). The private key is what OpenSRE needs — treat it like a password; it's routed to `~/.opensre/credentials.json`, never plain `.env`. * If your relay has `BUZZ_REQUIRE_RELAY_MEMBERSHIP` enabled, the agent's public key must be registered as a relay member before it can post — ask your Buzz admin to add it, or messages will fail silently. ## Setup ### Find a channel Buzz channels are identified by **UUID**, not by name. List the channels your agent identity can see: ```bash theme={null} BUZZ_PRIVATE_KEY= BUZZ_RELAY_URL= buzz channels list ``` Copy the UUID of the channel messages should land in. ### Option 1: CLI setup (recommended) ```bash theme={null} opensre integrations setup buzz ``` Interactive shell: `/integrations setup buzz`. The setup prompts for: * **Relay URL** (`BUZZ_RELAY_URL`, defaults to `http://localhost:3000`) * **Private key** (`BUZZ_PRIVATE_KEY` in `.env` and `~/.opensre/credentials.json`) * **Default channel** — the UUID from above (`BUZZ_DEFAULT_CHANNEL`) * **Auth tag** — optional NIP-OA JSON for owner attestation (`BUZZ_AUTH_TAG`) * **CLI binary path** — only if `buzz` isn't on `PATH` (`BUZZ_PATH`) Credentials are also saved to `~/.opensre/integrations.json` via `upsert_integration("buzz", ...)`. ### Option 2: Environment variables ```bash theme={null} BUZZ_RELAY_URL=https://buzz.example.com BUZZ_PRIVATE_KEY= BUZZ_DEFAULT_CHANNEL= # BUZZ_AUTH_TAG= BUZZ_PATH=buzz ``` | Variable | Description | | - | - | | `BUZZ_RELAY_URL` | Buzz relay base URL. Defaults to `http://localhost:3000`. | | `BUZZ_PRIVATE_KEY` | Agent's Nostr private key (hex or `nsec1...`). Required. Resolved via env then keyring. | | `BUZZ_DEFAULT_CHANNEL` | Default delivery destination — a channel UUID from `buzz channels list`. Required for delivery. | | `BUZZ_AUTH_TAG` | Optional NIP-OA owner-attestation JSON, injected into every signed event. | | `BUZZ_PATH` | Override when the `buzz` binary isn't on `PATH`. Defaults to `buzz`. | | `BUZZ_ALLOWED_PUBKEYS` | Two-way chat only. Comma-separated 64-char hex pubkeys allowed to talk to the agent. | | `BUZZ_GATEWAY_POLL_INTERVAL_SECONDS` | Two-way chat only. How often to check for mentions. Defaults to `15`. | | `BUZZ_GATEWAY_MAX_CONCURRENT` | Two-way chat only. Turns run in parallel. Defaults to `4`. | | `BUZZ_GATEWAY_AUTO_START` | Two-way chat only. Set `false` to keep the gateway from starting Buzz polling. | > [!NOTE] > > **Credential resolution.** `BUZZ_PRIVATE_KEY` resolves via store → env → `~/.opensre/credentials.json`. The rest (relay URL, default channel, auth tag, binary path) stay plain env / store. ## Credentials 1. Build/install the `buzz` CLI. 2. Generate a keypair with `buzz-admin generate-key`. 3. Register the public key as a relay member if your relay requires membership. 4. List channels and copy the destination UUID. 5. Store the private key via setup (keyring) or `BUZZ_PRIVATE_KEY`. ## Buzz tools | Tool | What it does | | - | - | | `buzz_send_message` | Post a message to the configured (or explicit) Buzz channel | ## Verify > [!NOTE] > > **Verify does not test two-way chat.** It confirms the relay accepts your key; it does not start a listening process. If you `@mention` the agent at this point, nothing will answer. Complete Step 4 for that. *** ## Step 4: Enable two-way chat > **Skip this step if you only need outbound delivery.** Steps 1–3 are sufficient for that. ### 4a — Give the agent a name and add it to the channel Run both with the agent's own `BUZZ_PRIVATE_KEY`: ```bash theme={null} buzz users set-profile --name opensre buzz channels join --channel ``` The name is what makes `@opensre` work: Buzz resolves `@Name` against the channel's member profiles to get a pubkey. Without one the agent shows as a raw hex key and is effectively unmentionable. Skipping the join is the most common cause of "the gateway is running but nothing happens" — the agent only sees messages that mention it, and only members can be mentioned. > [!NOTE] > > This is **not** the same as a Buzz *managed agent* (`buzz agents draft-create`). Those are agents Buzz itself runs — the record holds a system prompt, provider, and model, and Buzz drives the loop. OpenSRE runs its own agent loop and uses Buzz purely as a chat surface, so it joins as a normal identity. Mention delivery works the same either way. ### 4b — Allow your own pubkey ```text theme={null} /messaging allow -p buzz -u ``` CLI: ```bash theme={null} opensre messaging allow -p buzz -u ``` Or set in `.env`: ```bash theme={null} BUZZ_ALLOWED_PUBKEYS= # comma-separated for multiple people ``` > [!WARNING] > > Use the **64-character hex** public key, not the `npub1...` form. Inbound authorization matches the hex key that Nostr events actually carry, so an `npub` entry can never match a real sender. `opensre messaging allow -p buzz` rejects anything that isn't 64 hex characters. Your own hex key is the `pubkey` field of `buzz users get` (run with your key, no `--pubkey` flag), or your key generator's hex output. ### 4c — Start the gateway daemon ```bash theme={null} opensre gateway start ``` Then `@mention` the agent in the channel. Follow-up messages **in the same channel do not need a mention** — the agent keeps your session and picks up the thread. Each person gets their own private session, so two people in one channel never see each other's context. | You type | What happens | | - | - | | `@agent why is checkout-service erroring?` | Starts (or continues) your session | | `/new` | Drops your session history and starts fresh | | `/help` | Built-in command list | | `/pair ` | Completes pairing when your pubkey isn't allowlisted yet | Manage the daemon with `opensre gateway status`, `opensre gateway logs -f`, and `opensre gateway stop`. > [!NOTE] > > Buzz has no push socket here — the gateway polls the relay every 15 seconds (`BUZZ_GATEWAY_POLL_INTERVAL_SECONDS`). Expect up to that long before the agent starts typing. Restarting the gateway can re-deliver the single most recent mention; it never replays your whole history. ### Approving write actions Tools that change something ask before running. The agent posts a prompt in the channel and you answer by **replying to that message** with `approve` or `deny` — no mention needed, and no buttons (Buzz has none). * **Only the person whose request triggered it can answer**, in the channel it was posted to. Another channel member replying `approve` is ignored, even if they are allowlisted. * A reply that is neither approve nor deny (`hold on, checking`) leaves the prompt open rather than counting as a refusal. * Unanswered prompts expire after 3 minutes and the action is skipped. *** ```bash theme={null} opensre integrations verify buzz ``` Interactive shell: `/integrations verify buzz`. This resolves the `buzz` binary, then runs `buzz channels list` against the configured relay. A missing binary or key reports **missing** with an install/setup hint; an unreachable relay or rejected key reports **failed**. ## Troubleshooting | Symptom | Fix | | - | - | | **missing binary / key** | Install `buzz` CLI or set `BUZZ_PATH`; configure `BUZZ_PRIVATE_KEY` | | **failed: unreachable relay** | Check `BUZZ_RELAY_URL` and network path | | **Messages fail silently** | If `BUZZ_REQUIRE_RELAY_MEMBERSHIP` is on, register the agent public key | | **Wrong destination** | Channels are UUIDs — use `buzz channels list`, not `#names` | ## Security * Treat the Nostr private key like a password; prefer `~/.opensre/credentials.json`. * Use a dedicated agent identity for OpenSRE. * Do not commit private keys to source control. # Discord Source: https://opensre.com/docs/integrations/messaging/discord Two-way OpenSRE chat in Discord DMs, channels, and threads. ## Overview OpenSRE's Discord integration supports **two-way gateway chat** (DMs, @mentions, active threads). The gateway connects over Discord's Gateway WebSocket with `discord.py` — you do **not** need a public `/discord/interactions` URL for normal operation. There are no registered Discord tool packages today — Discord is delivery + gateway chat. ## Prerequisites * A Discord server where you can add applications (**Manage Server**). * An OpenSRE host running `opensre gateway start` (local or deployed). * Discord **Privileged Gateway Intent: Message Content** enabled for the bot (required to read @mentions and thread replies in servers). ## Setup ### Option 1: CLI setup ```bash theme={null} opensre integrations setup discord ``` ### Option 2: Environment variables | Variable | Description | | - | - | | `DISCORD_BOT_TOKEN` | Bot token (**required**) | | `DISCORD_APPLICATION_ID` | Application ID from the Developer Portal | | `DISCORD_PUBLIC_KEY` | Stored for integrations; not required for gateway chat | | `DISCORD_DEFAULT_CHANNEL_ID` | Optional fallback channel for outbound-only delivery | | `DISCORD_ALLOWED_USERS` | Comma-separated Discord user snowflakes allowed to chat | | `DISCORD_ALLOW_OPEN_GUILD` | Set to `1` to allow any member in server channels (dogfood only). DMs still require the allowlist | | `DISCORD_SILO_GUILD_IDS` | Comma-separated server ids allowed to use this deployment's organization. Unset serves every server the bot is in | Discord turns are billed and stored against the organization in `ORGANIZATION_ID`. Without it every turn is refused — the bot appears online and silently does nothing, and the reason is only in the gateway log. Set `DISCORD_SILO_GUILD_IDS` on any deployment holding real credentials. Leaving it unset means any server that adds the bot inherits that organization's integrations. Approving a write action always requires `DISCORD_ALLOWED_USERS`. `DISCORD_ALLOW_OPEN_GUILD` opens chat, never approvals, so with no allowlist every Approve/Deny click is ignored. Allow yourself explicitly: ```bash theme={null} opensre messaging allow -p discord -u ``` Start the gateway: ```bash theme={null} opensre gateway start ``` When configured, status includes `discord: connected via gateway`. ## Credentials ### Create a Discord application and bot 1. Go to the [Discord Developer Portal](https://discord.com/developers/applications) and click **New Application**. 2. Open the **Bot** tab: * Copy the **Token** (`DISCORD_BOT_TOKEN`). * Enable **Message Content Intent** under **Privileged Gateway Intents**. 3. On **General Information**, copy **Application ID** and **Public Key** (still used by setup and optional HTTP tooling). ### Invite the bot with the right permissions In **OAuth2 → URL Generator**, select the `bot` scope, then bot permissions: * **View Channels** * **Read Message History** * **Send Messages** * **Send Messages in Threads** * **Embed Links** * **Add Reactions** Open the generated URL and add the bot to your server. ## Chatting with the agent No Discord-specific tools are registered. **DM** the bot, **@mention** it in a channel, or reply in a **thread** where it is already engaged. Gateway turns use the same agent session model as Slack and Telegram: `/new`, `/help`, and `/pair ` work in Discord text. Each member gets their own conversation, even in a shared channel: your history and `/new` never affect a teammate's. Integrations you connect are shared by the whole organization. Pairing: ```bash theme={null} opensre messaging pair -p discord # then send /pair to the bot ``` ## Verify ```bash theme={null} opensre integrations verify discord ``` Verification checks the bot token against Discord `/users/@me` only. ## Troubleshooting | Symptom | Fix | | - | - | | **Bot online but ignores channel messages** | Enable **Message Content Intent** and restart the gateway. Ensure your user id is on the allowlist (`opensre messaging status --platform discord` or `DISCORD_ALLOWED_USERS`). | | **Gateway shows Discord not configured** | Set `DISCORD_BOT_TOKEN` and at least one allowed user (or `DISCORD_ALLOW_OPEN_GUILD=1`). | | **Bot online but silent turns** | Set `ORGANIZATION_ID` — without it every turn is refused. | ## Security * Prefer a dedicated Discord application for OpenSRE. * Keep `DISCORD_ALLOWED_USERS` set in production; treat `DISCORD_ALLOW_OPEN_GUILD=1` as dogfood only. * Set `DISCORD_SILO_GUILD_IDS` on deployments with real credentials. * Store the bot token in the store / keyring / secret manager — not in source control. # Messaging Source: https://opensre.com/docs/integrations/messaging/index Chat with the OpenSRE agent and receive scheduled reports via Slack, Discord, Telegram, Rocket.Chat, WhatsApp, Twilio SMS, Buzz, or email (SMTP). ## Overview OpenSRE can chat two-way on some platforms and post messages — agent replies and scheduled reports — through messaging and email. Pick the channel where your team already responds to incidents. ## Prerequisites * Admin or install rights on the platform you choose * An OpenSRE host that can reach the platform API (and run `opensre gateway start` for two-way chat) ## Setup | Platform | Best for | Two-way chat? | Configured by | Setup time | | - | - | - | - | - | | [**Slack**](/docs/integrations/messaging/slack) | Teams already paged in Slack channels | ✅ Socket Mode chat via `opensre gateway start` (+ webhook delivery) | `opensre integrations setup slack` | \~10 min | | [**Discord**](/docs/integrations/messaging/discord) | Communities and teams using Discord servers | ✅ Gateway chat via `opensre gateway start` | `opensre integrations setup discord` | \~10 min | | [**Telegram**](/docs/integrations/messaging/telegram) | Mobile-first and on-call rotations | ✅ DM text chat via `opensre gateway start` | `opensre integrations setup telegram` | \~5 min | | [**Rocket.Chat**](/docs/integrations/messaging/rocketchat) | Self-hosted and open-source workspaces | ❌ delivery only | `opensre integrations setup rocketchat` | \~5 min | | [**WhatsApp**](/docs/integrations/messaging/whatsapp) | Mobile alerting via Twilio WhatsApp | ❌ delivery only | `opensre integrations setup whatsapp` | \~5 min | | [**Twilio SMS**](/docs/integrations/messaging/twilio-sms) | SMS paging via Twilio (separate from WhatsApp) | ❌ delivery only | `opensre integrations setup twilio` | \~5 min | | [**Buzz**](/docs/integrations/messaging/buzz) | Block Buzz / Nostr workspaces | ✅ channel mentions via `opensre gateway start` | `opensre integrations setup buzz` | \~5 min | | [**SMTP**](/docs/integrations/messaging/smtp) | Outbound email | ❌ email only | `opensre integrations setup smtp` | \~5 min | If you're not sure, start with Slack. Pick Discord if you want chat with threaded follow-ups. Pick WhatsApp or Twilio SMS when your team already pages via Twilio. You can configure more than one. Each is independent and uses its own credentials. Two-way pairing (`opensre messaging pair|allow|revoke`) applies to **Telegram, Slack, Discord, and Buzz** only. In the interactive shell, run `/messaging` to choose an action and platform; `allow` and `revoke` also collect a user ID. Outside the shell, pass `--platform` and, when needed, `--user-id`. ## Credentials Each platform guide covers: 1. **Prerequisites** — workspace/server admin rights, Twilio account, etc. 2. **Create the bot/app** — BotFather, Discord Developer Portal, Slack App, etc. 3. **Credentials** — the exact tokens, IDs, and keys to copy. 4. **OpenSRE setup** — `opensre integrations setup …`. 5. **Verify** — how to confirm delivery is working. 6. **Troubleshooting** — common failures. ### Common environment variables ```bash theme={null} # Slack — webhook delivery and/or Socket Mode gateway SLACK_WEBHOOK_URL= SLACK_BOT_TOKEN= SLACK_APP_TOKEN= SLACK_ALLOWED_USERS= # Discord DISCORD_BOT_TOKEN= DISCORD_APPLICATION_ID= DISCORD_PUBLIC_KEY= DISCORD_DEFAULT_CHANNEL_ID= # Telegram TELEGRAM_BOT_TOKEN= TELEGRAM_DEFAULT_CHAT_ID= TELEGRAM_ALLOWED_USERS= # Rocket.Chat (token mode and/or webhook mode) ROCKETCHAT_SERVER_URL= ROCKETCHAT_AUTH_TOKEN= ROCKETCHAT_USER_ID= ROCKETCHAT_DEFAULT_CHANNEL= ROCKETCHAT_WEBHOOK_URL= # WhatsApp (Twilio) TWILIO_ACCOUNT_SID= TWILIO_AUTH_TOKEN= TWILIO_WHATSAPP_FROM= WHATSAPP_DEFAULT_TO= # Twilio SMS (shares TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN) TWILIO_SMS_FROM= TWILIO_SMS_MESSAGING_SERVICE_SID= TWILIO_SMS_DEFAULT_TO= # Buzz BUZZ_RELAY_URL= BUZZ_PRIVATE_KEY= BUZZ_DEFAULT_CHANNEL= # SMTP (email) SMTP_HOST= SMTP_PORT=587 SMTP_FROM_ADDRESS= SMTP_DEFAULT_TO= ``` ## Agent tools Platform-specific send/read tools and gateway behavior are documented on each page (for example `slack_send_message`, `telegram_send_message`, `buzz_send_message`, `twilio_notify`). ## Verify After editing `.env` or the store, run: ```bash theme={null} opensre integrations verify ``` Examples: `telegram`, `slack`, `whatsapp`, `twilio`, `buzz`, `smtp`. Note that `opensre doctor` only inspects the integrations store (`~/.opensre/integrations.json`), so it does **not** detect env-var-only configurations. ### Re-running setup ```bash theme={null} opensre integrations setup ``` Existing values are often pre-populated so you can press Enter to keep what you have. Slack, Telegram, Discord, Rocket.Chat, Buzz, WhatsApp, Twilio SMS, and SMTP each have a dedicated setup path as listed in the table above. ## Troubleshooting | Symptom | Fix | | - | - | | **Doctor misses env-only config** | Expected — use `opensre integrations verify ` | | **Gateway ignores chat** | Pairing/allow-list applies to Telegram, Slack, Discord, and Buzz — see that platform's page | | **Not sure which platform** | Start with Slack webhook for delivery; add Socket Mode / Telegram later for two-way chat | ## Security * Treat webhooks and bot tokens like passwords. * Prefer dedicated bot accounts and least-privilege scopes. * Store secrets in the integration store / `~/.opensre/credentials.json` / secret manager — not in source control. # Rocket.Chat Source: https://opensre.com/docs/integrations/messaging/rocketchat Deliver agent messages, alarms, and scheduled reports to a Rocket.Chat channel. ## Overview OpenSRE's Rocket.Chat integration delivers agent-requested posts and scheduled reports to any channel your user or bot account can post to. Useful for teams running self-hosted or cloud Rocket.Chat workspaces. Start the interactive shell with `opensre` (no subcommand). Slash commands below are run from that REPL. > [!NOTE] > > Rocket.Chat support is **outbound delivery only**. Chatting with the agent from Rocket.Chat is not supported yet. There is no gateway inbound / `messaging pair` for Rocket.Chat. ## Prerequisites * A Rocket.Chat workspace (self-hosted or cloud) and its base URL, e.g. `https://chat.example.com`. * An account allowed to post in the destination channel. A dedicated bot account is recommended so messages are not attributed to a personal user. * The **Personal Access Tokens** feature enabled on the server (`Admin → Settings → Accounts → Personal Access Tokens`, on by default in most installs) — or admin access to create an **incoming webhook**. ## Setup ### Pick a mode | Mode | Credentials | Destination | Best for | | - | - | - | - | | **Personal Access Token** (recommended) | `server_url` + `auth_token` + `user_id` | Any channel, chosen per message | Dynamic channel targeting, verifiable via `/api/v1/me` | | **Incoming webhook** | single `webhook_url` | Fixed channel chosen when the webhook is created | Simplest setup; no bot account needed | You can configure both — when both are present, delivery prefers the webhook. Incomplete token trio fails verify. ### Option 1: CLI setup (recommended) ```bash theme={null} opensre integrations setup rocketchat ``` Interactive shell: `/integrations setup rocketchat`. Choose **token**, **webhook**, or **both**. The setup prompts for: * Token mode: **Server URL** (`ROCKETCHAT_SERVER_URL`), **personal access token** (`ROCKETCHAT_AUTH_TOKEN` in `.env` and `~/.opensre/credentials.json`), **user ID** (`ROCKETCHAT_USER_ID`), and **default channel** (`ROCKETCHAT_DEFAULT_CHANNEL`) * Webhook mode: **webhook URL** — saved to the integration store (`~/.opensre/integrations.json`) and/or `ROCKETCHAT_WEBHOOK_URL` in `.env`; never written to the keyring (`*_URL` is non-keyring config). The URL often embeds a token — treat the whole URL like a password. Credentials are also saved via `upsert_integration("rocketchat", ...)`. ### Option 2: Environment variables ```bash theme={null} # Token mode ROCKETCHAT_SERVER_URL=https://chat.example.com ROCKETCHAT_AUTH_TOKEN= ROCKETCHAT_USER_ID= ROCKETCHAT_DEFAULT_CHANNEL=#incidents # Webhook mode (either mode alone is enough; both may be set) ROCKETCHAT_WEBHOOK_URL=https://chat.example.com/hooks// ``` | Variable | Description | | - | - | | `ROCKETCHAT_SERVER_URL` | Base URL of your Rocket.Chat server. Required for token mode. | | `ROCKETCHAT_AUTH_TOKEN` | Personal access token. Required for token mode. Resolved via env then keyring. | | `ROCKETCHAT_USER_ID` | User ID shown alongside the token. Required for token mode. | | `ROCKETCHAT_DEFAULT_CHANNEL` | Default delivery destination (`#channel` or `@user`). Required for token-mode delivery. | | `ROCKETCHAT_WEBHOOK_URL` | Incoming webhook URL. Store/env only — never keyring. Enables webhook mode on its own; preferred over token mode when both are set. | > [!NOTE] > > **Credential resolution.** Token mode: store → `resolve_env_credential("ROCKETCHAT_AUTH_TOKEN")` (env then keyring). Webhook mode: store → plain `ROCKETCHAT_WEBHOOK_URL` env — never keyring. Server URL, user id, and default channel stay plain env / store. ## Credentials ### Token mode: Create a Personal Access Token 1. Sign in to Rocket.Chat with the account that should post messages. 2. Open **My Account → Personal Access Tokens** (avatar menu → **My Account**). 3. Enter a token name (e.g. `opensre`) and select **Add**. Leave *Ignore Two Factor Authentication* checked unless your policy requires otherwise. 4. Rocket.Chat shows the **token** and your **user ID** once. Copy both — the token is your `auth_token` and the ID is your `user_id`. Treat the token like a password. > [!TIP] > > If you missed the user ID, it is also shown when you regenerate the token, or under `Admin → Users` for admins. ### Token mode: Pick a destination channel Messages are posted with the standard `chat.postMessage` REST endpoint: | Destination | Format | Example | | - | - | - | | Public/private channel | `#channel-name` | `#incidents` | | Direct message to a user | `@username` | `@marcos` | Make sure the token's account is a member of the channel (or has permission to post there). ### Webhook mode: Create an incoming webhook 1. As an admin, open **Administration → Workspace → Integrations → New → Incoming**. 2. Enable the integration, set a name (e.g. `opensre`), pick the destination **channel**, and choose the user the messages post as. 3. Select **Save**. Rocket.Chat shows the **Webhook URL** (`https:///hooks//`). Copy it — treat the whole URL like a password. ### Quick local test with Docker (token mode) Requires `--platform linux/amd64` on Apple Silicon (no arm64 image) and **MongoDB 7.0**, not 8.0 — 8.0 refuses to start under Docker Desktop's Linux VM ("kernel versions 6.19 and newer has a known incompatibility"). ```bash theme={null} docker network create rc-net docker run -d --name rc-mongo --network rc-net --platform linux/amd64 \ mongo:7.0 --replSet rs0 --bind_ip_all i=0 until docker exec rc-mongo mongosh --quiet --eval "db.runCommand('ping')" > /dev/null 2>&1; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: MongoDB never became ready" >&2; exit 1; } sleep 2 done docker exec rc-mongo mongosh --eval "rs.initiate()" docker run -d --name rc-dev --network rc-net --platform linux/amd64 -p 127.0.0.1:3201:3000 \ -e 'MONGO_URL=mongodb://rc-mongo:27017/rocketchat?replicaSet=rs0' \ -e 'MONGO_OPLOG_URL=mongodb://rc-mongo:27017/local?replicaSet=rs0' \ -e 'ROOT_URL=http://localhost:3201' \ -e 'ADMIN_USERNAME=verifyadmin' \ -e 'ADMIN_PASS=verify-admin-pass' \ -e 'ADMIN_EMAIL=verifyadmin@example.com' \ -e 'OVERWRITE_SETTING_Show_Setup_Wizard=completed' \ -e 'OVERWRITE_SETTING_Accounts_TwoFactorAuthentication_Enabled=false' \ rocket.chat:8.5.1 ``` The `OVERWRITE_SETTING_Show_Setup_Wizard=completed` and `ADMIN_*` variables skip the manual setup wizard. The 2FA override lets this disposable instance generate a PAT without a database change or restart. ```bash theme={null} i=0 until curl -sf -o /dev/null http://localhost:3201/api/info; do i=$((i + 1)) [ "$i" -ge 40 ] && { echo "ERROR: Rocket.Chat never became ready" >&2; exit 1; } sleep 10 done ``` Log in and generate a PAT via the API. The password stays out of curl's own argv (piped via stdin, not `-d "password=..."`), and the token is captured into a variable rather than printed: ```bash theme={null} RC_LOGIN=$(printf 'user=verifyadmin&password=verify-admin-pass' | curl -s -X POST http://localhost:3201/api/v1/login --data-binary @-) RC_AUTH_TOKEN=$(echo "$RC_LOGIN" | python3 -c "import json,sys;print(json.load(sys.stdin)['data']['authToken'])") RC_USER_ID=$(echo "$RC_LOGIN" | python3 -c "import json,sys;print(json.load(sys.stdin)['data']['userId'])") unset RC_LOGIN RC_PAT=$(printf 'tokenName=opensre-verify' | curl -s -X POST http://localhost:3201/api/v1/users.generatePersonalAccessToken \ -H "X-Auth-Token: $RC_AUTH_TOKEN" -H "X-User-Id: $RC_USER_ID" --data-binary @- \ | python3 -c "import json,sys;print(json.load(sys.stdin)['token'])") ``` ```bash theme={null} export ROCKETCHAT_SERVER_URL="http://localhost:3201" export ROCKETCHAT_AUTH_TOKEN="$RC_PAT" export ROCKETCHAT_USER_ID="$RC_USER_ID" export ROCKETCHAT_DEFAULT_CHANNEL="#general" unset RC_PAT RC_AUTH_TOKEN RC_USER_ID ``` Verify: ```bash theme={null} opensre integrations verify rocketchat ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL rocketchat │ local env │ ✓ passed │ Connected to Rocket.Chat as @verifyadmin. ``` Teardown: ```bash theme={null} docker rm -f rc-dev rc-mongo && docker network rm rc-net unset ROCKETCHAT_SERVER_URL ROCKETCHAT_AUTH_TOKEN unset ROCKETCHAT_USER_ID ROCKETCHAT_DEFAULT_CHANNEL ``` ## Rocket.Chat tools | Tool | What it does | | - | - | | `rocketchat_send_message` | Send a message to the default channel or an explicit `channel` | ```json theme={null} { "name": "rocketchat_send_message", "arguments": { "channel": "#incidents", "message": "DB CPU is back below 70%; keeping the incident open for 10 minutes." } } ``` The tool resolves credentials internally from the integration store, then the same env/keyring rules as setup (PAT via `resolve_env_credential`; webhook URL via plain env) — the agent never sees them. In token mode an explicit `channel` (or the configured `default_channel`) is targeted via `chat.postMessage`; in webhook-only mode messages go to the webhook's fixed destination, and an explicit `channel` returns a configuration error instead of being silently ignored. Delivery is an external side effect but does not require a separate approval. The result includes a stable `status`, `sent`, `error_type`, `channel`, and `message_length` shape so follow-up tool calls can tell configuration failures from Rocket.Chat delivery failures. ### Scheduled deliveries (cron) ```bash theme={null} opensre cron add --kind github_pr_sweep --cron "0 9 * * mon-fri" \ --tz America/Sao_Paulo --provider rocketchat --chat-id "#ops" ``` `--chat-id` is the Rocket.Chat destination (`#channel` or `@user`). Scheduled deliveries always target that explicit destination, so they require **token credentials** — an incoming webhook's destination is fixed at creation time and cannot honor `--chat-id`. See [Cron](/docs/platform/cron). ## Verify ```bash theme={null} opensre integrations verify rocketchat ``` Interactive shell: `/integrations verify rocketchat` or `/verify rocketchat`. With token credentials configured, this calls Rocket.Chat's [`/api/v1/me`](https://developer.rocket.chat/apidocs/get-user-information) endpoint and reports the authenticated `@username`. With webhook-only configuration, it runs a **non-posting reachability probe** against the webhook URL (a 404 means the URL or its embedded token is wrong; no message is delivered by the probe). Long messages are truncated to 4,096 characters. ## Troubleshooting `/integrations verify rocketchat` only calls `/api/v1/me` (or the webhook reachability probe), so it surfaces credential errors but cannot detect channel-routing problems. Delivery-time errors only show up when a message actually posts; they appear in OpenSRE logs as `[rocketchat] post message failed: `. | Symptom | Fix | | - | - | | **`Rocket.Chat auth failed: auth_token or user_id is invalid or expired.`** | Regenerate the token under **My Account → Personal Access Tokens** and update both values. | | **`error-room-not-found` (delivery-time)** | Check `ROCKETCHAT_DEFAULT_CHANNEL` spelling (including `#`) and membership. | | **`Rocket.Chat API check failed: `** | Confirm `ROCKETCHAT_SERVER_URL` includes the scheme and is reachable. | | **`Rocket.Chat webhook returned 404`** | Re-copy the full enabled webhook URL from **Administration → Integrations**. | | **Messages never arrive, but verify passes** | In token mode, confirm `ROCKETCHAT_DEFAULT_CHANNEL` is set — without it (and without a webhook), delivery is silently skipped. In webhook mode, confirm the webhook posts to the channel you expect. | ## Security * Prefer a dedicated bot account and Personal Access Token for OpenSRE. * Treat PAT and webhook URLs like passwords; webhook URLs are never stored in the keyring. * Store secrets in the integration store / keyring / secret manager — not in source control. # Slack Source: https://opensre.com/docs/integrations/messaging/slack Post messages via webhook, and chat with the agent via Socket Mode. ## Overview The Slack integration covers two surfaces: 1. **Incoming webhook** — outbound delivery of agent messages and scheduled reports to a channel. 2. **Socket Mode bot** — two-way chat with the OpenSRE agent (mentions and DMs), managed the same way as the [Telegram gateway](/docs/integrations/messaging/telegram#two-way-chat-gateway-dm-text). ## Prerequisites * A Slack workspace where you can create or install apps (workspace admin or app-install permissions). * The channel you want messages posted to (webhook) and/or where the bot will be invited (Socket Mode). ## Setup ### Option 1: Direct setup command (recommended) ```bash theme={null} opensre integrations setup slack ``` Choose **webhook**, **Socket Mode**, or **both**. Credentials are persisted to `~/.opensre/integrations.json` (and can be merged on re-run). Picking one mode clears the other when you intentionally switch. ### Option 2: Environment variables ```bash theme={null} SLACK_WEBHOOK_URL=https://hooks.slack.com/services/// # Optional Socket Mode gateway: SLACK_BOT_TOKEN=xoxb-… SLACK_APP_TOKEN=xapp-… SLACK_ALLOWED_USERS=U0123ABCD # Optional default channel for scheduled bot-token delivery: # SLACK_DEFAULT_CHAT_ID=C0123ABCD ``` | Variable | Description | | - | - | | `SLACK_WEBHOOK_URL` | Incoming webhook URL. Required for outbound delivery. | | `SLACK_BOT_TOKEN` | Bot token (`xoxb-…`) for the two-way Slack bot. Env or integration store. | | `SLACK_DEFAULT_CHAT_ID` | Default channel (`C…`) for scheduled delivery when using a bot token. Env or integration store; pair it with the same source as the bot token. | | `SLACK_APP_TOKEN` | App-level token (`xapp-…`) for Socket Mode. Env or integration store. | | `SLACK_ACCESS_TOKEN` | User token (`xoxp-…`) with `search:read`. Required only for `slack_search_messages` — Slack rejects bot tokens there. | | `SLACK_ALLOWED_USERS` | Comma-separated Slack user IDs allowed to talk to the bot (required unless open workspace). | | `SLACK_ALLOW_OPEN_WORKSPACE` | Set to `1` to allow any workspace member (dogfood escape hatch). | | `SLACK_TEAM_TASKS_LIST_ID` | Optional Slack List id (`F…`) for `slack_read_list` without a name search. | OpenSRE reads env as a fallback when no Slack entry exists in `~/.opensre/integrations.json`. Bot / app tokens use `resolve_env_credential` (process env, then the credentials file). `SLACK_WEBHOOK_URL` is store/env only. > [!NOTE] > > **Credential resolution.** Store first when present. Then: `SLACK_BOT_TOKEN` / `SLACK_APP_TOKEN` → env then keyring; `SLACK_WEBHOOK_URL` → plain env only (never keyring). Socket Mode needs **both** `xoxb-` and `xapp-` tokens (prefixes enforced). ## Credentials ### Create a Slack incoming webhook 1. Visit [https://api.slack.com/apps](https://api.slack.com/apps) and click **Create New App → From scratch**. 2. Name the app (e.g. `OpenSRE`) and pick your workspace. 3. In the left sidebar, open **Incoming Webhooks** and toggle the feature **On**. 4. Click **Add New Webhook to Workspace**. 5. Pick the channel where messages should be posted and click **Allow**. 6. Copy the generated URL: ```text theme={null} https://hooks.slack.com/services/// ``` > [!NOTE] > > Treat this URL like a password — anyone holding it can post to your channel. If your workspace already has a Slack app you want to reuse, you can add a new webhook to it instead of creating a fresh app; the URL format is the same. ### Create Socket Mode tokens 1. In your Slack app settings, enable **Socket Mode** and create an app-level token with the `connections:write` scope (`xapp-…`). 2. Under **OAuth & Permissions**, grant the bot scopes below, then install the app and copy the bot token (`xoxb-…`). 3. Under **Event Subscriptions**, subscribe to the bot events `app_mention` and `message.im`. Add `agent_session_stopped` to make Slack's stop button work while OpenSRE is replying. 4. Optional: under **Interactivity & Shortcuts**, turn interactivity on and create a shortcut **On messages** named `Stop OpenSRE` with the callback ID `stop_opensre`. It shows in every message's hover menu (⋮) and stops the reply running in that thread. **Bot Token Scopes (Socket Mode chat + teammate tools):** | Scope | Needed for | | - | - | | `app_mentions:read` | Gateway `@mention` inbound | | `chat:write` | Gateway replies + `slack_reply_message` | | `im:history` | Gateway DMs + reading DM history | | `channels:history` | `slack_read_messages` + thread seeding in public channels | | `groups:history` | `slack_read_messages` in private channels | | `mpim:history` | `slack_read_messages` in multi-party DMs | | `groups:read` | Resolve private channel names | | `groups:write` | Manage / create private channels | | `files:read` | Inbound file-attachment downloads + discover Slack Lists (`files.list`) | | `files:write` | Upload files as the bot | | `lists:read` | `slack_read_list` — read Slack List rows (`slackLists.items.list`) | | `usergroups:read` | View workspace user groups | | `users.profile:read` | Speaker profile details | | `users:read` | `slack_list_team_members` + user lookup | > [!NOTE] > > Add these only if you use the feature — they are not in the current app config: `reactions:write` (gateway ack 👀 / ✓ reactions + `slack_add_reaction`), `channels:read` / `im:read` / `mpim:read` (resolve `#channel-name` and DM names → ID), and `channels:join` (`slack_join_channel`). Reinstall the app after adding scopes so the bot token picks them up. Invite the bot (`/invite @OpenSRE`) to private channels it should read or post in. > [!NOTE] > > `slack_search_messages` is the one tool a bot token can never run — Slack refuses bot tokens for workspace message search, no matter which scopes you grant. To enable search, add `search:read` under **OAuth & Permissions → User Token Scopes**, reinstall, and set `SLACK_ACCESS_TOKEN` to the resulting user token (`xoxp-…`). Leave it unset and search simply stays hidden from the agent; every other Slack tool works on the bot token alone. ### Allow your Slack user Find your Slack **member ID** (`U…`) in the Slack app: profile → **⋯** → **Copy member ID**. ```bash theme={null} opensre messaging allow -p slack -u U0123ABCD # or in REPL: /messaging allow -p slack -u U0123ABCD ``` or set `SLACK_ALLOWED_USERS=U0123ABCD` in `.env`. The integration store takes precedence when both are set. > [!WARNING] > > Use the Slack **member ID** (`U…`) from **Copy member ID**, not `@display-name`. Handles can be reassigned; inbound authorization only matches stable user IDs. For dogfood only, you may set `SLACK_ALLOW_OPEN_WORKSPACE=1` instead. ### DM pairing (optional) ```bash theme={null} opensre messaging pair -p slack ``` Then DM the bot (or mention it) and send `/pair `. ```bash theme={null} opensre messaging status -p slack opensre messaging revoke -p slack -u U0123ABCD ``` ## Slack tools Teammate tools (bot token) plus webhook blast. Credentials resolve inside the tools (never in tool-call traces). Shared client: `integrations/slack/web_client.py`. | Tool | What it does | Needs | Approval | | - | - | - | - | | `slack_send_message` | Post to the webhook's fixed channel | `SLACK_WEBHOOK_URL` | No | | `slack_reply_message` | Post to any channel/thread (`C…` or `#name`) | bot token, `chat:write` | No | | `slack_read_messages` | Read channel history or a thread (`thread_ts`) | bot token, history + list scopes | No | | `slack_search_messages` | Workspace message search | **user** token (`SLACK_ACCESS_TOKEN`), `search:read` | No | | `slack_list_team_members` | List workspace members | bot token, `users:read` | No | | `slack_join_channel` | Join a public channel | bot token, `channels:join` | Yes | | `slack_add_reaction` | React to a message ts | bot token, `reactions:write` | No | | `slack_read_list` | Read Slack List rows | bot token, `lists:read` | No | | `replay_slack_thread_locally` | Replay a Slack thread into the local session | bot token | No | Prefer `slack_reply_message` / `slack_read_messages` when the bot token is configured and the user names a channel or thread. Use `slack_send_message` only for the fixed webhook channel. For "add task", "remind me", and "what should we focus on next" requests, use OpenSRE work management. Gateway turns can default reminders and recurring check-ins to the current Slack channel. Example prompts: * "Read the last 20 messages in #incidents and summarize." * "Search Slack for 'Windows install' this week." * "Join #ops then reply that mitigation is rolled out." * "Who is on the team, and what does Vaibhav do?" * "Add task: ping owners about the deploy window." * "Get GitHub star velocity for the last 30 days on `your-org/your-repo` and post it to Slack." * "Generate a Slack-ready work status report for `your-org/your-repo` and post it." The last two need [GitHub configured](/docs/integrations/code/github) and name the repository explicitly (`your-org/your-repo`) so the GitHub tools can resolve it — see [GitHub workflow tools](/docs/integrations/code/github-workflow-tools) for what they can report on. Message tools post without a separate approval prompt; check the target channel for the delivered message. ### Production Engineer schedules (Slack) Wire morning digests, PR sweeps, and `/loops` prompt loops to Slack. The gateway daemon (or `opensre cron start`) must be running so the scheduler can deliver when a tick finishes — the report arrives proactively; you do not need to ask in chat. **Destination options:** | Setup | How to target Slack | | - | - | | Incoming webhook | Omit `--chat-id` — the webhook URL is bound to one channel | | Bot token (Socket Mode) | Set `SLACK_DEFAULT_CHAT_ID` (or pass `--chat-id` / `/loops add --slack-chat-id`) | Copy the channel id (`C…`) from Slack channel details. During `opensre integrations setup slack` (Socket Mode or Both), you can persist a default channel id for scheduled delivery. The default channel is taken from the same place as the bot token: if the token is in `.env`, set `SLACK_DEFAULT_CHAT_ID` there too. A channel saved only in the integration store is not paired with an env or task-level token. ```bash theme={null} opensre sentry digest schedule add \ --cron "0 8 * * mon-fri" --tz Europe/London \ --provider slack --chat-id C0123ABCD # Webhook: omit --chat-id opensre cron add --kind github_pr_sweep --cron "0 9 * * mon-fri" \ --tz Europe/London --provider slack # Bot token: default channel or explicit --chat-id export SLACK_DEFAULT_CHAT_ID=C0123ABCD opensre cron add --kind github_pr_sweep --cron "0 9 * * mon-fri" \ --tz Europe/London --provider slack --chat-id C0123ABCD ``` See [Scheduled deliveries](/docs/platform/cron) and [Sentry morning digest](/docs/integrations/incidents/sentry#morning-digest-scheduled). ### Two-way chat gateway (Socket Mode) No public inbound HTTPS URL is required — the gateway holds an outbound websocket. Each thread is its own conversation. ```bash theme={null} opensre gateway start opensre gateway status # want: slack: connected via socket mode ``` | Command | What it does | | - | - | | `opensre gateway start` | Start the daemon (web, Telegram, Slack, scheduler) | | `opensre gateway status` | Show daemon and component state | | `opensre gateway logs -f` | Follow live gateway logs | | `opensre gateway stop` | Stop the daemon | Built-in chat commands: `/new` (fresh session), `/help`, `/pair `, and `stop` to stop the reply running in the thread. While a turn runs, the gateway adds an 👀 (`eyes`) reaction on the inbound message, then swaps to ✅ (`white_check_mark`) when the reply is finalized. If Slack is not configured the daemon still runs the other components and `opensre gateway status` shows `slack: not configured`. ### Deploying the Slack gateway The Slack gateway backend is deployed and operated separately — not from this repo. Configure the integration here (tokens and scopes above); provisioning the hosted backend is out of scope for this repo. The EC2 gateway path (`make deploy-gateway`) is **Telegram-only** and ignores `SLACK_*` variables with a validation warning: Slack Socket Mode is single-consumer, so a second gateway holding the same tokens would split events. ## Verify ```bash theme={null} opensre integrations verify slack ``` A successful run reports the integration as `passed`. Webhook-only configs confirm the URL is present; Socket Mode configs also run Slack `auth.test` on the bot token. To also confirm webhook delivery, add the `--send-slack-test` flag: ```bash theme={null} opensre integrations verify slack --send-slack-test ``` This posts a small test message to the configured channel. ## Troubleshooting | Symptom | Fix | | - | - | | **`error: webhook_url is required.` from setup** | You chose webhook (or both) and left the URL empty. Re-run and paste the full URL including `https://`, or choose Socket Mode only. | | **`slack: not configured` from gateway status** | Missing `SLACK_BOT_TOKEN` / `SLACK_APP_TOKEN` or empty allowlist without `SLACK_ALLOW_OPEN_WORKSPACE=1`. Allow yourself, then `gateway stop` / `start`. | | **Connected, but deny reply** | Your `U…` is not in the allowlist. Pair with `/pair ` or `messaging allow`. | | **`invalid_payload` or `channel_not_found`** | Recreate the webhook for the current channel and update `SLACK_WEBHOOK_URL`. | | **Messages posted to the wrong channel** | A webhook is bound to the channel it was created against — create a new webhook for the new channel. | | **Webhook returns `no_service`** | The Slack app or webhook was deleted. Re-create it and update the URL. | ## Security * Treat webhook URLs and bot/app tokens like passwords. * Prefer least-privilege bot scopes; add optional scopes only when you use those tools. * Keep `SLACK_ALLOWED_USERS` set in production; treat open-workspace mode as dogfood only. * Store secrets in the integration store / keyring / secret manager — not in source control. # SMTP Source: https://opensre.com/docs/integrations/messaging/smtp Configure SMTP so OpenSRE can send email through your existing relay. ## Overview OpenSRE can send email through any existing SMTP relay you already use. You do **not** need to run your own mail server. Typical providers: * Google Workspace / Gmail SMTP * Microsoft 365 SMTP * AWS SES SMTP * SendGrid / Postmark / Mailgun SMTP * local dev SMTP sinks like Mailpit or MailHog ## Prerequisites * An SMTP host you can reach from OpenSRE * A sender address the relay will accept * Optional username/password if the relay requires auth ## Setup ### Option 1: CLI wizard ```bash theme={null} opensre integrations setup smtp ``` You'll be prompted for: * SMTP host * SMTP port * security mode: `starttls`, `ssl`, or `none` * optional username/password * sender address * optional default recipient SMTP is configured separately from first-run onboarding — use this setup command. ### Option 2: Environment variables ```bash theme={null} SMTP_HOST=smtp.example.com SMTP_PORT=587 SMTP_SECURITY=starttls SMTP_USERNAME=mailer SMTP_PASSWORD=secret SMTP_FROM_ADDRESS=opensre@example.com SMTP_DEFAULT_TO=team@example.com ``` | Variable | Default | Description | | - | - | - | | `SMTP_HOST` | — | SMTP hostname | | `SMTP_PORT` | `587` | SMTP port | | `SMTP_SECURITY` | `starttls` | `starttls`, `ssl`, or `none` | | `SMTP_USERNAME` | — | Optional auth username (set with password) | | `SMTP_PASSWORD` | — | Optional auth password (set with username) | | `SMTP_FROM_ADDRESS` | — | Sender address | | `SMTP_DEFAULT_TO` | — | Default recipient for outbound email | If authentication is required, set **both** `SMTP_USERNAME` and `SMTP_PASSWORD`. ## Credentials Use any SMTP relay your org already trusts. For local testing, Mailpit is a good fake SMTP target: ```bash theme={null} docker run -p 1025:1025 -p 8025:8025 axllent/mailpit ``` ```bash theme={null} SMTP_HOST=localhost SMTP_PORT=1025 SMTP_SECURITY=none SMTP_FROM_ADDRESS=opensre@example.com SMTP_DEFAULT_TO=team@example.com ``` ## Verify ```bash theme={null} opensre integrations verify smtp ``` A passing verification confirms OpenSRE can: * connect to the SMTP server * negotiate TLS when configured * authenticate when credentials are provided Sample success output: ```text theme={null} smtp: passed — Connected to SMTP server successfully. ``` ## Troubleshooting | Detail | Likely cause | | - | - | | `Missing recipient email address` | No `SMTP_DEFAULT_TO` is configured yet. | | `username and password must both be set` | Only one auth field was provided. | | `from_address must look like an email address` | Sender address is malformed. | | `SMTP connection failed` | Host/port/TLS settings are wrong, or the relay is unreachable. | ## Security * Prefer app passwords / SMTP credentials scoped to a mailer account. * Store passwords in `.env` or your secret manager — not in source control. * Use `starttls` or `ssl` in production; reserve `none` for local sinks. # Telegram Source: https://opensre.com/docs/integrations/messaging/telegram Chat with the agent and deliver alerts and reports to a Telegram chat or channel. ## Overview OpenSRE's Telegram integration delivers agent messages, scheduled reports, and alarms to any chat your bot has been added to — useful for mobile-first on-call rotations and personal alerting. With the gateway running, you can also DM the agent for two-way chat. Start the interactive shell with `opensre` (no subcommand). Slash commands below are run from that REPL. ## Prerequisites * A Telegram account. * The Telegram mobile or desktop app, signed in. * The chat (group, channel, or direct message) where you want messages delivered. ## Setup ### Option 1: CLI setup (recommended) ```bash theme={null} opensre integrations setup telegram ``` Interactive shell: `/integrations setup telegram`. You can also ask: “Can you configure Telegram for me?” 1. **Connect your bot.** Setup links to [BotFather](https://t.me/BotFather) and explains how to create a bot or retrieve an existing token before asking you to paste it into a hidden input. 2. **Choose a chat.** Press Start in your bot's private chat, or add it to your group and send the command shown by setup. OpenSRE finds recent chats and lets you choose by name. For a public channel, you can also enter its `@name`. 3. **Verify and save.** OpenSRE checks the bot and selected chat, saves the connection, and shows a verified summary. Failed checks can be retried without starting over. Press Ctrl+C to cancel. If chat discovery finds nothing, send a new message and choose **look again**. If the bot already uses a webhook, setup keeps it active; use a public `@channelname` or an existing chat ID instead. > [!NOTE] > > Setup confirms the bot can *see* the chat, not that it may post there. For a channel the bot still needs the **Post Messages** permission. ### Option 2: Environment variables ```bash theme={null} TELEGRAM_BOT_TOKEN=: TELEGRAM_DEFAULT_CHAT_ID= TELEGRAM_ALLOWED_USERS=123456789 # for two-way gateway chat ``` | Variable | Description | | - | - | | `TELEGRAM_BOT_TOKEN` | Bot HTTP API token from BotFather. Required. | | `TELEGRAM_DEFAULT_CHAT_ID` | Default delivery destination. Required for delivery to work. | | `TELEGRAM_ALLOWED_USERS` | Comma-separated **numeric** user ids allowed for gateway chat. | > [!NOTE] > > **Credential resolution.** Telegram delivery surfaces resolve the bot token from the integration store first, then `resolve_env_credential("TELEGRAM_BOT_TOKEN")` (process env, then the credentials file). Chat id is non-secret: `--chat-id` → store `default_chat_id` → `TELEGRAM_DEFAULT_CHAT_ID` env (plain `os.getenv`). ## Credentials ### Create a bot with BotFather [BotFather](https://t.me/BotFather) is Telegram's official bot for creating other bots. 1. Open Telegram and search for `@BotFather`. Open the chat and tap **Start**. 2. Send `/newbot`. 3. When prompted, send a **display name** for your bot (e.g. `OpenSRE Alerts`). 4. Send a **username** that ends in `bot` (e.g. `opensre_alerts_bot`). It must be globally unique. 5. BotFather replies with an **HTTP API token** of the form `:`. Copy it — treat it like a password. > [!TIP] > > You can change the bot name, picture, and description later by sending `/mybots` to BotFather and selecting your bot. ### Add the bot to a chat **Group chat** 1. Open the group where you want messages to land. 2. Tap the group name → **Add members** → search for your bot username → **Add**. 3. By default, bots in groups only see messages addressed to them, which is fine for delivery-only. **Channel** 1. Open your channel and tap its name. 2. Tap **Administrators → Add Administrator**, search for your bot, and add it. 3. Grant the **Post Messages** permission. No other admin permissions are required. **Direct message** Open a chat with your bot directly (search its username) and send `/start`. The bot will not reply, but Telegram registers the chat so the bot can message you. ### Find your `chat_id` The **chat ID** identifies where the bot should post. It is required — without it the bot has nowhere to send anything. > [!TIP] > > **Posting to a public channel?** Skip this step — setup accepts the channel's `@name` (for example `@acme_alerts`) directly. Private groups and DMs have no `@name`, so they need the steps below. 1. Send any message in the destination chat — for a channel, post anything; for a DM, send `/start` to your bot. 2. In a browser, open: ``` https://api.telegram.org/bot/getUpdates ``` 3. In the JSON response, look for a `chat.id` field: | Chat type | Format | Example | | - | - | - | | Direct message with a user | Positive integer | `123456789` | | Group | Negative integer | `-987654321` | | Large group or channel | Negative integer starting with `-100` | `-1001234567890` | Copy the entire value, **including the leading minus sign** for groups and channels. > [!NOTE] > > If `getUpdates` returns an empty array, post a fresh message in the chat and reload — Telegram only buffers recent updates. ## Telegram tools | Tool | What it does | | - | - | | `telegram_send_message` | Send a message to the default chat or an explicit `chat_id` | ```json theme={null} { "name": "telegram_send_message", "arguments": { "chat_id": "-1001234567890", "message": "DB CPU is back below 70%; keeping the incident open for 10 minutes." } } ``` The tool resolves the bot token from the integration store, then `resolve_env_credential("TELEGRAM_BOT_TOKEN")` (env then keyring). If `chat_id` is omitted, it sends through the configured `default_chat_id`. Delivery is an external side effect. Its result includes a stable `status`, `sent`, `error_type`, `chat_id`, `reply_to_message_id`, and `message_length` shape so follow-up tool calls can tell configuration failures from Telegram delivery failures. ### Two-way chat gateway (DM text) > **Skip this if you only need outbound delivery** (alerts, cron reports). Setup + verify are sufficient for that. v1 supports **text-only direct messages** (no groups, voice, or attachments). #### Allow your Telegram user Find your **numeric user id** with [@userinfobot](https://t.me/userinfobot), then: ```bash theme={null} opensre messaging allow -p telegram -u 123456789 # or: TELEGRAM_ALLOWED_USERS=123456789 ``` Interactive shell: `/messaging allow -p telegram -u 123456789`. > [!WARNING] > > Use the **numeric user id** (Telegram's `from.id`), not a `@username` and not the bot handle. Inbound authorization only ever matches the numeric id. The CLI rejects non-numeric Telegram ids, but older entries may still be wrong — check with `/messaging status -p telegram`. #### DM pairing (optional) ```bash theme={null} opensre messaging pair -p telegram ``` Then DM your bot and send `/pair `. ```bash theme={null} opensre messaging status -p telegram opensre messaging revoke -p telegram -u 123456789 ``` #### Start the gateway daemon ```bash theme={null} opensre gateway start ``` | Command | What it does | | - | - | | `opensre gateway start` | Start the daemon (web, Telegram chat, task scheduler) | | `opensre gateway start -f` | Run attached to the terminal instead | | `opensre gateway status` | Show the daemon and each component's state | | `opensre gateway logs [-n N] [-f]` | Print (or follow) the daemon logs | | `opensre gateway stop` | Stop the daemon | Also available in the REPL via `/gateway start|status|logs|stop`. Logs: `~/.opensre/gateway/gateway.log`. If Telegram is not configured the daemon still runs the other components and status shows `telegram: not configured`. The gateway uses long polling — no public HTTPS URL or port forwarding is required for local use. Built-in commands: `/new`, `/help`, `/pair `. The gateway uses the same headless agent harness as the interactive shell: the agent grounds its prompt in your configured integrations and calls their tools to answer. Cron deliveries use the outbound-only path — they do not require the gateway process. #### Deploying the gateway to a remote host Running the gateway on a server with `make deploy-gateway` (EC2) is different from local use: **the remote host cannot read your local machine's keychain**. Guided setup stores the bot token (and your LLM API key) in the system keyring, which does not travel to the deployed instance. So `make deploy-gateway` validates that the required secrets are present as **plaintext env vars** in `.env`: ```bash theme={null} TELEGRAM_BOT_TOKEN=: TELEGRAM_ALLOWED_USERS=123456789 ANTHROPIC_API_KEY=... # or your provider's key env ``` If deploy reports `MISSING: TELEGRAM_BOT_TOKEN` even though local setup succeeded, copy the values into `.env` (or `.env.deploy.example`) for the deploy. ### Scheduled delivery Configure Telegram first (setup + verify — the gateway is **not** required). #### Cron (recurring reports) ```bash theme={null} opensre cron add --kind github_pr_sweep --cron "0 9 * * mon-fri" \ --tz Asia/Kolkata --provider telegram --chat-id ``` See [Cron](/docs/platform/cron). ## Verify ```bash theme={null} opensre integrations verify telegram ``` Interactive shell: `/integrations verify telegram` or `/verify telegram`. This calls Telegram's [`getMe`](https://core.telegram.org/bots/api#getme) endpoint. On success it reports the bot `@username`. It does **not** start a listening process and does not test two-way communication or delivery chat routing. ## Troubleshooting Verify only calls `getMe`, so it surfaces token-validity errors but cannot detect chat-routing problems. Delivery-time errors appear in OpenSRE logs as `[telegram] post message failed: `. | Symptom | Fix | | - | - | | **`Missing bot_token`** | Set `TELEGRAM_BOT_TOKEN` and restart long-running processes. | | **`401 Unauthorized`** | Token invalid/revoked — regenerate in BotFather and update credentials. | | **`chat not found` (delivery)** | Re-add the bot and re-fetch `chat_id` from `getUpdates`. | | **`bot was kicked…`** | Re-add the bot; channels need **Post Messages** admin permission. | | **Messages never arrive, but verify passes** | Re-fetch `chat_id` — groups upgraded to `-100…` prefixes may have changed. | | **Gateway: `User is not in the allowed users list`** | Allow the numeric user id or complete `/messaging pair -p telegram`. | ## Security * Treat the bot token like a password; prefer `~/.opensre/credentials.json`. * Use **numeric** user ids on the allow-list — never `@username`. * Prefer a dedicated bot for OpenSRE. * For remote deploy, put secrets in plaintext `.env` on the host (keyring does not travel). * Store secrets out of source control. # Twilio SMS Source: https://opensre.com/docs/integrations/messaging/twilio-sms Send OpenSRE notifications via Twilio SMS as a standalone, independently configurable channel. ## Overview OpenSRE's Twilio integration sends notifications via **SMS** using the Twilio Messaging API. It is configured independently of the existing `whatsapp` integration, though both can share the same Twilio account credentials. Delivery-only — no gateway or pairing. > [!NOTE] > > WhatsApp delivery is owned entirely by the separate `whatsapp` integration (`opensre integrations setup whatsapp`) — see the [WhatsApp](/docs/integrations/messaging/whatsapp) page. The `twilio` integration documented here is SMS-only. For voice paging on critical incidents, route through PagerDuty, which provides voice escalation with proper acknowledgement. ## Prerequisites * A Twilio account: [Sign up](https://www.twilio.com/try-twilio). * A Twilio-provisioned phone number **or** a Messaging Service SID. ## Setup ### Option 1: CLI wizard (recommended) ```bash theme={null} opensre integrations setup twilio ``` You'll be prompted for: * **Twilio Account SID** (starts with `AC...`) * **Twilio Auth Token** * **Twilio SMS From number** (E.164, e.g. `+14155551234`) — or leave blank and provide a **Messaging Service SID** (starts with `MG...`) * optional **default recipient** Twilio SMS is configured separately from first-run onboarding — use this setup command. The service key is `twilio` (docs path `/messaging/twilio-sms`). ### Option 2: Environment variables ```bash theme={null} TWILIO_ACCOUNT_SID=AC... TWILIO_AUTH_TOKEN=your_auth_token # Set either TWILIO_SMS_FROM or TWILIO_SMS_MESSAGING_SERVICE_SID TWILIO_SMS_FROM=+14155551234 TWILIO_SMS_MESSAGING_SERVICE_SID= TWILIO_SMS_DEFAULT_TO=+1234567890 ``` | Variable | Description | | - | - | | `TWILIO_ACCOUNT_SID` | Twilio account SID (shared with WhatsApp) | | `TWILIO_AUTH_TOKEN` | Twilio auth token (shared with WhatsApp) | | `TWILIO_SMS_FROM` | SMS from number (XOR with Messaging Service SID) | | `TWILIO_SMS_MESSAGING_SERVICE_SID` | Messaging Service SID (XOR with from number) | | `TWILIO_SMS_DEFAULT_TO` | Default recipient for auto-delivery | The `twilio` env-bootstrap activates when the account + token are set and an SMS sender (`TWILIO_SMS_FROM` or `TWILIO_SMS_MESSAGING_SERVICE_SID`) is present. The legacy `whatsapp` record is bootstrapped independently from `TWILIO_WHATSAPP_FROM`. ## Credentials 1. Create or open a Twilio account. 2. Provision a phone number **or** Messaging Service for SMS. 3. Copy Account SID and Auth Token. 4. Set exactly one SMS sender: `TWILIO_SMS_FROM` **or** `TWILIO_SMS_MESSAGING_SERVICE_SID` (enforced by the verifier). 5. Set `TWILIO_SMS_DEFAULT_TO` as the default recipient. ## Twilio tools | Tool | What it does | | - | - | | `twilio_notify` | Send a short SMS notification body; returns the Twilio Message SID | ```json theme={null} { "name": "twilio_notify", "arguments": { "to": "+14155550000", "body": "DB CPU > 95% — paging on-call" } } ``` Message bodies are truncated to the SMS body limit (1600 chars) and posted via the Twilio Messaging API to the configured recipient. ## Verify ```bash theme={null} opensre integrations verify twilio ``` A passing verification confirms: * The Twilio account credentials authenticate against the Twilio Account API. * The SMS channel is enabled and has a usable sender (`from_number` or `messaging_service_sid`). Sample passing output: ``` twilio: passed — Connected to Twilio account My Company; SMS channel ready. ``` If the account authenticates but the SMS channel has no sender, verification returns `failed` with a message telling you to set a `from_number` or `messaging_service_sid`. ## Troubleshooting | Detail | Likely cause | | - | - | | `Missing account_sid` | `TWILIO_ACCOUNT_SID` is unset. | | `Missing auth_token` | `TWILIO_AUTH_TOKEN` is unset. | | `Twilio API check failed: 401` | The SID/token pair is invalid. | | `SMS channel is not ready` | No SMS sender — set `TWILIO_SMS_FROM` or `TWILIO_SMS_MESSAGING_SERVICE_SID`. | | SMS never arrives but verify passes | Confirm sender is provisioned for the destination country; recipient in E.164; trial accounts need verified caller IDs. Inspect Twilio Console Messages for error codes (`30007`, `21610`, etc.). | ## Security * Treat Twilio auth tokens like passwords. * Prefer a dedicated Twilio project for OpenSRE. * Store credentials in `.env` or your secret manager — not in source control. # WhatsApp Source: https://opensre.com/docs/integrations/messaging/whatsapp Deliver OpenSRE messages to WhatsApp via Twilio. ## Overview OpenSRE's WhatsApp integration delivers messages to any WhatsApp number via Twilio's Messaging API. Delivery-only — no gateway or pairing. > [!NOTE] > > This integration uses Twilio as the WhatsApp provider. SMS is a separate integration — see [Twilio SMS](/docs/integrations/messaging/twilio-sms). ## Prerequisites * A Twilio account: [Sign up](https://www.twilio.com/try-twilio) * Twilio WhatsApp Sandbox (for demos) or a production WhatsApp-enabled sender ## Setup ### Option 1: CLI wizard (recommended) ```bash theme={null} opensre integrations setup whatsapp ``` You'll be prompted for: * **Twilio Account SID** (starts with `AC...`) * **Twilio Auth Token** * **Twilio WhatsApp From number** (for example `whatsapp:+14155238886`) * **Default recipient phone number** (optional) WhatsApp is configured separately from first-run onboarding — use this setup command. ### Option 2: Environment variables ```bash theme={null} TWILIO_ACCOUNT_SID=AC... TWILIO_AUTH_TOKEN=your_auth_token TWILIO_WHATSAPP_FROM=whatsapp:+14155238886 WHATSAPP_DEFAULT_TO=+1234567890 ``` | Variable | Description | | - | - | | `TWILIO_ACCOUNT_SID` | Twilio account SID (shared with SMS) | | `TWILIO_AUTH_TOKEN` | Twilio auth token (shared with SMS) | | `TWILIO_WHATSAPP_FROM` | WhatsApp sender (auto-prefixed `whatsapp:` if needed) | | `WHATSAPP_DEFAULT_TO` | Default recipient (international format) | ## Credentials 1. Create or open a Twilio account. 2. Enable WhatsApp Sandbox or a production WhatsApp sender. 3. Copy Account SID and Auth Token from the Twilio Console. 4. Set `TWILIO_WHATSAPP_FROM` to your WhatsApp-enabled sender. 5. Set `WHATSAPP_DEFAULT_TO` as the default recipient. From/to numbers are auto-prefixed with `whatsapp:` in delivery. No WhatsApp-specific agent tools are registered — this is a delivery channel only. Long messages are truncated to WhatsApp limits. ## Verify ```bash theme={null} opensre integrations verify whatsapp ``` This calls the Twilio Account API to verify the credentials. It does **not** prove WhatsApp sender readiness for a specific destination. ## Troubleshooting | Symptom | Fix | | - | - | | **`Missing account_sid`** | Set `TWILIO_ACCOUNT_SID` | | **`Missing auth_token`** | Set `TWILIO_AUTH_TOKEN` | | **`Twilio API check failed: 401`** | The SID/token pair is invalid | | **Messages never arrive but verify passes** | Confirm `TWILIO_WHATSAPP_FROM` is a valid WhatsApp sender; recipient in international format; for sandbox, join from the destination WhatsApp number | ## Security * Treat Twilio auth tokens like passwords. * Prefer a dedicated Twilio project for OpenSRE. * Store credentials in `.env` or your secret manager — not in source control. # Azure Monitor Source: https://opensre.com/docs/integrations/monitoring/azure-monitor Connect Azure Monitor Log Analytics so OpenSRE can pull KQL log evidence ## Overview OpenSRE queries Azure Monitor Log Analytics through the public Query REST API. Each query is bounded by a `take` clause so result sets stay within a safe row limit. ## Prerequisites * Azure subscription with at least one **Log Analytics Workspace** collecting logs * A **Microsoft Entra ID (Azure AD) app registration** authorized to query the workspace * The **Log Analytics Reader** role granted on the workspace (or its resource group / subscription) to the app's service principal * Network access from the OpenSRE environment to `https://api.loganalytics.io` (or the sovereign cloud equivalent) over HTTPS ## Setup ```bash theme={null} opensre integrations setup azure ``` That writes the workspace id, bearer token, and optional fields to the store, keyring, and project `.env` in one pass. You can still set the same values by hand. ### Option 1: Environment variables Add to your `.env`: ```bash theme={null} AZURE_LOG_ANALYTICS_WORKSPACE_ID=00000000-0000-0000-0000-000000000000 AZURE_LOG_ANALYTICS_TOKEN= AZURE_LOG_ANALYTICS_ENDPOINT=https://api.loganalytics.io AZURE_TENANT_ID= # optional, informational AZURE_SUBSCRIPTION_ID= # optional, informational AZURE_MAX_RESULTS=100 # optional, capped at 200 ``` | Variable | Default | Description | | - | - | - | | `AZURE_LOG_ANALYTICS_WORKSPACE_ID` | — | **Required.** Log Analytics Workspace ID (GUID) from the Azure portal | | `AZURE_LOG_ANALYTICS_TOKEN` | — | **Required.** Microsoft Entra ID OAuth2 bearer token with `Data.Read` on the workspace | | `AZURE_LOG_ANALYTICS_ENDPOINT` | `https://api.loganalytics.io` | Override for sovereign clouds (for example `https://api.loganalytics.azure.us` for Azure Government) | | `AZURE_TENANT_ID` | — | Microsoft Entra ID tenant ID (informational; useful for multi-tenant audits) | | `AZURE_SUBSCRIPTION_ID` | — | Azure subscription ID (informational) | | `AZURE_MAX_RESULTS` | `100` | Per-query row cap; OpenSRE clamps to a hard maximum of `200` | ### Option 2: Persistent store Credentials are persisted to `~/.opensre/integrations.json` with `0o600` permissions: ```json theme={null} { "version": 1, "integrations": [ { "id": "azure-prod", "service": "azure", "status": "active", "credentials": { "workspace_id": "00000000-0000-0000-0000-000000000000", "access_token": "", "endpoint": "https://api.loganalytics.io", "tenant_id": "", "subscription_id": "", "max_results": 100 } } ] } ``` ## Credentials ### 1. Find the Workspace ID 1. In the Azure portal, open **Log Analytics workspaces** and select your workspace. 2. On the workspace **Overview** page, copy **Workspace ID** (a GUID). ### 2. Register an Azure AD application 1. Open **Microsoft Entra ID** → **App registrations** → **New registration**. 2. Give the app a name (for example `opensre-log-analytics`) and register it as a single-tenant app. 3. From the app's **Overview** page, copy the **Application (client) ID** and the **Directory (tenant) ID**. 4. Open **Certificates & secrets** → **New client secret**, copy the **secret value** (it is shown only once). ### 3. Grant `Log Analytics Reader` on the workspace 1. Open the Log Analytics workspace in the portal. 2. Go to **Access control (IAM)** → **Add** → **Add role assignment**. 3. Pick the **Log Analytics Reader** role and assign it to the service principal created above. ### 4. Obtain a bearer token (client credentials flow) ```bash theme={null} curl -s -X POST \ "https://login.microsoftonline.com/$AZURE_TENANT_ID/oauth2/v2.0/token" \ -d "grant_type=client_credentials" \ -d "client_id=$AZURE_CLIENT_ID" \ -d "client_secret=$AZURE_CLIENT_SECRET" \ -d "scope=https://api.loganalytics.io/.default" \ | jq -r '.access_token' ``` Set the resulting token as `AZURE_LOG_ANALYTICS_TOKEN`. Tokens expire (usually after 60 minutes) — see the **Token rotation** note in [Security](#security). ## Tools OpenSRE exposes one tool against an Azure Monitor workspace: ### `query_azure_monitor_logs` `POST`s a KQL query to `/v1/workspaces//query` and returns the first table flattened into row dicts. Arguments the planner supplies: | Argument | Default | Description | | - | - | - | | `query` | `AppTraces \| order by TimeGenerated desc \| take ` | KQL query text | | `time_range_minutes` | `60` | Sent as the `timespan` (`PTM`) | | `limit` | `50` | Per-query row cap; clamped to `max_results` (hard limit `200`) | OpenSRE always appends a `| take ` clause when one is missing, so the workspace never returns more rows than the configured cap. `AZURE_MAX_RESULTS` defaults to `100` and is hard-capped at `200`. ## Verify ```bash theme={null} opensre integrations verify azure ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL azure local env passed Configured for Azure Log Analytics workspace 00000000-0000-0000-0000-000000000000 via https://api.loganalytics.io. ``` The verify step is a credential-shape check — it does not call the workspace. To exercise the live path, point OpenSRE at a synthetic alert that names `azure` as the source and inspect the resulting evidence. ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Token is missing, expired, or scoped to the wrong audience. Regenerate with `scope=https://api.loganalytics.io/.default` and confirm the service principal has **Log Analytics Reader** on the workspace. | | **403 Forbidden** | The token is valid but the principal lacks `Data.Read`. Re-check the role assignment on the workspace (or its parent resource group). | | **Empty result set** | Either the KQL `where` filter excludes everything or the workspace has no data in the requested timespan. Run the same query in **Logs** in the portal to confirm. | | **Wrong endpoint / DNS error** | Sovereign clouds use a different host (for example `https://api.loganalytics.azure.us` for US Government, `https://api.loganalytics.azure.cn` for China). Set `AZURE_LOG_ANALYTICS_ENDPOINT` accordingly. | | **`Missing workspace_id` / `Missing access_token`** | One or both required credentials are absent. Confirm both env vars (or both fields in the persistent store) are populated. | ## Security * Use a **dedicated app registration** for OpenSRE — do not reuse a personal token or a broadly-scoped service principal. * Grant only **Log Analytics Reader** on the workspace; OpenSRE only needs read access to query data. * Keep the client secret out of source control — store it in `.env` or in a secret manager and only export it long enough to mint a token. * **Rotate the bearer token** before its 60-minute expiry. Long-lived deployments should re-mint the token from the client secret on a schedule rather than pasting a static token into `.env`. * The integration is **read-only**: OpenSRE only issues `POST /v1/workspaces//query` requests with a `take`-bounded KQL string. ## Extras ### Example KQL queries Recent application errors: ```kql theme={null} AppTraces | where SeverityLevel >= 3 | where TimeGenerated > ago(15m) | project TimeGenerated, OperationName, Message, AppRoleName | order by TimeGenerated desc | take 50 ``` Error count by severity over the last hour: ```kql theme={null} AppTraces | where TimeGenerated > ago(1h) | summarize count() by SeverityLevel | order by SeverityLevel desc ``` Failed dependency calls correlated with a request id: ```kql theme={null} AppDependencies | where Success == false | where TimeGenerated > ago(30m) | project TimeGenerated, Name, ResultCode, DurationMs, OperationId | order by TimeGenerated desc | take 50 ``` # Better Stack Telemetry Source: https://opensre.com/docs/integrations/monitoring/betterstack Connect Better Stack so OpenSRE can pull log evidence from your Telemetry sources ## Overview OpenSRE uses Better Stack's ClickHouse SQL Query API to read log evidence. It queries the configured source via `remote(_logs)` for recent rows and `s3Cluster(primary, _s3)` for historical rows, bounded by the incident time window. ## Prerequisites * A Better Stack account with at least one **Telemetry source** collecting logs * A **ClickHouse HTTP client** credential pair (username + password) generated from the dashboard * The **region-specific query endpoint** for your workspace (for example `https://eu-nbg-2-connect.betterstackdata.com`) * Network access from the OpenSRE environment to that endpoint over HTTPS ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup betterstack ``` You will be prompted for the query endpoint, username, password, and an optional comma-separated list of source IDs (planner hint). ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} BETTERSTACK_QUERY_ENDPOINT=https://eu-nbg-2-connect.betterstackdata.com BETTERSTACK_USERNAME= BETTERSTACK_PASSWORD= BETTERSTACK_SOURCES=t123456_myapp,t123456_gateway ``` | Variable | Default | Description | | - | - | - | | `BETTERSTACK_QUERY_ENDPOINT` | — | **Required.** Region-specific SQL API host (for example `https://eu-nbg-2-connect.betterstackdata.com`) | | `BETTERSTACK_USERNAME` | — | **Required.** Username from **Connect ClickHouse HTTP client** | | `BETTERSTACK_PASSWORD` | — | **Required.** Password from the same dashboard flow | | `BETTERSTACK_SOURCES` | *(empty)* | Optional comma-separated list of **base source IDs** (for example `t123456_myapp`). The integration appends `_logs` / `_s3` internally. If omitted, the planner must derive the source from alert metadata (via a `betterstack_source` annotation) | ### Option 3: Persistent store Credentials are persisted to `~/.opensre/integrations.json` with `0o600` permissions: ```json theme={null} { "version": 1, "integrations": [ { "id": "betterstack-prod", "service": "betterstack", "status": "active", "credentials": { "query_endpoint": "https://eu-nbg-2-connect.betterstackdata.com", "username": "", "password": "", "sources": ["t123456_myapp"] } } ] } ``` ## Credentials From the Better Stack dashboard: 1. Open **Telemetry** and pick the source you want OpenSRE to query. 2. In the source sidebar, open **Integrations** → **Connect ClickHouse HTTP client**. 3. Copy the generated **username**, **password**, and **query endpoint**. The endpoint's subdomain encodes the region (for example `eu-nbg-2-connect`, `us-connect`). 4. The **base source ID** (for example `t123456_myapp`) is shown above the integration panel and is the value to supply for `BETTERSTACK_SOURCES`. Use the base name only — OpenSRE appends `_logs` and `_s3` internally. ## Tools OpenSRE exposes one tool against a Better Stack source: ### `query_betterstack_logs` Returns `(dt, raw)` pairs by UNIONing: * Recent rows from `remote(_logs)` * Historical rows from `s3Cluster(primary, _s3) WHERE _row_type = 1` Arguments the planner supplies: | Argument | Default | Description | | - | - | - | | `source` | first configured `sources` entry | Base identifier (for example `t123456_myapp`) | | `since` / `until` | — | ISO-8601 timestamps that bound the `dt` column; typically derived from the alert window | | `limit` | `500` | Row cap | All queries run with `FORMAT JSONEachRow` and `output_format_pretty_row_numbers=0`. Source names are validated against `^[A-Za-z0-9_]+$` to prevent identifier injection. ## Verify ```bash theme={null} opensre integrations verify betterstack ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL betterstack store passed Connected to Better Stack SQL API at https://eu-nbg-2-connect.betterstackdata.com ``` The verify step issues a cheap probe (`SELECT 1 FORMAT JSONEachRow`) against the configured endpoint using the stored credentials. ## Troubleshooting | Symptom | Fix | | - | - | | **Authentication failed (401)** | Regenerate credentials via the dashboard's **Connect ClickHouse HTTP client** flow; confirm `BETTERSTACK_USERNAME` / `BETTERSTACK_PASSWORD` match exactly. | | **Endpoint not found / DNS error** | The region subdomain is wrong. Copy the endpoint directly from the dashboard (for example `eu-nbg-2-connect`, `us-connect`, `eu-fsn-3-connect`). | | **`invalid source name` error on query** | The `source` argument contains characters outside `[A-Za-z0-9_]`. Use the base ID shown in the dashboard — no dashes, no quotes, no whitespace. | | **Empty result set for a known-busy source** | Check that the alert window (`since` / `until`) actually overlaps with the source's `dt` range; historical rows older than recent retention live in `s3Cluster(primary, _s3)`. | | **`planner did not configure a source` error** | Either set `BETTERSTACK_SOURCES`, or ensure the alert payload includes a `betterstack_source` annotation. | ## Security * Use a **dedicated ClickHouse HTTP client credential** for OpenSRE — not your personal dashboard login. * Keep the credential pair out of source control — use `.env` or the persistent store (`~/.opensre/integrations.json`). * The integration is **read-only**: OpenSRE only issues `SELECT` statements against `remote(...)` and `s3Cluster(...)` table functions. * Source identifiers are allowlisted against `^[A-Za-z0-9_]+$` before being interpolated into SQL, preventing identifier-injection attacks from the alert payload. * Rotate credentials periodically via the Better Stack dashboard. # Coralogix Source: https://opensre.com/docs/integrations/monitoring/coralogix Connect Coralogix so OpenSRE can query logs ## Overview OpenSRE queries Coralogix with DataPrime to retrieve logs — correlating log patterns with incidents and identifying root causes. ## Prerequisites * Coralogix account with DataPrime query access * Logs Query API key ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup coralogix ``` You will be prompted for the DataPrime (Logs Query) API key, API URL, and optional application / subsystem filters. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} CORALOGIX_API_KEY=your-logs-api-key CORALOGIX_API_URL=https://api.coralogix.com # optional CORALOGIX_APPLICATION_NAME=my-app # optional filter CORALOGIX_SUBSYSTEM_NAME=my-service # optional filter ``` | Variable | Default | Description | | - | - | - | | `CORALOGIX_API_KEY` | — | **Required.** Coralogix Logs Query API key | | `CORALOGIX_API_URL` | `https://api.coralogix.com` | Endpoint for your Coralogix cluster | | `CORALOGIX_APPLICATION_NAME` | — | Filter queries to this application | | `CORALOGIX_SUBSYSTEM_NAME` | — | Filter queries to this subsystem | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "coralogix-prod", "service": "coralogix", "status": "active", "credentials": { "api_key": "your-logs-api-key", "base_url": "https://api.coralogix.com", "application_name": "my-app", "subsystem_name": "my-service" } } ] } ``` Store credentials use `base_url` (not `CORALOGIX_API_URL`). ### Option 4: Multi-instance For multiple Coralogix accounts or regions, use [multi-instance](/docs/platform/multi-instance-integrations) with `CORALOGIX_INSTANCES`: ```bash theme={null} CORALOGIX_INSTANCES='[ {"name":"prod","tags":{"env":"prod"},"credentials":{"api_key":"...","base_url":"https://api.coralogix.com","application_name":"my-app"}}, {"name":"eu","tags":{"env":"eu"},"credentials":{"api_key":"...","base_url":"https://api.eu1.coralogix.com"}} ]' ``` When `CORALOGIX_INSTANCES` is set, the single-instance `CORALOGIX_*` variables are ignored. ## Credentials 1. In Coralogix, go to **Data Flow** → **API Keys** 2. Click **+ New API Key** 3. Select **Logs Query** as the key type 4. Copy the key ## Tools | Tool | Use it for | | - | - | | `query_coralogix_logs` | DataPrime log search for error signatures and incident context | Arguments the planner commonly supplies: | Argument | Default | Description | | - | - | - | | `query` | — | **Required.** DataPrime query text | | `time_range_minutes` | `60` | Lookback window | | `limit` | `50` | Row cap | | `application_name` | — | Optional application filter | | `subsystem_name` | — | Optional subsystem filter | | `trace_id` | — | Optional trace correlation | ## Verify ```bash theme={null} opensre integrations verify coralogix ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL coralogix local env passed Connected to https://api.coralogix.com (application my-app); DataPrime returned 0 row(s) ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Check the API key and ensure it is a Logs Query key | | **Connection timeout** | Verify `CORALOGIX_API_URL` matches your cluster region | | **No results** | Confirm `CORALOGIX_APPLICATION_NAME` and `CORALOGIX_SUBSYSTEM_NAME` match your data | | **Multiple Coralogix accounts** | Use `CORALOGIX_INSTANCES` — see [Multi-instance integrations](/docs/platform/multi-instance-integrations) | ## Security * Use a **read-only** Logs Query key — avoid admin or send-data keys. * Store the API key in `.env` or your secret manager — not in source control. ## Extras ### Cluster endpoints | Region | API URL | | - | - | | US1 | `https://api.coralogix.com` | | EU1 | `https://api.eu1.coralogix.com` | | EU2 | `https://api.eu2.coralogix.com` | | AP1 | `https://api.ap1.coralogix.com` | | AP2 | `https://api.ap2.coralogix.com` | # Datadog Source: https://opensre.com/docs/integrations/monitoring/datadog Connect Datadog so OpenSRE can query logs, metrics, monitors, and events ## Overview OpenSRE queries Datadog when you ask about an incident — logs, metrics, monitor state, and events correlated with it. ## Prerequisites * Datadog account with API access * API key (organization settings) * Application key with read scopes (`events_read`, `logs_read_data`, `logs_read_index_data`, `monitors_read`) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup datadog ``` You will be prompted for API key, application key, and site (for example `datadoghq.com` or `datadoghq.eu`). ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} DD_API_KEY=your-api-key DD_APP_KEY=your-application-key DD_SITE=datadoghq.com # optional — default datadoghq.com; use datadoghq.eu for EU ``` | Variable | Default | Description | | - | - | - | | `DD_API_KEY` | — | **Required.** Datadog API key | | `DD_APP_KEY` | — | **Required.** Datadog application key | | `DD_SITE` | `datadoghq.com` | Datadog site hostname (EU: `datadoghq.eu`) | For multiple Datadog orgs or sites, use [multi-instance](/docs/platform/multi-instance-integrations) with `DD_INSTANCES` (not `DATADOG_INSTANCES`). ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "datadog-prod", "service": "datadog", "status": "active", "credentials": { "api_key": "your-api-key", "app_key": "your-application-key", "site": "datadoghq.com" } } ] } ``` ### Option 4: Hosted web app (OpenSRE Cloud) > [!NOTE] > > Hosted OpenSRE Cloud is **coming soon**. Until then, use the local CLI, environment variables, or persistent store above. When available, organization-wide Datadog connectors will be configured in the hosted web app: 1. In [app.opensre.com](https://app.opensre.com), go to **Integrations** → **Datadog** 2. Enter a name, and paste your API and application keys ![Connect Datadog](https://mintcdn.com/tracer/Iv727munhErPWZ_V/images/connect_datadog.png?fit=max\&auto=format\&n=Iv727munhErPWZ_V\&q=85\&s=918a59eea13d56569e99a4247636542d) ## Credentials ### API key 1. In Datadog, go to **Organization Settings** → **API Keys** ([direct link](https://app.datadoghq.com/organization-settings/api-keys)) 2. Click **+ New Key** and name it (for example `opensre`) 3. Copy the key ![Datadog API Key](https://mintcdn.com/tracer/YVmZyi6JA4r86I72/images/datadog_api_key.webp?fit=max\&auto=format\&n=YVmZyi6JA4r86I72\&q=85\&s=986a5465483e9e46729e9c12d769b576) ### Application key 1. Go to **User Settings** → **Application Keys** ([direct link](https://app.datadoghq.com/personal-settings/application-keys)) 2. Click **+ New Key** and name it (for example `opensre`) 3. Under **Edit Scope**, enable at minimum: `events_read`, `logs_read_data`, `logs_read_index_data`, `monitors_read` 4. Under **Actions API Access**, click **Enable** 5. Copy the key ![Datadog App Key](https://mintcdn.com/tracer/YVmZyi6JA4r86I72/images/datadog_app_key.webp?fit=max\&auto=format\&n=YVmZyi6JA4r86I72\&q=85\&s=4076ff6036ae39c80549faa0dce564aa) ![Datadog App Key Scope](https://mintcdn.com/tracer/YVmZyi6JA4r86I72/images/datadog_app_key_scope.webp?fit=max\&auto=format\&n=YVmZyi6JA4r86I72\&q=85\&s=1cfee9f64e5b2e23dcdee159bdd55a31) > [!NOTE] > > Some Datadog UI labels may still show an older product name. Any key name works — use `opensre` for clarity. ## Tools | Tool | Use it for | | - | - | | `query_datadog_logs` | Search logs with a concrete query (for example `service:checkout status:error`) | | `query_datadog_metrics` | Infrastructure and application metrics | | `query_datadog_monitors` | Monitor configuration and current states | | `query_datadog_events` | Deployments, alerts, and system change events | | `query_datadog_all` | Logs, monitors, and events in parallel for a fast first pass | ## Verify ```bash theme={null} opensre integrations verify datadog ``` Expected output: ``` Service: datadog Status: passed Detail: Connected to api.{site} and listed {N} monitors. ``` Example for the default US site with 12 monitors: ``` Detail: Connected to api.datadoghq.com and listed 12 monitors. ``` Inside the REPL: `/integrations verify datadog` or `/verify datadog`. ## Troubleshooting | Symptom | Fix | | - | - | | **403 Forbidden** | Confirm application key scopes include `logs_read_data` and `monitors_read` | | **Wrong site / EU account** | Set `DD_SITE=datadoghq.eu` (or your site hostname) | | **Verify passes but no logs returned** | Check index retention and that the app key can read the target log index | | **Multiple Datadog accounts** | Use `DD_INSTANCES` — see [Multi-instance integrations](/docs/platform/multi-instance-integrations) | ## Security * Create a dedicated application key with read-only scopes. * Store keys in `.env` or your secret manager — not in source control. # Grafana Source: https://opensre.com/docs/integrations/monitoring/grafana Connect Grafana so OpenSRE can query metrics, dashboards, and alerts ## Overview OpenSRE queries Grafana (Cloud or self-hosted) for logs, metrics, traces, alert rules, and annotations. For a local Minikube lab with Prometheus and a sample app, see [Extras](#extras). ## Prerequisites * Grafana instance URL (Cloud stack URL or self-hosted origin) * Service account token with read access — see [Credentials](#credentials) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup grafana ``` Provide the instance URL and service account token when prompted. ### Option 2: Environment variables ```bash theme={null} GRAFANA_INSTANCE_URL=https://your-stack.grafana.net GRAFANA_READ_TOKEN=glsa_your_service_account_token GRAFANA_VERIFY_SSL=true # optional — set false only for local/lab GRAFANA_CA_BUNDLE=/path/to/internal-ca.pem # optional — internal CA ``` | Variable | Default | Description | | - | - | - | | `GRAFANA_INSTANCE_URL` | — | **Required.** Grafana base URL | | `GRAFANA_READ_TOKEN` | — | **Required.** Service account token | | `GRAFANA_VERIFY_SSL` | `true` | Set `false` to skip TLS verification (lab only) | | `GRAFANA_CA_BUNDLE` | — | PEM file for private CA (recommended for internal Grafana) | For prod/staging pairs, use `GRAFANA_INSTANCES` — see [Multi-instance integrations](/docs/platform/multi-instance-integrations). ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "grafana-prod", "service": "grafana", "status": "active", "credentials": { "endpoint": "https://your-stack.grafana.net", "api_key": "glsa_your_token", "verify_ssl": true } } ] } ``` ### Option 4: Hosted web app (OpenSRE Cloud) > [!NOTE] > > Hosted OpenSRE Cloud is **coming soon**. Until then, use the local CLI, environment variables, or persistent store above. When available, organization-wide Grafana connectors will be configured in the hosted web app: 1. In [app.opensre.com](https://app.opensre.com), go to **Integrations** → **Grafana** 2. Enter a name, instance URL, and service account token 3. Click **Save** ![Connect Grafana](https://mintcdn.com/tracer/Iv727munhErPWZ_V/images/connect_grafana.png?fit=max\&auto=format\&n=Iv727munhErPWZ_V\&q=85\&s=d15a9bc1a2851a008cd13c6b7577ed44) ### Self-signed or internal CA certificates If Grafana uses a certificate signed by an internal CA, `opensre integrations setup grafana` prompts for: | Prompt | When to use it | | - | - | | **Verify SSL certificate?** | Answer `No` only for local/lab instances (skips TLS verification) | | **Path to CA bundle for SSL verification** | Point at a PEM file with your internal CA — keeps full TLS checks; preferred for real internal Grafana | You can also set these in `.env`: | Variable | Purpose | | - | - | | `GRAFANA_VERIFY_SSL` | `true` / `false` | | `GRAFANA_CA_BUNDLE` | Path to a PEM CA bundle | ## Credentials Create a Grafana service account token with read access. See [Grafana service account tokens](https://grafana.com/docs/grafana/latest/administration/service-accounts/#add-a-token-to-a-service-account-in-grafana). 1. In Grafana, open **Administration** → **Service accounts** (or your stack’s equivalent). 2. Create a service account with read access to the datasources you want OpenSRE to query. 3. Add a token to that service account and copy it (shown once). Use this value as `GRAFANA_READ_TOKEN` (CLI prompt: service account token). ## Tools | Tool | Use it for | | - | - | | `query_grafana_logs` | Loki log streams for a known `service_name` | | `query_grafana_metrics` | Mimir / Prometheus-style metrics | | `query_grafana_traces` | Tempo traces through the Grafana datasource proxy | | `query_grafana_alert_rules` | Alert rule configuration and thresholds | | `query_grafana_service_names` | Discover Loki `service_name` labels before querying logs | Deployment and config-change markers are covered on [Grafana Annotations](/docs/integrations/monitoring/grafana_annotations). For standalone Tempo (no Grafana proxy), see [Grafana Tempo](/docs/integrations/monitoring/tempo). ## Verify ```bash theme={null} opensre integrations verify grafana ``` ## Troubleshooting | Symptom | Fix | | - | - | | **TLS / certificate errors** | For lab only, set `GRAFANA_VERIFY_SSL=false`. For internal Grafana, set `GRAFANA_CA_BUNDLE` to a PEM with your CA (preferred). | | **Auth failures** | Confirm the service account token is valid and has read access to the target datasources. | | **Multiple Grafana stacks** | Use `GRAFANA_INSTANCES` — see [Multi-instance integrations](/docs/platform/multi-instance-integrations). | ## Security * Prefer a dedicated service account token with read-only access. * Prefer `GRAFANA_CA_BUNDLE` over disabling TLS verification for real internal Grafana. * Set `GRAFANA_VERIFY_SSL=false` only for local/lab instances. * Store tokens in `.env` or your secret manager — not in source control. ## Extras ### Local Grafana setup (Minikube example) Use this lab to run Grafana, Prometheus, and a sample app locally, then connect OpenSRE. #### Steps 1. Start Minikube: ```bash theme={null} minikube start ``` 2. Add Helm repositories and update: ```bash theme={null} helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add podinfo https://stefanprodan.github.io/podinfo helm repo update ``` 3. Install the kube-prometheus stack: ```bash theme={null} kubectl create namespace monitoring helm install kube-stack prometheus-community/kube-prometheus-stack -n monitoring ``` 4. Install the podinfo sample app: ```bash theme={null} kubectl create namespace podinfo helm install podinfo podinfo/podinfo -n podinfo --set serviceMonitor.enabled=true ``` 5. (Optional) Check pods: ```bash theme={null} kubectl -n podinfo get pods kubectl -n monitoring get pods ``` 6. Port-forward podinfo (separate terminal): ```bash theme={null} kubectl -n podinfo port-forward deploy/podinfo 8080:9898 ``` 7. Port-forward Prometheus (separate terminal): ```bash theme={null} kubectl -n monitoring port-forward svc/kube-stack-kube-prometheus-prometheus 9090:9090 ``` 8. Port-forward Grafana on all interfaces (separate terminal): ```bash theme={null} kubectl -n monitoring port-forward --address 0.0.0.0 svc/kube-stack-grafana 3000:80 ``` 9. Allow Prometheus to scrape podinfo ServiceMonitors: ```bash theme={null} kubectl patch prometheus kube-stack-kube-prometheus-prometheus \ -n monitoring \ --type=json \ -p '[{"op":"replace","path":"/spec/serviceMonitorSelector","value":{}}]' kubectl patch prometheus kube-stack-kube-prometheus-prometheus \ -n monitoring \ --type=json \ -p '[{"op":"replace","path":"/spec/serviceMonitorNamespaceSelector","value":{}}]' ``` > [!WARNING] > > A JSON **merge** patch (`--type=merge`) with `{}` for a nested object field is a > no-op, not a replacement -- RFC 7396 merge-patch semantics only remove keys set to > `null`; an empty object leaves the existing `matchLabels: {release: kube-stack}` > in place, so podinfo's ServiceMonitor (which carries no such label) never gets > scraped and this step silently does nothing. Confirmed live: `kubectl patch --type=merge` here reports `"patched (no change)"`, and `podinfo` never appears > in `http://localhost:9090/api/v1/targets` until switched to `--type=json` with an > explicit `replace` operation. #### Grafana credentials Get the admin password: ```bash theme={null} kubectl get secret kube-stack-grafana --namespace monitoring -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` Username is `admin`; password is the command output. #### Access | Service | URL | | - | - | | podinfo | [http://localhost:8080](http://localhost:8080) | | Prometheus | [http://localhost:9090](http://localhost:9090) | | Grafana | [http://localhost:3000](http://localhost:3000) | #### Simulate load ```bash theme={null} while true; do curl -s http://localhost:8080/status/500 >/dev/null; sleep 1; done & while true; do curl -s http://localhost:8080/delay/5 >/dev/null; sleep 1; done & ``` #### Sample Grafana queries Request rate by status: ```promql theme={null} sum(rate(http_requests_total{job="podinfo"}[1m])) by (status) ``` p95 latency: ```promql theme={null} histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="podinfo"}[1m])) by (le)) ``` ![Spike in Error Rate in Grafana](https://mintcdn.com/tracer/giAZ40u1tvodLGB8/images/grafana-prometheus-visualisation.png?fit=max\&auto=format\&n=giAZ40u1tvodLGB8\&q=85\&s=072e82b59886d1a58d9dba7578a9385c) #### Prometheus alert for high error rate ```bash theme={null} kubectl apply -f - <<'YAML' apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: podinfo-alerts namespace: monitoring labels: release: kube-stack spec: groups: - name: podinfo.rules rules: - alert: PodinfoHighErrorRate expr: increase(http_requests_total{status="500"}[5m]) > 5 for: 30s labels: severity: critical annotations: summary: "Podinfo error rate is high" YAML ``` The alert fires after about 30 seconds of elevated errors. Check [http://localhost:9090/alerts](http://localhost:9090/alerts). ![Prometheus Alert Firing](https://mintcdn.com/tracer/giAZ40u1tvodLGB8/images/prometheus-alert-firing.png?fit=max\&auto=format\&n=giAZ40u1tvodLGB8\&q=85\&s=7549fc028a04f17d8abe7a1091b82a00) #### Connect OpenSRE to the lab Grafana 1. Get your machine's LAN IP: ```bash theme={null} hostname -I | awk '{print $1}' # Linux ipconfig getifaddr en0 # macOS (Get-NetIPAddress -InterfaceAlias "Wi-Fi" -AddressFamily IPv4).IPAddress # Windows PowerShell # Replace "Wi-Fi" with your interface name if needed. ``` 2. Create a Grafana service account token ([Grafana docs](https://grafana.com/docs/grafana/latest/administration/service-accounts/#add-a-token-to-a-service-account-in-grafana)). 3. Run setup and enter: ```bash theme={null} opensre integrations setup grafana ``` | Prompt | Value | | - | - | | Instance URL | `http://:3000` | | Service account token | Token from step 2 | For local/lab TLS issues, set `GRAFANA_VERIFY_SSL=false` or answer the SSL prompts as described in [Setup](#setup). ![Successful Grafana Integration with OpenSRE](https://mintcdn.com/tracer/giAZ40u1tvodLGB8/images/successful-grafana-opensre-integration.png?fit=max\&auto=format\&n=giAZ40u1tvodLGB8\&q=85\&s=dfd7fd3ad331ac7d7555486b50c53408) ### Add Loki and Tempo for full tool coverage The steps above give OpenSRE working metrics querying and (once a rule/annotation exists) alert-rule and annotation querying, but log search, service-name discovery, and trace querying need Loki and Tempo datasources, which this lab doesn't install by default. Verified live: without them, those three capabilities return `"Loki datasource not found"` / `"Tempo datasource not found"`, not real data. 1. Add the Grafana chart repo and install Loki (bundled with Promtail) and Tempo: ```bash theme={null} helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install loki grafana/loki-stack -n monitoring --set promtail.enabled=true helm install tempo grafana/tempo -n monitoring kubectl -n monitoring wait --for=condition=Ready pods -l "app=loki" --timeout=180s kubectl -n monitoring wait --for=condition=Ready pods -l "app.kubernetes.io/name=tempo" --timeout=180s ``` 2. Register both as Grafana datasources (the chart doesn't auto-provision them): ```bash theme={null} GRAFANA_PW=$(kubectl get secret kube-stack-grafana --namespace monitoring \ -o jsonpath="{.data.admin-password}" | base64 --decode) curl -s -X POST -H "Content-Type: application/json" \ -d '{"name":"Loki","type":"loki","access":"proxy","url":"http://loki.monitoring:3100"}' \ "http://admin:${GRAFANA_PW}@localhost:3000/api/datasources" curl -s -X POST -H "Content-Type: application/json" \ -d '{"name":"Tempo","type":"tempo","access":"proxy","url":"http://tempo.monitoring:3200"}' \ "http://admin:${GRAFANA_PW}@localhost:3000/api/datasources" ``` > [!WARNING] > > Promtail's default relabeling produces `job`/`namespace`/`pod`/`app` labels on log > streams, and the ServiceMonitor stack's default metric labels are `job`/`service` -- > neither includes a `service_name` label. Log search and service-name discovery both > query Loki's `service_name` label specifically, and metrics querying filters on the same > label when a service name is passed, so all three return empty against an unmodified > stack even though the underlying log/metric data is present. Confirmed live: a Loki label query and a > `service_name="podinfo"` Mimir query both returned nothing until the two relabelings > below were added; a real user's own services will hit the same gap unless they already > emit or relabel a `service_name` label. 3. Add a `service_name` relabel to Promtail (derived from the pod's `app.kubernetes.io/name` label) by upgrading the release with an extra relabel config alongside Promtail's defaults: ```bash theme={null} cat > /tmp/promtail-values.yaml << 'EOF' promtail: config: snippets: common: - action: replace source_labels: [__meta_kubernetes_pod_node_name] target_label: node_name - action: replace source_labels: [__meta_kubernetes_namespace] target_label: namespace - action: replace replacement: $1 separator: / source_labels: [namespace, app] target_label: job - action: replace source_labels: [__meta_kubernetes_pod_name] target_label: pod - action: replace source_labels: [__meta_kubernetes_pod_container_name] target_label: container - action: replace source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] target_label: service_name - action: replace replacement: /var/log/pods/*$1/*.log separator: / source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name] target_label: __path__ EOF helm upgrade loki grafana/loki-stack -n monitoring -f /tmp/promtail-values.yaml \ --set promtail.enabled=true --reuse-values ``` > [!WARNING] > > Keep the heredoc terminator (`EOF`) flush against the left margin, not indented to > match this list item -- `<<'EOF'` requires an exact, unindented match to end the > here-document. An indented closing line is swallowed as YAML content instead of > ending the file, and the `helm upgrade` line after it gets swallowed too, silently > turning into dead text inside the values file instead of running. Confirmed live > by copy-pasting an indented version of this exact block. 4. Enable request tracing on podinfo -- it ships an OpenTelemetry exporter but keeps it disabled until both `--otel-service-name` and an OTLP endpoint are set (the chart's `extraEnvs` alone is not enough; confirmed via `podinfo --help`, which states tracing is disabled unless `--otel-service-name` is set): ```bash theme={null} cat > /tmp/podinfo-tracing-values.yaml << 'EOF' serviceMonitor: enabled: true extraArgs: - "--otel-service-name=podinfo" extraEnvs: - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: "http://tempo.monitoring:4317" EOF helm upgrade podinfo podinfo/podinfo -n podinfo -f /tmp/podinfo-tracing-values.yaml kubectl -n podinfo wait --for=condition=Ready pods --all --timeout=60s ``` The upgrade replaces the podinfo pod, so re-run the port-forward from [Steps](#steps) (step 6) if it's still attached to the old pod. 5. Add the same `service_name` relabel to the podinfo ServiceMonitor for metrics (`kubectl patch` here, since the podinfo chart doesn't expose `metricRelabelings` as a values key): ```bash theme={null} kubectl patch servicemonitor podinfo -n podinfo --type=json -p '[{ "op": "add", "path": "/spec/endpoints/0/metricRelabelings", "value": [{"sourceLabels": ["job"], "targetLabel": "service_name"}] }]' ``` > [!WARNING] > > Do this **after** the tracing upgrade above, not before. `helm upgrade` re-applies > the ServiceMonitor via server-side apply; if `kubectl patch` already owns > `.spec.endpoints` from a prior patch, a later `helm upgrade podinfo` fails with > `Apply failed with 1 conflict: conflict with "kubectl-patch"`. Confirmed live by > running these two steps in the opposite order. 6. Generate some traffic so there's real data for every tool, then confirm each datasource actually has it: ```bash theme={null} for i in $(seq 1 10); do curl -s http://localhost:8080/ >/dev/null; done for i in $(seq 1 5); do curl -s http://localhost:8080/status/500 >/dev/null; done ``` ### Ask the agent ```bash theme={null} opensre integrations verify grafana ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL grafana │ local env │ ✓ passed │ Connected to http://localhost:3000 and │ │ │ discovered loki, prometheus, tempo datasources. ``` Then start `opensre` and ask about the failing app from [Steps](#steps), e.g. *Why is podinfo's error rate high? Check Grafana logs, metrics, and traces.* All 6 registered tools return real data against this lab once the datasources and relabelings above are in place -- confirmed by direct calls to each tool: metrics querying (real `http_requests_total` series, `service_name="podinfo"`), log search (real podinfo log lines), trace querying (real spans like `GET /status/{code:[0-9]+}` from the generated load), service-name discovery (`["podinfo", ...]`), alert-rule querying (the rule created via `/api/v1/provisioning/alert-rules` below), and annotation querying (an annotation created via `/api/annotations`). > [!NOTE] > > Alert-rule and annotation querying use Grafana's own > alerting/annotation APIs, not the raw `PrometheusRule` CRD from > [Prometheus alert for high error rate](#prometheus-alert-for-high-error-rate) -- that > CRD is evaluated by Prometheus itself and never becomes a Grafana-managed rule. > To exercise these two tools with real data, create a rule and annotation through > Grafana directly, for example: > > ```bash theme={null} > GRAFANA_PW=$(kubectl get secret kube-stack-grafana --namespace monitoring \ > -o jsonpath="{.data.admin-password}" | base64 --decode) > > FOLDER_UID=$(curl -s -X POST -H "Content-Type: application/json" \ > -d '{"title":"podinfo-alerts"}' \ > "http://admin:${GRAFANA_PW}@localhost:3000/api/folders" | python3 -c 'import json,sys; print(json.load(sys.stdin)["uid"])') > > curl -s -X POST -H "Content-Type: application/json" \ > -d "{ > \"title\": \"PodinfoHighErrorRateGrafana\", \"ruleGroup\": \"podinfo-group\", > \"folderUID\": \"${FOLDER_UID}\", \"condition\": \"B\", \"noDataState\": \"NoData\", > \"execErrState\": \"Error\", \"for\": \"30s\", > \"data\": [ > {\"refId\": \"A\", \"relativeTimeRange\": {\"from\": 300, \"to\": 0}, \"datasourceUid\": \"prometheus\", > \"model\": {\"expr\": \"increase(http_requests_total{status=\\\"500\\\"}[5m])\", \"refId\": \"A\", \"instant\": true}}, > {\"refId\": \"B\", \"relativeTimeRange\": {\"from\": 0, \"to\": 0}, \"datasourceUid\": \"__expr__\", > \"model\": {\"type\": \"threshold\", \"expression\": \"A\", \"conditions\": [{\"evaluator\": {\"type\": \"gt\", \"params\": [5]}}], \"refId\": \"B\"}} > ] > }" \ > "http://admin:${GRAFANA_PW}@localhost:3000/api/v1/provisioning/alert-rules" > > curl -s -X POST -H "Content-Type: application/json" \ > -d "{\"text\":\"podinfo v6.14.1 deployed with tracing enabled\",\"tags\":[\"deployment\",\"podinfo\"],\"time\":$(($(date +%s) * 1000))}" \ > "http://admin:${GRAFANA_PW}@localhost:3000/api/annotations" > ``` ### Teardown ```bash theme={null} kind delete cluster --name grafana-lab # or: minikube delete ``` If you started separate port-forward terminals per [Steps](#steps), `Ctrl-C` each one first (deleting the cluster also kills them, but not always cleanly on every platform). # Grafana Annotations Source: https://opensre.com/docs/integrations/monitoring/grafana_annotations Query Grafana annotations to correlate deploys and config changes with incidents ## Overview Correlate incidents with **deployments and config changes** from any source. OpenSRE reads [Grafana annotations](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/annotate-visualizations/) — the standard “what changed and when” marker — so the agent can answer whether a deploy or config change preceded an alert, even when the change did **not** come from a GitHub push (Argo CD/Flux syncs, `helm upgrade`, Jenkins/CircleCI jobs, Terraform applies, manual hotfixes). This complements the GitHub deploy timeline, which only sees GitHub-originated deploys. ## Prerequisites No new credentials. The tool reuses your existing [Grafana](/docs/integrations/monitoring/grafana) integration. Once Grafana is connected, `query_grafana_annotations` is available in chat. ## Setup No separate OpenSRE setup. Configure the [Grafana](/docs/integrations/monitoring/grafana) integration (CLI, environment variables, or persistent store). Annotation query uses that connection. ## Credentials None beyond your Grafana service account token. See [Grafana → Credentials](/docs/integrations/monitoring/grafana#credentials). ## Tools ### `query_grafana_annotations` | Parameter | Description | | - | - | | `from` | ISO 8601 window start (for example `2026-05-30T14:00:00Z`). Overrides the default. | | `to` | ISO 8601 window end. Overrides the default. | | `tags` | Optional list of annotation tags to filter by (for example `["deployment"]`). | | `time_range_minutes` | Window size when `from`/`to` are omitted (default `60`, ending now). | | `limit` | Maximum annotations to return (default `100`). | Example: ```text theme={null} query_grafana_annotations(from="2026-05-30T14:00:00Z", to="2026-05-30T15:00:00Z", tags=["deployment"]) ``` ```json theme={null} { "source": "grafana_annotations", "total": 1, "annotations": [ { "time": "2026-05-30T14:41:09Z", "time_end": null, "text": "deploy checkout-api v2.8.1", "tags": ["deployment", "checkout-api"], "dashboard_uid": null } ] } ``` Each annotation may also include: | Field | When it is set | | - | - | | `time_end` | Region annotations only; otherwise `null` | | `dashboard_uid` | Annotations attached to a dashboard; otherwise `null` | The agent uses results like this to flag a likely change-induced regression and tie the suspected root cause to a specific deploy. ## Verify No separate verify command. Use `opensre integrations verify grafana` to confirm the underlying Grafana connection. ## Troubleshooting | Symptom | Fix | | - | - | | Tool unavailable | Confirm the [Grafana](/docs/integrations/monitoring/grafana) integration is configured and verified. | | Empty annotation results | Widen the time window, relax `tags`, or confirm annotations exist in Grafana for that period. | ## Security Annotation query is read-only through your existing Grafana credentials. Prefer a read-scoped service account token as described in [Grafana → Security](/docs/integrations/monitoring/grafana#security). ## Extras ### Emitting annotations Make deploys visible by writing a Grafana annotation when you ship. Most CD tools can post to Grafana’s annotations API, for example: ```bash theme={null} curl -s -X POST "$GRAFANA_URL/api/annotations" \ -H "Authorization: Bearer $GRAFANA_TOKEN" \ -H "Content-Type: application/json" \ -d '{"time": 1717079669000, "text": "deploy checkout-api v2.8.1", "tags": ["deployment", "checkout-api"]}' ``` Tag deploy annotations consistently (for example `deployment`) so the agent can filter precisely. # groundcover Source: https://opensre.com/docs/integrations/monitoring/groundcover Connect groundcover so OpenSRE can query logs and traces with gcQL ## Overview Investigate incidents with [groundcover](https://groundcover.com) logs and traces, queried with **gcQL** (groundcover Query Language) over groundcover’s public, read-only **MCP** endpoint. OpenSRE supports configuration, verification, and three read-only tools (logs, traces, and the gcQL reference). More signals (metrics, APM, Kubernetes events/entities, monitors, monitor issues) and alert-source routing land in follow-up releases. The integration is read-only — no monitor creation, silencing, or other mutating actions. ## Prerequisites * A groundcover account with access to create a service-account API key * A **read-only** service-account token (see [Credentials](#credentials)) * For multi-workspace / multi-backend accounts: tenant UUID and/or backend ID (verification tells you which values to set) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup groundcover ``` ### Option 2: Environment variables ```bash theme={null} export GROUNDCOVER_API_KEY="" # alias: GROUNDCOVER_MCP_TOKEN # Optional — defaults shown: export GROUNDCOVER_MCP_URL="https://mcp.groundcover.com/api/mcp" export GROUNDCOVER_TIMEZONE="UTC" # Only for multi-workspace / multi-backend accounts: export GROUNDCOVER_TENANT_UUID="" export GROUNDCOVER_BACKEND_ID="" ``` | Variable | Alias / notes | Description | | - | - | - | | `GROUNDCOVER_API_KEY` | Alias: `GROUNDCOVER_MCP_TOKEN` | **Required.** Service-account token (setup writes `GROUNDCOVER_API_KEY`) | | `GROUNDCOVER_MCP_URL` | Default `https://mcp.groundcover.com/api/mcp` | MCP endpoint | | `GROUNDCOVER_TIMEZONE` | Default `UTC` | Timezone for query windows | | `GROUNDCOVER_TENANT_UUID` | Multi-workspace only | Workspace / tenant to route to | | `GROUNDCOVER_BACKEND_ID` | Multi-backend only | Backend to select when several exist | | `GROUNDCOVER_INSTANCES` | Multi-instance JSON | Prod/staging (or other) instances — see below | Single-workspace accounts need only the token. If your account has multiple workspaces or backends, verification tells you which value to set. Multiple instances: ```bash theme={null} export GROUNDCOVER_INSTANCES='[ {"name":"prod","api_key":"...","tenant_uuid":"...","backend_id":"prod"}, {"name":"staging","api_key":"...","tenant_uuid":"...","backend_id":"staging"} ]' ``` `opensre integrations list` shows integrations saved to the local store (via `setup`). Environment-variable configuration is still picked up by `verify` and at runtime. ### Commands | Command | What it does | | - | - | | `opensre integrations setup groundcover` | Store a groundcover service-account token and routing settings | | `opensre integrations verify groundcover` | Connect to the MCP endpoint, list tools, and check workspace routing | ## Credentials In groundcover: **Settings → Access → Service Accounts** → create a service account with a **read-only** policy → create an **API key** and copy it (shown once). OpenSRE never performs write actions. ## Tools | Tool | Use it for | | - | - | | `get_groundcover_query_reference` | gcQL syntax reference — call once before writing queries | | `query_groundcover_logs` | Application errors, exceptions, and log events | | `query_groundcover_traces` | Slow/failing spans and request correlations | ## Verify ```bash theme={null} opensre integrations verify groundcover ``` Verification connects to the MCP endpoint, confirms the expected read-only tool surface is present, and lists your workspaces. It returns `passed`, `missing` (token not configured), or `failed` with an actionable message — for example, naming the missing or mistyped `GROUNDCOVER_TENANT_UUID` / `GROUNDCOVER_BACKEND_ID` when the account is ambiguous. Tokens are never printed. ## Troubleshooting | Symptom | Fix | | | | - | - | - | - | | `missing` on verify | Set `GROUNDCOVER_API_KEY` (or `GROUNDCOVER_MCP_TOKEN`). | | | | `401 Unauthorized` | Regenerate the service-account token; ensure it has read access. | | | | `Account has N workspaces…` | Set `GROUNDCOVER_TENANT_UUID` to the listed tenant. | | | | `…has N backends…` | Set `GROUNDCOVER_BACKEND_ID` to the listed backend. | | | | `failed to query …` | Project with \` | fields …`or aggregate with` | stats …\`; narrow the time window; avoid a bare raw select-all. | | Empty results | Run \`\* | field\_names\` to find real field names; avoid leading-wildcard globs. | | | Not shown in `integrations list` | `list` shows the local store — run `opensre integrations setup groundcover` (env vars still work for `verify`/runtime). | | | ## Security * Use a **read-only** service-account token. OpenSRE never performs write actions. * Store the token in `.env` or the local integrations store — not in source control. * Verification never prints tokens. ## Extras ### Writing efficient gcQL gcQL is pipe-based: a query starts with a filter (or `*`) and pipes through operators. These rules keep queries fast and valid: * **Lead with the filter** — `level:error | …`, not `* | filter level:error`. The `| filter` pipe is for post-aggregation conditions on computed aliases. * **Project or aggregate** — use `| fields a, b, …` or `| stats …`. A bare select-all (` | limit N`) can be rejected by the backend. * **Keep the time window narrow.** Default is the last 1 hour; widen only after an empty or inconclusive result. * **Always include `| limit N`** — it caps rows returned (not data scanned). For wide ranges, prefer `stats` / aggregations. * **Discover fields** with `* | field_names` (or `get_groundcover_query_reference`). Examples: ```text theme={null} # Recent error logs for a workload (projected) level:error workload:checkout | fields _time, instance, content | limit 50 # Error count per workload in one query level:error | stats by (env, cluster, namespace, workload) count() as errors | sort by (errors desc) | limit 20 # Slowest spans for a service (projected) duration_seconds>0.5 workload:checkout | fields _time, span_name, duration_seconds, status_code | limit 50 # 5xx rate per workload (HTTP spans use status_code; status:error is universal) status_code>=500 | stats by (workload) count() as errors | sort by (errors desc) | limit 20 ``` # Honeycomb Source: https://opensre.com/docs/integrations/monitoring/honeycomb Connect Honeycomb so OpenSRE can query traces ## Overview OpenSRE queries Honeycomb for distributed traces — identifying slow spans, high-error services, and query patterns correlated with incidents. ## Prerequisites * Honeycomb account (classic or environments-based) * API key with query access ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup honeycomb ``` You will be prompted for the API key, dataset (defaults to `__all__`), and API URL. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} HONEYCOMB_API_KEY=your-api-key HONEYCOMB_DATASET=your-dataset # optional, defaults to __all__ HONEYCOMB_API_URL=https://api.honeycomb.io # optional ``` | Variable | Default | Description | | - | - | - | | `HONEYCOMB_API_KEY` | — | **Required.** Honeycomb API key | | `HONEYCOMB_DATASET` | `__all__` | Dataset to query (classic) or `__all__` for environments | | `HONEYCOMB_API_URL` | `https://api.honeycomb.io` | Override for EU region or proxies | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "honeycomb-prod", "service": "honeycomb", "status": "active", "credentials": { "api_key": "your-api-key", "dataset": "__all__", "base_url": "https://api.honeycomb.io" } } ] } ``` Store credentials use `base_url` (not `HONEYCOMB_API_URL`). ### Option 4: Multi-instance For multiple Honeycomb environments or keys, use [multi-instance](/docs/platform/multi-instance-integrations) with `HONEYCOMB_INSTANCES`: ```bash theme={null} HONEYCOMB_INSTANCES='[ {"name":"prod","tags":{"env":"prod"},"credentials":{"api_key":"...","dataset":"__all__","base_url":"https://api.honeycomb.io"}}, {"name":"eu","tags":{"env":"eu"},"credentials":{"api_key":"...","dataset":"__all__","base_url":"https://api.eu1.honeycomb.io"}} ]' ``` When `HONEYCOMB_INSTANCES` is set, the single-instance `HONEYCOMB_*` variables are ignored. ## Credentials 1. In Honeycomb, go to **Account** → **API Keys** 2. Click **Create API Key** 3. Set the environment and enable **Query Data** permission 4. Copy the key > [!TIP] > > Use `__all__` as the dataset to query across all datasets in an environment. For EU accounts, set `HONEYCOMB_API_URL=https://api.eu1.honeycomb.io`. ## Tools | Tool | Use it for | | - | - | | `query_honeycomb_traces` | Trace and span groups related to an incident | Arguments the planner commonly supplies: | Argument | Default | Description | | - | - | - | | `dataset` | from config (`__all__`) | **Required** at tool call time | | `service_name` | — | Filter spans for a service | | `trace_id` | — | Look up a specific trace | | `time_range_seconds` | `3600` | Lookback window | | `limit` | `20` | Result cap | ## Verify ```bash theme={null} opensre integrations verify honeycomb ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL honeycomb local env passed Connected to https://api.honeycomb.io (environment production) and queried dataset __all__ ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Confirm the API key is correct and has Query Data permission | | **Dataset not found** | Use `__all__` or check the exact dataset name in Honeycomb | | **EU account unreachable** | Set `HONEYCOMB_API_URL=https://api.eu1.honeycomb.io` | | **Multiple Honeycomb accounts** | Use `HONEYCOMB_INSTANCES` — see [Multi-instance integrations](/docs/platform/multi-instance-integrations) | ## Security * Use a dedicated **read-only** API key scoped to query access. * Store the API key in `.env` or your secret manager — not in source control. # New Relic Source: https://opensre.com/docs/integrations/monitoring/new_relic Connect New Relic so OpenSRE can query alerts and NRQL metrics during incidents OpenSRE queries New Relic when investigating alerts — pulling open and recent alert incidents (with the condition and policy that fired) and running NRQL metric queries correlated with incidents. ## Prerequisites * New Relic account * A **User key** (starts with `NRAK-`) — **not** a License key. This is the most common setup mistake: License keys authenticate agents that *send* data to New Relic, not the NerdGraph API that OpenSRE reads from. A License key will fail verification. * Your account ID ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup new_relic ``` You will be prompted for the API key, account ID, and API URL. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} NEW_RELIC_API_KEY=NRAK-your-user-key NEW_RELIC_ACCOUNT_ID=your-account-id NEW_RELIC_API_URL=https://api.newrelic.com # optional — default US endpoint; use https://api.eu.newrelic.com (EU) or https://api.jp.newrelic.com (JP) otherwise ``` | Variable | Default | Description | | - | - | - | | `NEW_RELIC_API_KEY` | — | **Required.** New Relic User key (`NRAK-...`) | | `NEW_RELIC_ACCOUNT_ID` | — | **Required.** New Relic account ID | | `NEW_RELIC_API_URL` | `https://api.newrelic.com` | Override for EU (`https://api.eu.newrelic.com`) or JP (`https://api.jp.newrelic.com`) accounts | Both `NEW_RELIC_API_KEY` and `NEW_RELIC_ACCOUNT_ID` must be set — either one alone leaves the integration unconfigured. For multiple New Relic accounts, use [multi-instance](/docs/platform/multi-instance-integrations) with `NEW_RELIC_INSTANCES`. ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "new-relic-prod", "service": "new_relic", "status": "active", "credentials": { "api_key": "NRAK-your-user-key", "account_id": "your-account-id", "base_url": "https://api.newrelic.com" } } ] } ``` ## Creating a User key 1. In New Relic, go to your user menu → **API keys** 2. Click **Create a key** 3. Set the key type to **User key** (not **License key**) 4. Copy the key — it starts with `NRAK-` ## Finding your account ID Your account ID is shown in New Relic under **Administration** → **Account settings**, and also appears in the URL when you're viewing an account-scoped page (`.../accounts//...`). ## Verify ```bash theme={null} opensre integrations verify new_relic ``` Expected output: ``` Service: new_relic Status: passed Detail: Connected to https://api.newrelic.com and validated account access ``` Inside the REPL: `/integrations verify new_relic` or `/verify new_relic`. ## Alerts require an existing alert condition New Relic alert data becomes available only after at least one configured alert condition has fired. If your account has no alert policies and conditions yet, OpenSRE will correctly return zero incidents — that's not a misconfiguration, there's simply nothing to read yet. Set up an alert condition in New Relic first if you want alert investigations to return data. ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized / invalid key** | Confirm you're using a User key (`NRAK-...`), not a License key | | **Valid key but account unreachable** | Double check `NEW_RELIC_ACCOUNT_ID` — the key is valid but doesn't have access to that account | | **429 Too Many Requests** | You've hit NerdGraph's rate limit; retry after a short wait | | **Alerts tool returns no incidents** | No alert condition has fired yet on this account — see above | | **Metrics query fails or times out** | NRQL queries via the API have a 5-second timeout; narrow the time window or simplify the query | | **EU account unreachable** | Set `NEW_RELIC_API_URL=https://api.eu.newrelic.com` | | **JP account unreachable** | Set `NEW_RELIC_API_URL=https://api.jp.newrelic.com` | | **Multiple New Relic accounts** | Use `NEW_RELIC_INSTANCES` — see [Multi-instance integrations](/docs/platform/multi-instance-integrations) | ## Security best practices * This integration is read-only: OpenSRE only ever runs NerdGraph `query` operations against your account, never a `mutation`. * Store the User key in `.env` or your secret manager — not in source control. # OpenObserve Source: https://opensre.com/docs/integrations/monitoring/openobserve Connect OpenObserve so OpenSRE can pull structured logs as evidence ## Overview OpenSRE connects to OpenObserve to retrieve log evidence when an alert fires — correlating errors, anomalies, and service activity during incidents. ## Prerequisites * An OpenObserve instance (self-hosted or cloud-hosted) * The URL of your OpenObserve deployment * Either an access token **or** a username and password * Access to the organization you want OpenSRE to query ## Setup There is no `opensre integrations setup openobserve` handler. Configure OpenObserve with environment variables or the persistent store only. ### Option 1: Environment variables Add to your `.env`: ```bash theme={null} OPENOBSERVE_URL=https://openobserve.example.com OPENOBSERVE_TOKEN=your_access_token OPENOBSERVE_ORG=default OPENOBSERVE_STREAM=logs OPENOBSERVE_MAX_RESULTS=100 ``` | Variable | Default | Description | | - | - | - | | `OPENOBSERVE_URL` | — | **Required.** URL of your OpenObserve instance | | `OPENOBSERVE_TOKEN` | — | Authentication token (use **or** username/password) | | `OPENOBSERVE_USERNAME` | — | Username when not using a token | | `OPENOBSERVE_PASSWORD` | — | Password when not using a token | | `OPENOBSERVE_ORG` | `default` | Organization name | | `OPENOBSERVE_STREAM` | *(empty)* | Optional stream to query | | `OPENOBSERVE_MAX_RESULTS` | `100` | Maximum number of results returned | #### Alternative authentication If your deployment uses username/password instead of a token: ```bash theme={null} OPENOBSERVE_URL=https://openobserve.example.com OPENOBSERVE_USERNAME=your_username OPENOBSERVE_PASSWORD=your_password OPENOBSERVE_ORG=default ``` Use **either** `OPENOBSERVE_TOKEN` **or** `OPENOBSERVE_USERNAME` + `OPENOBSERVE_PASSWORD`. Verify requires the URL plus one of those auth pairs. ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "openobserve-prod", "service": "openobserve", "status": "active", "credentials": { "base_url": "https://openobserve.example.com", "api_token": "your_access_token", "org": "default", "stream": "logs", "max_results": 100 } } ] } ``` Store credentials use `base_url` and `api_token` (not `url` / `token`). For username/password auth, set `username` and `password` instead of `api_token`. ## Credentials 1. Log in to your OpenObserve instance 2. Open user or organization settings 3. Create or retrieve an access token 4. Copy your OpenObserve URL 5. Note the organization name you want OpenSRE to query 6. Add these values to your OpenSRE configuration ## Tools | Tool | Use it for | | - | - | | `query_openobserve_logs` | Bounded read-only log search (timestamp, service, level, message) | OpenSRE does not expose separate OpenObserve traces or metrics tools. ## Verify ```bash theme={null} opensre integrations verify openobserve ``` Verify is a config-shape check only — it does not call OpenObserve. Expected output: ``` SERVICE SOURCE STATUS DETAIL openobserve local env passed Configured for OpenObserve at https://openobserve.example.com ``` ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Regenerate the access token or confirm username/password credentials. | | **404 on query** | Check `OPENOBSERVE_ORG` matches your organization slug. Verify the stream name if `OPENOBSERVE_STREAM` is set. | | **Connection refused** | Confirm `OPENOBSERVE_URL` includes the correct protocol and port. Ensure network access from OpenSRE to the instance. | | **Empty log results** | Widen the time range in the query. Confirm logs are ingested into the target stream. | | **Missing credentials on verify** | Set `OPENOBSERVE_URL` and either `OPENOBSERVE_TOKEN` or both `OPENOBSERVE_USERNAME` and `OPENOBSERVE_PASSWORD`. | ## Security Use a token with the minimum permissions required for the agent's queries whenever possible. # PostHog Source: https://opensre.com/docs/integrations/monitoring/posthog Connect PostHog REST credentials so OpenSRE can verify access to your project ## Overview OpenSRE stores **your** PostHog project credentials for REST API access (project metadata and correlation). Configure with `opensre integrations setup posthog` or `POSTHOG_*` environment variables. This REST integration does **not** register agent tools. For agent tools (analytics, feature flags, HogQL), use [PostHog (MCP)](/docs/integrations/monitoring/posthog-mcp). | What you want | Use | Setup | | - | - | - | | Store PostHog REST credentials for your project | **PostHog** (this page) | `opensre integrations setup posthog` or env vars below | | Let the agent explore PostHog during incidents | [PostHog (MCP)](/docs/integrations/monitoring/posthog-mcp) | `opensre integrations setup posthog_mcp` | | OpenSRE’s own product usage telemetry | Built-in analytics | Controlled by `OPENSRE_NO_TELEMETRY` — not your PostHog project | ## Prerequisites * A PostHog account (US, EU, or self-hosted) * Your PostHog **project ID** * A **personal API key** with read access to that project — see [PostHog personal API keys](https://posthog.com/docs/api/personal-api-keys) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup posthog ``` In the interactive shell, use `/integrations setup posthog` or ask “Can you configure PostHog for me?” 1. Choose US Cloud, EU Cloud, or your self-hosted app address. Setup links directly to that account's personal API key settings and explains the required read permissions before asking for a key. 2. Choose your project by name. OpenSRE discovers the project ID for you. If it is not listed, you can retry, change the key or region, or enter the ID from project settings. 3. OpenSRE verifies project access, saves the connection, and shows a summary. Failed checks can be retried in place; Ctrl+C cancels. Use a **personal API key** (`phx_...`) with `project:read` for setup and `query:read` for analytics queries. A project capture key (`phc_...`) cannot read your analytics. ### Option 2: Environment variables ```bash theme={null} POSTHOG_PROJECT_ID=your_project_id POSTHOG_PERSONAL_API_KEY=phx_your_personal_api_key POSTHOG_BASE_URL=https://us.posthog.com # optional — US Cloud POSTHOG_TIMEOUT_SECONDS=15.0 # optional ``` ## Credentials | Variable | Default | Description | | - | - | - | | `POSTHOG_PROJECT_ID` | — | **Required.** Your PostHog project ID | | `POSTHOG_PERSONAL_API_KEY` | — | **Required.** Personal API key for the project | | `POSTHOG_BASE_URL` | `https://us.posthog.com` | App URL (`https://eu.posthog.com` for EU, or your self-hosted origin) | | `POSTHOG_TIMEOUT_SECONDS` | `15.0` | HTTP timeout in seconds | Get the **project ID** from PostHog project settings. Create a **personal API key** (`phx_...`) with read access — see [PostHog personal API keys](https://posthog.com/docs/api/personal-api-keys). ## Tools This REST integration does **not** register agent tools. For agent tools (analytics, feature flags, HogQL), use [PostHog (MCP)](/docs/integrations/monitoring/posthog-mcp). ## Verify ```bash theme={null} opensre integrations verify posthog ``` A successful check prints `PostHog validated.` For the MCP integration, use `opensre integrations verify posthog_mcp` instead. ## Troubleshooting | Symptom | Fix | | - | - | | **Authentication failed (401)** | Check `POSTHOG_PERSONAL_API_KEY` is valid and has read access | | **Project not found (404)** | Confirm `POSTHOG_PROJECT_ID` matches the instance in `POSTHOG_BASE_URL` | | **Verify says not configured** | Both `POSTHOG_PROJECT_ID` and `POSTHOG_PERSONAL_API_KEY` must be set | ## Security * Use a dedicated personal API key with the minimum read permissions needed * Store credentials in environment variables or a secret manager — not in source code * Rotate keys periodically # PostHog (MCP) Source: https://opensre.com/docs/integrations/monitoring/posthog-mcp Connect PostHog's hosted MCP server so OpenSRE can query analytics, feature flags, error tracking, and HogQL ## Overview OpenSRE connects to PostHog’s hosted [Model Context Protocol (MCP)](https://posthog.com/docs/model-context-protocol) server. The agent can call PostHog products — product analytics, feature flags, error tracking, experiments, surveys, and HogQL — through MCP tools. This is separate from the [PostHog REST integration](/docs/integrations/monitoring/posthog), which only stores project credentials for the REST API. Configure REST with `opensre integrations setup posthog`. ## Prerequisites * A PostHog account (US or EU — the hosted server routes you automatically) * A PostHog **personal API key** created with the **MCP Server** preset. See [PostHog personal API keys](https://posthog.com/docs/api/personal-api-keys). > [!NOTE] > > OpenSRE defaults to **read-only** access (`x-posthog-read-only: true`) so the agent cannot mutate your PostHog project. Set `POSTHOG_MCP_READ_ONLY=false` only if you explicitly want the agent to perform writes. ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup posthog_mcp opensre integrations verify posthog_mcp ``` Or run `opensre integrations setup` and select **PostHog (MCP)**. Paste your personal API key. Setup uses the hosted Streamable HTTP transport; keep the default URL unless you have a reason to change it. For a local server, set `POSTHOG_MCP_MODE=stdio` (see below). ### Option 2: Environment variables ```bash theme={null} POSTHOG_MCP_MODE=streamable-http POSTHOG_MCP_URL=https://mcp.posthog.com/mcp POSTHOG_MCP_AUTH_TOKEN=phx_your_personal_api_key POSTHOG_MCP_PROJECT_ID=12345 # optional, scope to one project POSTHOG_MCP_ORGANIZATION_ID= # optional, scope to one organization POSTHOG_MCP_FEATURES= # optional, comma-separated feature filter POSTHOG_MCP_READ_ONLY=true # optional, default true ``` | Variable | Default | Description | | - | - | - | | `POSTHOG_MCP_AUTH_TOKEN` | — | **Required** (hosted). Personal API key with the `MCP Server` preset | | `POSTHOG_MCP_URL` | `https://mcp.posthog.com/mcp` | MCP server URL (use `https://mcp-eu.posthog.com/mcp` to pin EU) | | `POSTHOG_MCP_MODE` | `streamable-http` | Transport: `streamable-http`, `sse`, or `stdio` | | `POSTHOG_MCP_PROJECT_ID` | — | Scope tools to a specific PostHog project | | `POSTHOG_MCP_ORGANIZATION_ID` | — | Scope tools to a specific organization | | `POSTHOG_MCP_FEATURES` | — | Comma-separated feature filter (e.g. `flags,error-tracking`) | | `POSTHOG_MCP_READ_ONLY` | `true` | Send the read-only header so the agent cannot mutate PostHog | | `POSTHOG_MCP_COMMAND` | — | Command to launch a local MCP server (`stdio` mode only) | | `POSTHOG_MCP_ARGS` | — | Arguments for the local MCP command (`stdio` mode only) | Local PostHog MCP server via `stdio`: ```bash theme={null} POSTHOG_MCP_MODE=stdio POSTHOG_MCP_COMMAND=npx POSTHOG_MCP_ARGS=-y @posthog/mcp-server@latest POSTHOG_MCP_AUTH_TOKEN=phx_your_personal_api_key ``` ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "posthog-mcp-prod", "service": "posthog_mcp", "status": "active", "credentials": { "url": "https://mcp.posthog.com/mcp", "mode": "streamable-http", "auth_token": "phx_your_personal_api_key", "project_id": "12345", "read_only": true } } ] } ``` ## Credentials Use a PostHog **personal API key** created with the **MCP Server** preset. See [PostHog personal API keys](https://posthog.com/docs/api/personal-api-keys). Set it as `POSTHOG_MCP_AUTH_TOKEN` (or paste it during interactive setup). Optionally scope with `POSTHOG_MCP_PROJECT_ID` and `POSTHOG_MCP_ORGANIZATION_ID`. ## Tools | Tool | What it does | | - | - | | `list_posthog_tools` | List tools the connected PostHog MCP server exposes (compact, filterable) | | `call_posthog_tool` | Call a named PostHog MCP tool (for example run a HogQL query, list feature flags, inspect an error) | Typical flow: call `list_posthog_tools` first, then `call_posthog_tool` with the chosen name and arguments. The hosted PostHog MCP server exposes a large vendor catalog of tools (often 240+), each with a full input schema. Returning all of them at once exceeds most model context windows, so `list_posthog_tools` returns a **compact, bounded listing** — names plus short descriptions, without schemas. | Tip | Detail | | - | - | | Narrow the list | Pass `name_filter` (space- or comma-separated terms, e.g. `"events query sql"`) | | Fetch schemas | Pass `include_schema=true` on a narrowed list for the tool you intend to call | | Query events | Use `call_posthog_tool` with `tool_name="execute-sql"` and a HogQL query (e.g. `SELECT event, count() FROM events WHERE ... GROUP BY event`). There is no `search_events` tool. | ## Verify ```bash theme={null} opensre integrations verify posthog_mcp ``` A successful check connects to the MCP server and reports how many tools it discovered. ## Troubleshooting | Symptom | Fix | | - | - | | Missing or invalid personal API key | Use a key created with the `MCP Server` preset | | Blocked outbound HTTPS | Confirm reachability to `mcp.posthog.com` (or `mcp-eu.posthog.com` if pinned to EU) | ## Security OpenSRE defaults to **read-only** access (`x-posthog-read-only: true`) so the agent cannot mutate your PostHog project. Set `POSTHOG_MCP_READ_ONLY=false` only if you explicitly want the agent to perform writes. A successful check connects to the MCP server and reports how many tools it discovered. If it fails, the most common cause is a missing or invalid personal API key — confirm the key was created with the `MCP Server` preset and that outbound HTTPS to `mcp.posthog.com` is allowed. ## Metric report (scheduled) Deliver a per-metric PostHog analytics pulse to Telegram, Slack, or Rocket.Chat, on demand or on a schedule — coworker-style digests (what moved and why it matters), not a raw dashboard dump. Slack delivery needs a bot token (a webhook alone cannot honor `--chat-id`). This uses the headless **summarizing-posthog-analytics** skill path (schema discovery, then one bounded HogQL query per metric), not generic `opensre cron` kinds. It requires the PostHog **MCP** integration above — the REST `posthog` integration alone cannot serve it. | Command | What it does | | - | - | | `opensre posthog report run` | Run once and print the per-metric report to stdout | | `opensre posthog report run --period 30d --metrics "active users,pageviews"` | Run once for a custom window and metric focus | | `opensre posthog report schedule add` | Schedule recurring delivery (cron + provider + chat) | | `opensre posthog report schedule list` | List PostHog report schedules | | `opensre posthog report schedule run TASK_ID` | Run a scheduled report immediately | | `opensre posthog report schedule remove TASK_ID` | Remove a schedule | Example — every Monday at 08:00 London time to Telegram: ```bash theme={null} opensre posthog report schedule add \ --cron "0 8 * * mon" \ --tz Europe/London \ --provider telegram \ --chat-id "-1001234567890" \ --period 7d ``` Example — same schedule to a Slack channel (`C…` member/channel id): ```bash theme={null} opensre posthog report schedule add \ --cron "0 8 * * mon" \ --tz Europe/London \ --provider slack \ --chat-id C0123ABCD \ --period 7d ``` Slack delivery needs `SLACK_BOT_TOKEN`. A `SLACK_WEBHOOK_URL` alone will not work here: a webhook always posts to the one channel it was created for, so it cannot honour `--chat-id`. `--period` accepts a relative window (`24h`, `7d`, `30d`; default `7d`). Optional `--metrics` narrows the report to a comma-separated set instead of the default metric set. The gateway daemon picks up scheduled reports automatically when it is running. If the LLM is unavailable, the run fails with an error (no deterministic fallback). > [!NOTE] > > The report compares the current window against the previous comparable window per metric. Real zeros are reported as zeros; failed queries are called out as failures — never silently widened or fabricated, and never presented as zero when the query did not run. # SigNoz Source: https://opensre.com/docs/integrations/monitoring/signoz Connect SigNoz so OpenSRE can query logs, metrics, and traces ## Overview Query logs, metrics, and traces from [SigNoz](https://signoz.io) via the Query Range API. OpenSRE uses `POST /api/v5/query_range` for `query_signoz_logs`, `query_signoz_metrics`, and `query_signoz_traces`. Configure **`SIGNOZ_URL`** and **`SIGNOZ_API_KEY`** (service account key). ## Prerequisites * A running SigNoz instance (Cloud, self-hosted, or [local Docker](#local-docker-quick-start)) * A SigNoz **service account API key** (see [Credentials](#credentials)) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup signoz ``` Setup prompts for SigNoz URL and API key only. ### Option 2: Environment variables ```bash theme={null} export SIGNOZ_URL="http://localhost:8080" export SIGNOZ_API_KEY="" ``` | Variable | Description | | - | - | | `SIGNOZ_URL` | SigNoz base URL (local Docker: `http://localhost:8080`) | | `SIGNOZ_API_KEY` | Service account API key | ### Local Docker quick start If you already run SigNoz, skip to credentials and env setup above. > [!WARNING] > > SigNoz has deprecated its `docker-compose.yaml` / `install.sh` install path entirely in > favor of a new installer called **Foundry** -- a checked-out `signoz` repo no longer has > a `signoz/docker/docker-compose.yaml` file at all. Confirmed live against the current > `SigNoz/signoz` repo: `deploy/install.sh` now just prints a deprecation notice and exits, > pointing to [signoz.io/docs/install/docker](https://signoz.io/docs/install/docker/). ```bash theme={null} curl -fsSL https://signoz.io/foundry.sh | bash ``` Piping a remote script into `bash` runs whatever that URL currently serves, with your shell's privileges. If that's a concern for your environment, download and verify a pinned release instead (`SigNoz/foundry` on GitHub publishes a checksums file with every release): ```bash theme={null} VERSION=v0.2.17 curl -fsSLO "https://github.com/SigNoz/foundry/releases/download/${VERSION}/foundry_darwin_arm64.tar.gz" curl -fsSLO "https://github.com/SigNoz/foundry/releases/download/${VERSION}/foundry_${VERSION#v}_checksums.txt" shasum -a 256 -c <(grep darwin_arm64 "foundry_${VERSION#v}_checksums.txt") && \ tar -xzf foundry_darwin_arm64.tar.gz && \ sudo mv foundryctl /usr/local/bin/ ``` Swap `darwin_arm64` for your platform (`linux_amd64`, `linux_arm64`, etc.) and use `sha256sum` in place of `shasum -a 256` on Linux. ```bash theme={null} cat > casting.yaml << 'EOF' apiVersion: v1alpha1 kind: Installation metadata: name: signoz spec: deployment: flavor: compose mode: docker EOF foundryctl cast -f casting.yaml ``` `foundryctl cast` validates the environment, generates the Docker Compose files under `pours/`, and starts every container in one step. Serves the UI and API on **[http://localhost:8080](http://localhost:8080)** (not 3301). ```bash theme={null} # ... later, teardown: docker compose -f pours/deployment/compose.yaml down ``` ## Credentials In SigNoz: **Settings → Service Accounts** → create a service account → **Keys** → **Add Key**. Copy the key (shown once). ## Tools | Tool | What it does | | - | - | | `query_signoz_logs` | Search logs by service, severity, and time window | | `query_signoz_metrics` | Query CPU, memory, and request-rate signals | | `query_signoz_traces` | Query error spans, latency percentiles, and dependencies | ### Supported metrics (V1) | Alias | Actual SigNoz metric | | - | - | | `cpu_usage` | `system_cpu_usage` | | `memory_usage` | `system_memory_usage` | | `request_rate` | `signoz_calls_total` | You can also pass any raw metric name known to SigNoz. For latency percentiles (p95/p99), prefer `query_signoz_traces`. ## Verify ```bash theme={null} opensre integrations verify signoz ``` Expected verify output mentions the SigNoz Query API (`/api/v2/metrics`, `/api/v5/query_range`). ## Troubleshooting | Symptom | Fix | | - | - | | `SigNoz configuration is incomplete` | Set both `SIGNOZ_URL` and `SIGNOZ_API_KEY`. | | `HTTP 401` on verify | Regenerate the service account key; check URL (local Docker: port **8080**). | | `No logs returned` | Confirm telemetry exists in SigNoz and filter fields match (`service.name` for logs). | | `No metrics returned` | Verify the metric name in SigNoz Metrics Explorer; empty results return a warning for unknown metrics. | ## Security * Use a dedicated SigNoz service-account key with the minimum permissions needed for query access. * Store `SIGNOZ_API_KEY` in `.env` or your secret manager — not in source control. ## Extras ### API reference * Endpoint: `POST {SIGNOZ_URL}/api/v5/query_range` * Auth header: `SigNoz-Api-Key: ` * Validation probe: `GET {SIGNOZ_URL}/api/v2/metrics` ### Local verification recipe Verified live against a real local SigNoz stack (via Foundry, above). Send some real telemetry first -- an empty stack proves connectivity, not that the tools return data: ```bash theme={null} python3 -m venv /tmp/signoz-otel-venv && source /tmp/signoz-otel-venv/bin/activate pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http python3 - << 'PY' import logging, time from opentelemetry import metrics, trace from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler from opentelemetry.sdk._logs.export import BatchLogRecordProcessor from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor resource = Resource.create({"service.name": "payment-service-verify"}) trace_provider = TracerProvider(resource=resource) trace_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))) trace.set_tracer_provider(trace_provider) tracer = trace.get_tracer(__name__) meter_provider = MeterProvider(resource=resource, metric_readers=[ PeriodicExportingMetricReader(OTLPMetricExporter(endpoint="http://localhost:4318/v1/metrics"), export_interval_millis=2000) ]) metrics.set_meter_provider(meter_provider) error_counter = metrics.get_meter(__name__).create_counter("payment_errors_total") log_provider = LoggerProvider(resource=resource) log_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint="http://localhost:4318/v1/logs"))) logger = logging.getLogger("payment-service-verify") logger.setLevel(logging.ERROR) logger.addHandler(LoggingHandler(level=logging.ERROR, logger_provider=log_provider)) with tracer.start_as_current_span("process_payment"): logger.error("Payment gateway timeout: connection refused to billing-api") error_counter.add(1, {"error.type": "gateway_timeout"}) time.sleep(4) trace_provider.shutdown(); meter_provider.shutdown(); log_provider.shutdown() PY ``` `opensre integrations verify` checks a saved store record before env vars, and chat sessions only fall through to env vars when the store has no records at all -- any existing record, for any service, blocks env-var resolution entirely. Point `OPENSRE_INTEGRATIONS_STORE_PATH` at a path inside a fresh empty directory before verifying, so a real saved record can't shadow the `SIGNOZ_*` vars above, and your real config is never read or written: ```bash theme={null} export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-signoz-demo.XXXXXX)" export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json" ``` Verify, then ask the agent: ```bash theme={null} opensre integrations verify signoz ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL signoz │ local env │ ✓ passed │ Connected to SigNoz Query API (/api/v2/metrics, │ │ │ /api/v5/query_range for logs/metrics/traces). ``` ```bash theme={null} opensre ``` Ask: *Why is payment-service-verify erroring? Check SigNoz logs, traces, and metrics.* The agent calls `query_signoz_logs`, `query_signoz_traces`, and `query_signoz_metrics` against this exact local stack and finds the seeded failure: a single ERROR log from `payment-service-verify` reporting `connection refused to billing-api`, with zero `billing-api` telemetry across the window — the service is not running. Teardown: ```bash theme={null} docker compose -f pours/deployment/compose.yaml down rm -rf "$OPENSRE_DEMO_STORE_DIR" /tmp/signoz-otel-venv unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH SIGNOZ_URL SIGNOZ_API_KEY ``` # Splunk Source: https://opensre.com/docs/integrations/monitoring/splunk Connect Splunk so OpenSRE can search logs using SPL ## Overview OpenSRE queries Splunk over the REST API. When you ask about logs, the agent writes an SPL `query` and calls `query_splunk_logs`. There is **no** `opensre integrations setup splunk` handler today. Configure Splunk via environment variables or the integration store, then verify with `opensre integrations verify splunk`. ## Prerequisites * Splunk Enterprise or Splunk Cloud (version 8.x or later) * REST API access on port 8089 * A bearer token with search capability (see [Credentials](#credentials)) ## Setup ### Option 1: Environment variables ```bash theme={null} SPLUNK_URL=https://splunk.corp.com:8089 # REST API base URL (port 8089 default) SPLUNK_TOKEN=your-bearer-token # API bearer token (NOT an HEC token) SPLUNK_INDEX=main # Default index to search (optional) SPLUNK_VERIFY_SSL=true # Set false to skip SSL verification (optional) SPLUNK_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem # Path to custom CA bundle (optional) ``` | Variable | Default | Description | | - | - | - | | `SPLUNK_URL` | — | **Required.** REST API base URL including port | | `SPLUNK_TOKEN` | — | **Required.** Bearer token with search capability | | `SPLUNK_INDEX` | `main` | Default index when the tool does not override `index` | | `SPLUNK_VERIFY_SSL` | `true` | Set to `false` to disable SSL verification (dev/local only) | | `SPLUNK_CA_BUNDLE` | — | Path to a PEM CA bundle for enterprise self-signed certificates | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "splunk-prod", "service": "splunk", "status": "active", "credentials": { "base_url": "https://splunk.corp.com:8089", "token": "your-bearer-token", "index": "main", "verify_ssl": true, "ca_bundle": "/etc/ssl/certs/corp-ca.pem" } } ] } ``` ### Option 4: Multi-instance ```bash theme={null} SPLUNK_INSTANCES='[ {"name":"prod","tags":{"env":"prod"},"credentials":{"base_url":"https://splunk-prod:8089","token":"prod-token","index":"prod"}}, {"name":"staging","tags":{"env":"staging"},"credentials":{"base_url":"https://splunk-staging:8089","token":"staging-token","index":"staging"}} ]' ``` When `SPLUNK_INSTANCES` is set it overrides the single-instance `SPLUNK_URL` / `SPLUNK_TOKEN` variables. ## Credentials OpenSRE uses **bearer tokens** — not basic auth and not HEC tokens. **Via the Splunk UI:** 1. Go to **Settings** → **Tokens** 2. Click **New Token** 3. In **User**, pick the existing Splunk user the token authenticates as 4. Fill in **Audience** (required) and an expiry date 5. Copy the generated token **Via the REST API** (replace `` with your admin password and `` with the service account the token should authenticate as): ```bash theme={null} curl -sk -u admin: \ https://splunk.corp.com:8089/services/authorization/tokens \ -X POST \ --data-urlencode "name=" \ --data-urlencode "audience=OpenSRE" \ --data-urlencode "expires_on=+90d" \ --data-urlencode "output_mode=json" \ | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['entry'][0]['content']['token'])" ``` > [!WARNING] > > `name` is the **existing Splunk user** the token authenticates as, not an arbitrary > label — Splunk rejects an unrecognized name with `User "..." does not exist.` The token > inherits that user's roles, so point it at a least-privilege service account rather than > `admin`. `audience` is also required (a short free-text description of the token's > purpose); omitting it fails with `The following required arguments are missing: audience.` The token needs the `search` capability. The `admin` role includes this by default. For a dedicated service account, ensure the role includes: * `search` * `read_splunkd_private_settings` (needed for the verify call against `/services/server/info`) ### Quick local test with Docker ```bash theme={null} docker run -d --name splunk-dev --platform linux/amd64 \ -p 127.0.0.1:8000:8000 \ -p 127.0.0.1:8089:8089 \ -e SPLUNK_START_ARGS=--accept-license \ -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \ -e SPLUNK_PASSWORD=VerifyPass123! \ splunk/splunk:latest i=0 until docker logs splunk-dev 2>&1 | grep -q "Ansible playbook complete"; do i=$((i + 1)) if [ "$i" -ge 180 ]; then echo "Splunk never became ready; check: docker logs splunk-dev" >&2 break fi sleep 5 done ``` > [!WARNING] > > `splunk/splunk` publishes **amd64 only**, which is why `--platform linux/amd64` is > already on the command above. On Apple Silicon it runs under emulation and can take > 15 minutes or more to report ready; on native amd64 the flag is a harmless no-op. > Without `SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com` alongside > `SPLUNK_START_ARGS=--accept-license`, the container exits immediately with > `License not accepted` — newer images require both, not just the license flag > documented in older Splunk Docker guides. Seed a test event and create a token: ```bash theme={null} curl -sk -u admin:VerifyPass123! \ "https://localhost:8089/services/receivers/simple?index=main&sourcetype=opensre_verify" \ --data-binary "$(date -u '+%Y-%m-%d %H:%M:%S') ERROR payment-service: connection refused to billing-api during checkout" SPLUNK_TOKEN=$(curl -sk -u admin:VerifyPass123! \ https://localhost:8089/services/authorization/tokens \ -X POST \ --data-urlencode "name=admin" \ --data-urlencode "audience=OpenSRE verification" \ --data-urlencode "expires_on=+90d" \ --data-urlencode "output_mode=json" \ | python3 -c "import sys,json; print(json.load(sys.stdin)['entry'][0]['content']['token'])") export SPLUNK_URL="https://localhost:8089" export SPLUNK_TOKEN export SPLUNK_VERIFY_SSL=false export SPLUNK_INDEX=main unset SPLUNK_INSTANCES SPLUNK_CA_BUNDLE ``` Then point OpenSRE at an empty store, so the container is the only Splunk either command can reach: ```bash theme={null} export OPENSRE_DEMO_STORE_DIR="$(mktemp -d /tmp/opensre-splunk-demo.XXXXXX)" export OPENSRE_INTEGRATIONS_STORE_PATH="$OPENSRE_DEMO_STORE_DIR/integrations.json" ``` > [!WARNING] > > Do this **before** verifying, and keep it for the rest of the recipe. A saved Splunk > record wins over `SPLUNK_URL` / `SPLUNK_TOKEN` for `opensre integrations verify`, so > with your real store in play the check can pass against production while appearing to > test localhost — the `SOURCE` column reads `local store` instead of `local env`. > Chat sessions fail the other way: they only fall through to env vars when the > store has **no records at all**, so a single record for any unrelated service blocks > env resolution entirely. An empty store is the one state both surfaces agree on. > `SPLUNK_INSTANCES` overrides `SPLUNK_URL` / `SPLUNK_TOKEN` outright, which is why the > block above clears it. Verify: ```bash theme={null} opensre integrations verify splunk ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL splunk │ local env │ ✓ passed │ Connected to Splunk 10.4.2 ``` Now ask the agent about the seeded event: ```bash theme={null} opensre ``` Ask: *Search Splunk for payment-service errors in the last hour.* The agent writes the SPL, calls `query_splunk_logs`, and surfaces the seeded `connection refused to billing-api` event from this exact local instance. Teardown: ```bash theme={null} docker rm -f splunk-dev rm -rf "$OPENSRE_DEMO_STORE_DIR" unset OPENSRE_DEMO_STORE_DIR OPENSRE_INTEGRATIONS_STORE_PATH \ SPLUNK_URL SPLUNK_TOKEN SPLUNK_VERIFY_SSL SPLUNK_INDEX ``` ## Tools ### `query_splunk_logs` | Argument | Required | Description | | - | - | - | | `query` | **Yes** | SPL search string (for example `index=main error \| head 50`). Do not include a leading `search` keyword. | | `time_range_minutes` | No | Look-back window in minutes (default `60`) | | `limit` | No | Max events to return (default `50`) | | `index` | No | Overrides the integration default index | The agent writes the SPL `query` and calls the tool. OpenSRE does **not** build SPL from a fixed priority table. ## Verify ```bash theme={null} opensre integrations verify splunk ``` Expected output: ``` Service: splunk Status: passed Detail: Connected to Splunk 9.x.x ``` ## Troubleshooting | Symptom | Fix | | - | - | | `SSL: CERTIFICATE_VERIFY_FAILED` | Set `SPLUNK_CA_BUNDLE=/path/to/corp-ca.pem` (preferred) or `SPLUNK_VERIFY_SSL=false` (dev only) | | `HTTP 401 Unauthorized` | Token expired or was generated with the wrong account — regenerate | | `HTTP 403 Forbidden` | Token lacks `search` capability — check the role assigned to the token | | Empty search results | Data may not have been ingested yet, or the index name is wrong | | `Connection refused` on port 8089 | Splunk management port may be firewalled; confirm network access | | `opensre integrations verify` fails | Check `SPLUNK_URL` includes the protocol and port (`https://host:8089`) | | `opensre integrations setup splunk` | Not available — use env vars or the store, then `verify` | ## Security * Use a **read-only bearer token** — never use an admin token in production. * Store `SPLUNK_TOKEN` in `.env` or the credential store, not in source code or CI logs. * Prefer a dedicated `opensre` service account with only the `search` capability (plus `read_splunkd_private_settings` for verify). * For enterprise self-signed certificates, set `SPLUNK_CA_BUNDLE` rather than disabling verification entirely. * Set `SPLUNK_VERIFY_SSL=false` only in local or dev environments when you cannot supply a CA bundle. * Rotate tokens on a schedule and revoke them when no longer needed. # Grafana Tempo Source: https://opensre.com/docs/integrations/monitoring/tempo Connect Grafana Tempo so OpenSRE can query distributed traces during incidents ## Overview OpenSRE uses Grafana Tempo to investigate trace-related alerts — searching spans by service, fetching full traces by ID, listing instrumented services, and filtering by error status or latency. This integration talks to Tempo **directly** via its HTTP API. It does not require a Grafana instance or datasource proxy. If you run the full Grafana stack, the [Grafana](/docs/integrations/monitoring/grafana) integration already surfaces Tempo through the datasource proxy — use this integration when you run Tempo standalone. ## Prerequisites * Grafana Tempo 1.4+ * Network access from the OpenSRE environment to your Tempo instance * Auth credentials only if your deployment requires them (many run without auth behind a gateway) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup tempo ``` You will be prompted for the Tempo URL and optional auth. Leave auth fields blank if Tempo runs without authentication. ### Option 2: Environment variables Add to your `.env`: ```bash theme={null} TEMPO_URL=http://localhost:3200 TEMPO_API_KEY= # optional TEMPO_USERNAME= # optional (basic auth) TEMPO_PASSWORD= # optional (basic auth) TEMPO_ORG_ID= # optional (X-Scope-OrgID for multi-tenant) ``` | Variable | Default | Description | | - | - | - | | `TEMPO_URL` | — | **Required.** Tempo HTTP API base URL | | `TEMPO_API_KEY` | *(empty)* | Bearer token for auth | | `TEMPO_USERNAME` | *(empty)* | Username for basic auth | | `TEMPO_PASSWORD` | *(empty)* | Password for basic auth | | `TEMPO_ORG_ID` | *(empty)* | Tenant ID sent as `X-Scope-OrgID` for multi-tenant deployments | ### Option 3: Persistent store Integrations are persisted to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "tempo-prod", "service": "tempo", "status": "active", "credentials": { "url": "http://localhost:3200", "api_key": "" } } ] } ``` ## Credentials Auth is optional. Many Tempo deployments run without authentication behind a gateway. When auth is required, supply one of: * Bearer token → `TEMPO_API_KEY` * Basic auth → `TEMPO_USERNAME` / `TEMPO_PASSWORD` * Multi-tenant → `TEMPO_ORG_ID` as `X-Scope-OrgID` Create tokens or accounts in your Tempo / gateway identity provider according to your deployment. Leave auth fields blank in setup if Tempo runs without authentication. ## Tools OpenSRE exposes a single `query_tempo` tool with an `action` parameter: | Action | Use it for | | - | - | | `search` | Search traces by service, span name, duration, and tags (TraceQL); one summary row per trace | | `get_trace` | Fetch a full trace by ID and flatten its spans | | `list_services` | List services registered in Tempo | | `list_span_names` | List span names registered in Tempo | ### search ```text theme={null} query_tempo(action="search", service="checkout-service", min_duration_ms=500) query_tempo(action="search", tags={"http.status_code": "500"}) ``` | Parameter | Description | | - | - | | `service` | Filter by `resource.service.name` | | `span_name` | Filter by span name | | `min_duration_ms` | Minimum trace duration | | `max_duration_ms` | Maximum trace duration | | `tags` | Key/value span attributes | | `time_range_minutes` | Lookback window (default 60) | | `limit` | Max traces to return (default 20) | ### get\_trace ```text theme={null} query_tempo(action="get_trace", trace_id="4f8c...e21") ``` ### list\_services ```text theme={null} query_tempo(action="list_services") ``` ### list\_span\_names ```text theme={null} query_tempo(action="list_span_names") ``` ## Verify ```bash theme={null} opensre integrations verify tempo ``` Expected output: ``` Service: tempo Status: passed Detail: Connected to Grafana Tempo HTTP API (/api/search/tags). ``` ### Local Docker verification Start a disposable all-in-one Tempo using the repository's local configuration: ```bash theme={null} docker run --rm --detach --name opensre-tempo \ --publish 127.0.0.1:3200:3200 \ --publish 127.0.0.1:4318:4318 \ --volume "$PWD/tests/e2e/tempo/tempo-local.yaml:/etc/tempo.yaml:ro" \ grafana/tempo:2.7.2 -config.file=/etc/tempo.yaml until curl --fail --silent http://127.0.0.1:3200/ready; do sleep 1; done ``` Seed a recent two-second checkout error trace through OTLP/HTTP: ```bash theme={null} uv run python -m tests.e2e.tempo.local_seed ``` From a source checkout, run `uv run opensre integrations setup tempo`. Enter `http://127.0.0.1:3200` as the URL and leave the bearer token, username, password, and tenant prompts blank. Then verify the connection: ```bash theme={null} uv run opensre integrations verify tempo ``` The verifier should report a successful connection to the search API. Exercise the search, tag-listing, and trace-fetch actions through one agent turn: ```bash theme={null} uv run opensre ask --allowed-tool query_tempo \ "Use query_tempo to list services, list span names, search the last 60 minutes for checkout-service, and fetch the full trace returned by the search. Report the trace ID, root span, duration, HTTP status, and span status." ``` The result should report `checkout-service`, `POST /checkout`, a two-second duration, HTTP status `500`, and an error span. Stop the disposable instance when finished: ```bash theme={null} docker stop opensre-tempo ``` This instance has no authentication or TLS. Both ports bind only to loopback, and all trace data is discarded with the container. ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused** | Check the URL and port. Default Tempo HTTP port is `3200`. | | **HTTP 401** | Set `TEMPO_API_KEY` or `TEMPO_USERNAME`/`TEMPO_PASSWORD` if your deployment requires auth. | | **HTTP 404 on verify** | Check Tempo version — `/api/search/tags` requires Tempo 1.4+. | | **No traces returned** | Confirm services send traces to Tempo and the time range covers the period of interest. | | **Multi-tenant 400** | Set `TEMPO_ORG_ID` to the correct tenant ID. | ## Security * Use a **read-only** token or service account if Tempo supports auth. * Store credentials in `.env`, never in code. * Restrict network access to Tempo — OpenSRE only needs the HTTP API port (`3200` by default). # VictoriaLogs Source: https://opensre.com/docs/integrations/monitoring/victoria_logs Connect VictoriaLogs so OpenSRE can pull structured logs as evidence ## Overview OpenSRE queries [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) with LogsQL to retrieve structured log evidence — correlating application errors, request anomalies, and stream-level events with the alert under investigation. ## Prerequisites * A reachable VictoriaLogs instance (for example `http://vmlogs:9428`) * LogsQL knowledge for any custom queries you want OpenSRE to run ## Setup There is no `opensre integrations setup victoria_logs` handler. Configure VictoriaLogs with environment variables or the persistent store only. ### Option 1: Environment variables Add to your `.env`: ```bash theme={null} VICTORIA_LOGS_URL=http://vmlogs.monitoring.svc:9428 # Optional — only set this if your deployment is multi-tenant. # Single-tenant clusters should leave it unset; OpenSRE will not send # the AccountID header at all. # VICTORIA_LOGS_TENANT_ID=acme ``` | Variable | Default | Description | | - | - | - | | `VICTORIA_LOGS_URL` | — | **Required.** Base URL of your VictoriaLogs instance | | `VICTORIA_LOGS_TENANT_ID` | *(unset)* | Optional. Sent as the `AccountID` header on multi-tenant deployments | Leave `VICTORIA_LOGS_TENANT_ID` unset for single-tenant clusters. OpenSRE does not send `AccountID: 0` implicitly — that would target the default tenant on every request. ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "victoria-logs-prod", "service": "victoria_logs", "status": "active", "credentials": { "base_url": "http://vmlogs.monitoring.svc:9428" } } ] } ``` ## Credentials VictoriaLogs typically needs no API token for OpenSRE. Point `VICTORIA_LOGS_URL` at a reachable instance. On multi-tenant deployments, set `VICTORIA_LOGS_TENANT_ID` so OpenSRE sends the correct `AccountID` header. ## Tools When VictoriaLogs is configured, the `victoria_logs_query` tool becomes available to the agent. It runs LogsQL queries against `/select/logsql/query` and returns structured rows. The default is a wildcard match over the past hour; the agent narrows the query from alert context (service, level, trace ID, time window). ## Verify ```bash theme={null} opensre integrations verify victoria_logs ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL victoria_logs local env passed Connected to VictoriaLogs at http://vmlogs.monitoring.svc:9428. ``` ## Troubleshooting | Symptom | Fix | | - | - | | **Status: missing** | Set `VICTORIA_LOGS_URL` or add a `victoria_logs` entry to your integrations store | | **Connection refused** | Verify the URL is reachable from this host; check firewall rules | | **404 on `/select/logsql/query`** | Confirm you are pointing at VictoriaLogs (not VictoriaMetrics for time-series); they expose different endpoints | | **Empty rows** | Normal when no logs match — widen the time range with a longer `start` (for example `-24h`) or relax the LogsQL filter | ## Security Leave `VICTORIA_LOGS_TENANT_ID` unset for single-tenant clusters. OpenSRE does not send `AccountID: 0` implicitly — that would target the default tenant on every request. ## Extras ### How it works Typical agent uses: * Pull recent error-level logs for the affected service to surface stack traces * Filter by request ID or trace ID to follow a single failing transaction across services * Compare log volume in the incident window against a known-good baseline to spot regressions # Airflow Source: https://opensre.com/docs/integrations/workflows/airflow Investigate DAG failures and extract execution context from Apache Airflow. ## Overview Ask OpenSRE about a failing DAG and it queries your Airflow REST API for the failing DAG run, task instances, and logs — then correlates that evidence with metrics and logs from your other integrations. It supports: * DAG run inspection * Task instance retrieval * Failure detection ## Prerequisites * A reachable Airflow REST API (Airflow 2.x `/api/v1`) * Network access from the OpenSRE environment * Auth: **token** (`AIRFLOW_AUTH_TOKEN`) **or** basic auth (`AIRFLOW_USERNAME` / `AIRFLOW_PASSWORD`) ## Setup There is no dedicated `opensre integrations setup airflow` wizard today. Configure via environment variables or the persistent store. ### Option 1: Environment variables ```bash theme={null} AIRFLOW_BASE_URL=http://localhost:8080 # Authentication (choose one) # Basic Auth AIRFLOW_USERNAME=your_username AIRFLOW_PASSWORD=your_password # Token-based (if supported) AIRFLOW_AUTH_TOKEN=your_token # Optional AIRFLOW_TIMEOUT_SECONDS=15 AIRFLOW_VERIFY_SSL=true AIRFLOW_MAX_RESULTS=50 AIRFLOW_DAG_ID=test_fail_dag # optional default DAG for tools ``` | Variable | Default | Description | | - | - | - | | `AIRFLOW_BASE_URL` | `http://localhost:8080/api/v1` | Airflow API base URL | | `AIRFLOW_USERNAME` | — | Basic auth username (with password) | | `AIRFLOW_PASSWORD` | — | Basic auth password | | `AIRFLOW_AUTH_TOKEN` | — | Bearer / token auth (alternative to basic) | | `AIRFLOW_TIMEOUT_SECONDS` | `15` | Request timeout | | `AIRFLOW_VERIFY_SSL` | `true` | Verify TLS certificates | | `AIRFLOW_MAX_RESULTS` | `50` | Result cap | | `AIRFLOW_DAG_ID` | — | Optional default DAG id for tools | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "airflow-prod", "service": "airflow", "status": "active", "credentials": { "base_url": "http://localhost:8080", "username": "your_username", "password": "your_password" } } ] } ``` ### Local smoke setup Start Airflow locally: ```bash theme={null} docker run -p 8080:8080 apache/airflow:2.8.1 standalone ``` Create a failing DAG: ```python theme={null} from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime def fail_task(): raise Exception("Intentional failure") with DAG( dag_id="test_fail_dag", start_date=datetime(2024, 1, 1), schedule=None, catchup=False, ) as dag: PythonOperator( task_id="fail_task", python_callable=fail_task, ) ``` Trigger the DAG: ```bash theme={null} airflow dags trigger test_fail_dag ``` Then ask the agent about the failure: ```bash theme={null} uv run opensre ``` Ask: *Why did the test\_fail\_dag DAG fail?* ## Credentials Provide either `AIRFLOW_AUTH_TOKEN` or `AIRFLOW_USERNAME` + `AIRFLOW_PASSWORD`. Prefer a dedicated read-only Airflow user. ## Tools | Tool | What it does | | - | - | | `get_recent_airflow_failures` | Recent failed / retrying task evidence for a DAG | | `get_airflow_dag_runs` | DAG run execution history | | `get_airflow_task_instances` | Task-level status for a run | These tools use the Airflow REST API via `integrations/airflow`. Tool selection is LLM-driven; there is no hard-coded Airflow bypass. Related (Tracer, not Airflow API): `get_airflow_metrics` pulls orchestration metrics from Tracer when a `trace_id` is available. See [integration configuration](/docs/integrations) for connection guidance. ### Behavior notes * Per-run failures are isolated — one failing request does not break the loop * Network/API errors are handled defensively; partial evidence is preserved when possible ## Verify There is no dedicated `opensre integrations verify airflow` target today. Confirm auth by asking the agent about a DAG, or by calling the Airflow API (`GET /dags`) from the same environment. ## Troubleshooting | Symptom | Fix | | - | - | | **Auth required / 401** | Set `AIRFLOW_AUTH_TOKEN` or `AIRFLOW_USERNAME` + `AIRFLOW_PASSWORD` | | **Connection refused** | Confirm Airflow is up and `AIRFLOW_BASE_URL` matches the REST API | | **Empty DAG runs** | Name the DAG in your question, or set `AIRFLOW_DAG_ID` | | **SSL errors** | Set `AIRFLOW_VERIFY_SSL=false` only for trusted lab endpoints | ### Limitations * Requires a reachable Airflow instance * No CI-backed Airflow instance by default (local validation required) * No setup/verify CLI wiring yet ## Security * Prefer a dedicated read-only Airflow account over admin credentials * Enable TLS verification in production (`AIRFLOW_VERIFY_SSL=true`) * Store tokens/passwords in `.env` or the integration store — not in source control # Dagster Source: https://opensre.com/docs/integrations/workflows/dagster Connect Dagster so OpenSRE can investigate pipeline run failures, asset materialization errors, and sensor or schedule misfires ## Overview OpenSRE uses the Dagster GraphQL API to investigate data-pipeline failures — fetching recent runs and their status, the full event log and root-cause exception for a failed run, asset materialization history, and sensor or schedule tick history. Works against both Dagster OSS (`dagster dev` and self-hosted dagster-webserver) and Dagster+ (the SaaS). ## Prerequisites * A reachable dagster-webserver instance: * **Dagster OSS:** run `dagster dev -f jobs.py` locally or deploy `dagster-webserver`. Default port `3000`. * **Dagster+:** an active deployment, e.g. `https://.dagster.cloud/` or `https://..dagster.cloud/`. * Network access from the OpenSRE environment to the webserver * For Dagster+: a **User Token** from **Organization Settings → Tokens → User Tokens** (not an Agent Token — Agent Tokens are rejected by the GraphQL endpoint) ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup dagster ``` You will be prompted for: * **Dagster webserver URL** — `http://localhost:3000` for OSS local dev, or `https://.dagster.cloud/` for Dagster+ (the client appends `/graphql` itself) * **Dagster API token** — required for Dagster+; leave blank for unauthenticated OSS Setup validates with a GraphQL `version` probe, writes `DAGSTER_ENDPOINT` to `.env`, and persists the API token (when provided) to `~/.opensre/credentials.json`. ### Option 2: Environment variables ```bash theme={null} DAGSTER_ENDPOINT=https://your-org.dagster.cloud/prod DAGSTER_API_TOKEN=... ``` | Variable | Default | Description | | - | - | - | | `DAGSTER_ENDPOINT` | — | **Required.** Base URL of the dagster-webserver. Trailing `/graphql` is accepted and stripped | | `DAGSTER_API_TOKEN` | *(empty)* | Required for Dagster+. Leave empty for unauthenticated local OSS. Sent as `Dagster-Cloud-Api-Token` | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "dagster-prod", "service": "dagster", "status": "active", "credentials": { "endpoint": "https://your-org.dagster.cloud/prod", "api_token": "..." } } ] } ``` ## Credentials **Endpoint:** the browser URL through the deployment name, e.g. `https://acme.dagster.cloud/prod` (from `…/prod/runs`). EU accounts use a regional subdomain such as `https://acme.eu.dagster.cloud/prod`. **API token (Dagster+):** 1. User menu → **Organization Settings** 2. **Tokens** tab → **+ Create user token** 3. Copy the token immediately (shown once) User Tokens inherit the user's per-deployment role. Viewer is enough for the read-only queries OpenSRE issues. > **Token type matters.** Use a **User Token**, not an **Agent Token**. Agent Tokens authenticate Hybrid agents and return HTTP 401 on GraphQL. ### Quick local test ```bash theme={null} uv pip install dagster dagster-webserver # or: pip install ... cat > jobs.py << 'EOF' from dagster import job, op @op def deploy_payment_service(): raise RuntimeError("payment-service deploy failed: connection refused to billing-api") @job def payment_service_deploy(): deploy_payment_service() EOF dagster dev -f jobs.py -p 3001 ``` In a second terminal, launch a real run so there is something to investigate: ```bash theme={null} curl -s -X POST http://localhost:3001/graphql -H "Content-Type: application/json" -d '{ "query": "mutation { launchRun(executionParams: {selector: {repositoryLocationName: \"jobs.py\", repositoryName: \"__repository__payment_service_deploy\", jobName: \"payment_service_deploy\"}}) { __typename ... on LaunchRunSuccess { run { runId status } } } }" }' ``` ```bash theme={null} export DAGSTER_ENDPOINT="http://localhost:3001" export OPENSRE_INTEGRATIONS_STORE_PATH="$(mktemp /tmp/opensre-dagster-demo.XXXXXX)" ``` The empty temporary integration store prevents saved integrations from overriding the demo endpoint. Verify: ```bash theme={null} opensre integrations verify dagster ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL dagster │ local env │ ✓ passed │ Connected to Dagster version 1.13.18. ``` The exported endpoint also makes Dagster available in chat; no separate setup step is required. Now ask the agent about the failed run: ```bash theme={null} opensre ``` Ask: *Why did the payment\_service\_deploy Dagster job fail?* The agent lists failed runs and pulls the run's event log, finding the real cause against this exact local job: the `deploy_payment_service` op raised `RuntimeError('payment-service deploy failed: connection refused to billing-api')` at `jobs.py:6`. Teardown by pressing `Ctrl-C` in the terminal running `dagster dev`, then remove the temporary store and unset the demo variables in the second terminal: ```bash theme={null} rm -f "$OPENSRE_INTEGRATIONS_STORE_PATH" unset DAGSTER_ENDPOINT OPENSRE_INTEGRATIONS_STORE_PATH ``` The temporary `DAGSTER_HOME` is removed when `dagster dev` exits. ## Tools | Tool | What it does | | - | - | | `list_dagster_runs` | Recent runs with status, job name, timestamps, duration; filterable by status/job | | `get_dagster_run_logs` | Event log for a run; surfaces user-code exceptions from `error.cause` | | `list_dagster_assets` | Assets with latest materialization timestamp + run id | | `list_dagster_sensor_ticks` | Tick history via full `SensorSelector` triplet | | `list_dagster_schedule_ticks` | Tick history via full `ScheduleSelector` triplet | GraphQL queries OpenSRE issues are **read-only** (no mutations). ## Verify ```bash theme={null} opensre integrations verify dagster ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL dagster local env passed Connected to Dagster version 1.13.6. ``` The verifier issues `query { version }` and reports the running Dagster version on success. ## Troubleshooting | Symptom | Fix | | - | - | | **HTTP 401 with HTML body** | Usually (1) Agent Token instead of User Token, (2) user lacks role on the deployment, or (3) token revoked — check Organization Settings → Tokens | | **Invalid JSON in response** | Wrong URL (UI path instead of GraphQL). Paste only the base through the deployment name; `/graphql` is appended automatically | | **Connection refused** | Start `dagster dev -f jobs.py` locally, or check Dagster+ deployment status | | **`InvalidPipelineRunsFilterError`** | Use a valid `RunStatus`: `QUEUED`, `NOT_STARTED`, `MANAGED`, `STARTING`, `STARTED`, `SUCCESS`, `FAILURE`, `CANCELING`, `CANCELED` | | **`RunNotFoundError`** | Confirm run id and deployment slug match | | **`SensorNotFoundError` / `ScheduleNotFoundError`** | Confirm the selector triplet names match the UI exactly | ## Security * Prefer a dedicated User Token on a service-style user account (Dagster+ has no first-class service accounts). * Keep tokens out of source control — use `.env` or `~/.opensre/integrations.json`. * Rotate/revoke tokens from Organization Settings → Tokens. * For local OSS without auth, restrict the webserver to localhost or a private network. # Kafka Source: https://opensre.com/docs/integrations/workflows/kafka Connect Kafka so OpenSRE can inspect topic health and consumer group lag during incidents ## Overview OpenSRE queries Kafka to retrieve topic partition health, consumer group lag, and broker metadata — helping diagnose lag spikes, under-replicated partitions, and consumer group failures during incidents. All operations are read-only. ## Prerequisites * Apache Kafka cluster (2.x or later) * Network access from the OpenSRE environment to the Kafka brokers * `confluent_kafka` available in the OpenSRE environment (installed with the project) ## Setup There is no dedicated `opensre integrations setup kafka` wizard today. Configure via environment variables or the persistent store, then verify. ### Option 1: Environment variables ```bash theme={null} KAFKA_BOOTSTRAP_SERVERS=broker1:9092,broker2:9092 KAFKA_SECURITY_PROTOCOL=PLAINTEXT # or SASL_SSL, SSL, SASL_PLAINTEXT KAFKA_SASL_MECHANISM=PLAIN # optional, for SASL KAFKA_SASL_USERNAME=your-username # optional KAFKA_SASL_PASSWORD=your-password # optional ``` | Variable | Default | Description | | - | - | - | | `KAFKA_BOOTSTRAP_SERVERS` | — | **Required.** Comma-separated broker addresses | | `KAFKA_SECURITY_PROTOCOL` | `PLAINTEXT` | `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL` | | `KAFKA_SASL_MECHANISM` | — | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, or `AWS_MSK_IAM` where supported | | `KAFKA_SASL_USERNAME` | — | SASL username | | `KAFKA_SASL_PASSWORD` | — | SASL password | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "kafka-prod", "service": "kafka", "status": "active", "credentials": { "bootstrap_servers": "broker1:9092,broker2:9092", "security_protocol": "SASL_SSL", "sasl_mechanism": "PLAIN", "sasl_username": "your-username", "sasl_password": "your-password" } } ] } ``` ## Credentials ### Common configurations **MSK (AWS Managed Kafka) with IAM:** ```bash theme={null} KAFKA_BOOTSTRAP_SERVERS=b-1.your-cluster.kafka.us-east-1.amazonaws.com:9098 KAFKA_SECURITY_PROTOCOL=SASL_SSL KAFKA_SASL_MECHANISM=AWS_MSK_IAM ``` **Confluent Cloud:** ```bash theme={null} KAFKA_BOOTSTRAP_SERVERS=pkc-xxxxx.us-east-1.aws.confluent.cloud:9092 KAFKA_SECURITY_PROTOCOL=SASL_SSL KAFKA_SASL_MECHANISM=PLAIN KAFKA_SASL_USERNAME=your-api-key KAFKA_SASL_PASSWORD=your-api-secret ``` ## Tools | Tool | What it does | | - | - | | `get_kafka_topic_health` | Partition metadata: leader, replicas, ISR, under-replicated partitions | | `get_kafka_consumer_group_lag` | Committed offsets vs high watermarks per partition for a consumer group | ## Verify ```bash theme={null} opensre integrations verify kafka ``` Expected output: ``` Service: kafka Status: passed Detail: Connected to Kafka cluster with 3 broker(s) and 42 topic(s) ``` Verify uses a live `AdminClient.list_topics` call. ## Troubleshooting | Symptom | Fix | | - | - | | **Connection timeout** | Check broker hostnames, ports, and firewall rules | | **Authentication failed** | Verify SASL credentials and mechanism match the broker config | | **SSL handshake error** | Ensure the broker's TLS certificate is trusted or configure a CA cert | | **Leader not available** | Broker may be restarting — wait and retry | ## Security * Use **SASL\_SSL** in production — avoid `PLAINTEXT` outside local development. * Create a dedicated Kafka user with **Describe** permissions only — no produce or consume. * Store credentials in `.env` or the integration store, not in source code. # Prefect Source: https://opensre.com/docs/integrations/workflows/prefect Connect Prefect so OpenSRE can inspect flow runs, workers, and deployments during incidents ## Overview OpenSRE queries Prefect to retrieve recent flow runs (including logs for failed runs) and worker health — helping diagnose pipeline failures and identify stalled or crashed orchestration jobs. ## Prerequisites * Prefect Cloud account or self-hosted Prefect Server * API key (Prefect Cloud) or accessible API URL (self-hosted) ## Setup There is no dedicated `opensre integrations setup prefect` wizard today. Configure via the persistent store (OpenSRE does not define product `PREFECT_*` env vars), then verify. ### Option 1: Persistent store Add to `~/.opensre/integrations.json`: ```json theme={null} { "version": 1, "integrations": [ { "id": "prefect-prod", "service": "prefect", "status": "active", "credentials": { "api_url": "https://api.prefect.cloud/api", "api_key": "your-prefect-api-key", "account_id": "your-account-id", "workspace_id": "your-workspace-id" } } ] } ``` | Field | Default | Description | | - | - | - | | `api_url` | `https://api.prefect.cloud/api` | Prefect API URL (override for self-hosted) | | `api_key` | — | Prefect Cloud API key | | `account_id` | — | Prefect Cloud account ID | | `workspace_id` | — | Prefect Cloud workspace ID | ## Credentials ### Prefect Cloud 1. In Prefect Cloud, go to your **profile icon** → **API Keys** 2. Click **Create API Key** 3. Copy the key, account ID, and workspace ID from the URL: `https://app.prefect.cloud/account//workspace//` ### Self-hosted Prefect Server Set `api_url` to your server's API endpoint (no API key required if unauthenticated): ```json theme={null} { "api_url": "http://prefect-server:4200/api", "api_key": "" } ``` ## Tools | Tool | What it does | | - | - | | `prefect_flow_runs` | Lists recent flow runs filtered by state (FAILED, CRASHED, etc.); can fetch logs for a specific run via `fetch_logs_for_run_id` | | `prefect_worker_health` | Worker health and heartbeat status across work pools | ## Verify ```bash theme={null} opensre integrations verify prefect ``` Expected output on success (self-hosted, `api_url` set): ``` Service: prefect Status: passed Detail: Configured for Prefect at http://prefect-server:4200/api. ``` For Prefect Cloud (only `api_key` set, no `api_url`): ``` Service: prefect Status: passed Detail: Configured for Prefect at cloud. ``` This is a **configuration-presence** check (`api_url` or `api_key` is set) — it does not call the Prefect API. ## Troubleshooting | Symptom | Fix | | - | - | | **401 Unauthorized** | Check your API key and account/workspace IDs | | **Connection refused** | Verify `api_url` is reachable — check firewall rules for self-hosted | | **No flow runs returned** | Flow runs may have been purged — check retention settings | | **Verify passed but tools fail** | Expected if credentials are present but wrong — confirm Cloud vs self-hosted fields | ## Security * Use a **service account API key** rather than a personal key. * For self-hosted, restrict network access to the Prefect API to trusted IPs. * Store credentials in `~/.opensre/integrations.json`, not in source code. # RabbitMQ Source: https://opensre.com/docs/integrations/workflows/rabbitmq Connect RabbitMQ so OpenSRE can diagnose queue backlogs, consumer issues, and broker health during incidents ## Overview OpenSRE uses the RabbitMQ Management HTTP API to investigate message-bus incidents — checking queue backlogs, consumer health, broker-wide resource usage, cluster partition state, and connection patterns. From OpenSRE's perspective the Management API is **read-only** — no messages are published, consumed, or deleted. ## Prerequisites * RabbitMQ 3.12+ (3.13 recommended) * The **`rabbitmq_management`** plugin enabled on the broker: ```bash theme={null} rabbitmq-plugins enable rabbitmq_management ``` * Network access from the OpenSRE environment to the Management API port (default **15672**, not AMQP 5672) * A user with at least the **`monitoring`** tag ## Setup There is no dedicated `opensre integrations setup rabbitmq` wizard today. Configure via environment variables or the persistent store, then verify. Alias service name: `amqp`. ### Option 1: Environment variables ```bash theme={null} RABBITMQ_HOST=rmq.example.com RABBITMQ_MANAGEMENT_PORT=15672 RABBITMQ_USERNAME=opensre_ro RABBITMQ_PASSWORD=... RABBITMQ_VHOST=/ RABBITMQ_SSL=false RABBITMQ_VERIFY_SSL=true ``` | Variable | Default | Description | | - | - | - | | `RABBITMQ_HOST` | — | **Required.** RabbitMQ server hostname or IP | | `RABBITMQ_MANAGEMENT_PORT` | `15672` | Management API HTTP port (use `15671` for HTTPS) | | `RABBITMQ_USERNAME` | — | **Required.** Management API user | | `RABBITMQ_PASSWORD` | *(empty)* | Management API password (empty allowed) | | `RABBITMQ_VHOST` | `/` | Target vhost — diagnostic queries are scoped here | | `RABBITMQ_SSL` | `false` | Use HTTPS for the Management API | | `RABBITMQ_VERIFY_SSL` | `true` | Verify TLS certificate; set `false` only for self-signed certs in trusted networks | ### Option 2: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "rabbitmq-prod", "service": "rabbitmq", "status": "active", "credentials": { "host": "rmq.example.com", "management_port": 15672, "username": "opensre_ro", "password": "...", "vhost": "/", "ssl": false, "verify_ssl": true } } ] } ``` ## Credentials ### Recommended user setup ```bash theme={null} # Create user rabbitmqctl add_user opensre_ro strong-password # Grant the monitoring tag (read-only management API access) rabbitmqctl set_user_tags opensre_ro monitoring # Grant read-only permissions on the target vhost rabbitmqctl set_permissions -p / opensre_ro "^$" "^$" ".*" ``` The `monitoring` tag grants read access to all management endpoints without the ability to publish, consume, create, or delete resources. The permissions line grants no configure or write access (`^$`), and read access to all resources (`.*`). ### TLS SSL is disabled by default because many Management API deployments use HTTP internally. For production endpoints exposed over the network: ```bash theme={null} RABBITMQ_SSL=true RABBITMQ_MANAGEMENT_PORT=15671 ``` Set `RABBITMQ_VERIFY_SSL=false` only for self-signed certificates in trusted networks. ## Tools | Tool | What it does | | - | - | | `get_rabbitmq_queue_backlog` | Queues ranked by pending messages (ready + unacked); scoped to vhost | | `get_rabbitmq_consumer_health` | Active consumers: tag, ack mode, prefetch, channel/connection | | `get_rabbitmq_broker_overview` | Cluster summary: version, rates, totals, memory/disk/FD alarms | | `get_rabbitmq_node_health` | Per-node memory/disk/FD/sockets/Erlang processes, partition state | | `get_rabbitmq_connection_stats` | Connections sorted by recv rate; filtered to configured vhost | ## Verify ```bash theme={null} opensre integrations verify rabbitmq ``` Expected output: ``` SERVICE SOURCE STATUS DETAIL rabbitmq local env passed Connected to RabbitMQ 3.13.0 (cluster: rabbit@prod-01, vhost: /). ``` Verify calls Management `GET /api/overview`. ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused on port 15672** | Enable `rabbitmq_management` and confirm the port is reachable | | **Management API not found (404)** | Run `rabbitmq-plugins enable rabbitmq_management` (restart if needed) | | **Authentication failed (401)** | Confirm username/password (`rabbitmqctl list_users`) | | **Forbidden (403)** | Grant at least `monitoring`: `rabbitmqctl set_user_tags opensre_ro monitoring` | | **SSL: CERTIFICATE\_VERIFY\_FAILED** | Install the correct CA or set `RABBITMQ_VERIFY_SSL=false` in trusted networks | | **Queues/consumers from other vhosts appear** | Set `RABBITMQ_VHOST` correctly | | **Empty consumer list** | Confirm consumers on the configured vhost (`rabbitmqctl list_consumers -p /your-vhost`) | ## Security * Use a **dedicated `monitoring` user** — never `guest` or an `administrator`-tagged user. * Enable **TLS** when the Management API is exposed over the network. * Keep passwords out of source control — use `.env` or the persistent store. * Rotate credentials periodically. # Temporal Source: https://opensre.com/docs/integrations/workflows/temporal Connect Temporal so OpenSRE can inspect workflow executions, event history, and worker health ## Overview OpenSRE queries Temporal's HTTP API to retrieve workflow executions, event history, task queue health, and namespace-level metrics — helping diagnose workflow failures, activity retries, and worker outages. > [!NOTE] > > OpenSRE connects to Temporal's **HTTP API** (the `/api/v1/...` REST interface served > by the frontend service). This is a **self-hosted** server feature, enabled with the > `--http-port` flag (dev server) or `frontend.httpPort` config. > > **Temporal Cloud is not currently supported**: Cloud exposes only gRPC/mTLS endpoints > for workflow data and an HTTP *Ops API* for control-plane management — neither is the > frontend HTTP API this integration uses. Point OpenSRE at a self-hosted Temporal > deployment. ## Prerequisites * A self-hosted Temporal Server with the HTTP API enabled * The HTTP API base URL (and an API key only if your deployment requires bearer auth) > [!WARNING] > > Port `7233` is the **gRPC** frontend port and will **not** work as `base_url` — the > HTTP API listens on a **separate** port. On the dev server it is set with > `--http-port` (it otherwise defaults to a random free port). The examples below use > `7243`. ## Setup ### Option 1: Interactive CLI ```bash theme={null} opensre integrations setup temporal ``` Prompts for HTTP API base URL, namespace, and optional API key. ### Option 2: Environment variables ```bash theme={null} export TEMPORAL_API_URL="http://localhost:7243" export TEMPORAL_NAMESPACE="default" export TEMPORAL_API_KEY="" # only if your deployment requires bearer auth ``` | Variable | Default | Description | | - | - | - | | `TEMPORAL_API_URL` | — | **Required.** Temporal **HTTP API** base URL (`--http-port` listener, not gRPC `7233`) | | `TEMPORAL_NAMESPACE` | `default` | Namespace to query | | `TEMPORAL_API_KEY` | — | Optional bearer token (`Authorization: Bearer `) | ### Option 3: Persistent store ```json theme={null} { "version": 1, "integrations": [ { "id": "temporal-prod", "service": "temporal", "status": "active", "credentials": { "base_url": "http://temporal-frontend:7243", "namespace": "default", "api_key": "" } } ] } ``` ## Credentials Set `base_url` / `TEMPORAL_API_URL` to the frontend's HTTP API endpoint. Leave `api_key` empty for unauthenticated clusters. ### Quick local test with Docker ```bash theme={null} docker run -d --rm \ --name temporal-dev \ -p 127.0.0.1:7233:7233 \ -p 127.0.0.1:8233:8233 \ -p 127.0.0.1:7243:7243 \ temporalio/temporal:latest \ server start-dev \ --ip 0.0.0.0 \ --http-port 7243 i=0 until curl -sf -o /dev/null http://localhost:7243/api/v1/namespaces/default; do i=$((i + 1)) [ "$i" -ge 30 ] && { echo "ERROR: Temporal never became ready" >&2; exit 1; } sleep 2 done [ $i -lt 30 ] || { echo "ERROR: Temporal never became ready" >&2; exit 1; } ``` | Port | Purpose | | - | - | | `7233` | gRPC frontend (SDKs, `temporal` CLI) | | `8233` | Web UI (`http://localhost:8233`) | | `7243` | **HTTP API** — set this as `base_url` | Confirm the HTTP API answers: ```bash theme={null} curl -s http://localhost:7243/api/v1/namespaces/default ``` ```bash theme={null} export TEMPORAL_API_URL="http://localhost:7243" ``` Verify: ```bash theme={null} opensre integrations verify temporal ``` ``` SERVICE │ SOURCE │ STATUS │ DETAIL temporal │ local env │ ✓ passed │ Successfully connected to Temporal. ``` A fresh namespace has no workflows, and `temporal_namespace_info`'s `workflow_count: "0"` on its own isn't useful evidence. Run a real, deliberately failing workflow first: ```bash theme={null} pip install temporalio cat > worker.py << 'EOF' import asyncio from datetime import timedelta from temporalio import activity, workflow from temporalio.client import Client from temporalio.common import RetryPolicy from temporalio.worker import Worker @activity.defn async def deploy_payment_service() -> str: raise RuntimeError("payment-service deploy failed: connection refused to billing-api") @workflow.defn class PaymentServiceDeploy: @workflow.run async def run(self) -> str: return await workflow.execute_activity( deploy_payment_service, start_to_close_timeout=timedelta(seconds=5), retry_policy=RetryPolicy(maximum_attempts=1), ) async def main(): client = await Client.connect("localhost:7233") worker = Worker(client, task_queue="payment-service-queue", workflows=[PaymentServiceDeploy], activities=[deploy_payment_service]) async with worker: try: await client.execute_workflow(PaymentServiceDeploy.run, id="payment-service-deploy-1", task_queue="payment-service-queue") except Exception as e: print("workflow failed:", e) if __name__ == "__main__": asyncio.run(main()) EOF python worker.py ``` Chat sessions only treat an integration as active once the store resolution has *something* to fall through to env vars with -- an untouched `~/.opensre/integrations.json` with an unrelated integration in it blocks env-var fallback entirely. Point `OPENSRE_INTEGRATIONS_STORE_PATH` at an empty, valid store instead, so your real config is never read or written and `TEMPORAL_API_URL` above is the only source of connection info: ```bash theme={null} export OPENSRE_INTEGRATIONS_STORE_PATH=$(mktemp /tmp/opensre-temporal-demo.XXXXXX) echo '{"version": 2, "integrations": []}' > "$OPENSRE_INTEGRATIONS_STORE_PATH" ``` A literal zero-byte file does not work here -- the store loader expects valid JSON and raises on an empty read, so this writes an explicitly empty (but valid) store rather than just creating the file with `mktemp` alone. Now ask the agent about the failed workflow: ```bash theme={null} opensre ``` Ask: *Why did the Temporal workflow payment-service-deploy-1 fail?* Against this exact local workflow the agent calls all 4 registered tools — namespace info, workflow list, workflow history, and task queue — and finds the real cause: the `deploy_payment_service` activity raised `RuntimeError('payment-service deploy failed: connection refused to billing-api')` and, with `maximumAttempts=1`, the failure propagated straight to `WORKFLOW_EXECUTION_FAILED`. An empty store makes the session fall through to env-var resolution for *every* integration, not just Temporal -- if your shell already resolves another integration from its own env vars, it rides along too (harmless). Teardown: ```bash theme={null} docker rm -f temporal-dev rm -f "$OPENSRE_INTEGRATIONS_STORE_PATH" unset OPENSRE_INTEGRATIONS_STORE_PATH TEMPORAL_API_URL ``` ## Tools | Tool | What it does | | - | - | | `temporal_namespace_info` | Namespace state and workflow counts by status (Running, Failed, TimedOut) | | `temporal_workflows` | Recent executions with status, type, task queue, timing | | `temporal_workflow_history` | Event history for a specific execution | | `temporal_task_queue` | Active pollers and backlog stats (depth, add/dispatch rates) | ### Typical flow when you ask about a failed workflow 1. **Namespace info** — how many workflows are running vs failed? 2. **Workflows** — filter to failed/timed-out executions; note workflow type and task queue 3. **Workflow history** — which activity failed and why? 4. **Task queue** — are workers polling? Is backlog growing? ## Verify ```bash theme={null} opensre integrations verify temporal ``` Verify probes `GET /api/v1/namespaces/{namespace}`. ## Troubleshooting | Symptom | Fix | | - | - | | **Connection refused / protocol errors** | You may be on the gRPC port — use the HTTP API port (`--http-port`, e.g. `7243`), not `7233` | | **404 on `/api/v1/...`** | Enable HTTP API (`--http-port` or `frontend.httpPort`) | | **401 Unauthorized** | Set `TEMPORAL_API_KEY` to a valid bearer token | | **404 Namespace not found** | Confirm `namespace` matches exactly (case-sensitive) | | **Empty workflow list** | Workflows may have passed retention — check namespace retention | | **No pollers on task queue** | Workers may be down — check worker deployment health | ## Security * Use a **read-only API key** where your deployment supports scoped auth — OpenSRE never writes to Temporal. * Restrict network access to the HTTP API to trusted IPs. * Store credentials in `~/.opensre/integrations.json` or environment variables, not in source code. # Interactive shell action policy Source: https://opensre.com/docs/interactive-shell-action-policy # Interactive Shell Action Policy (ADR) ## Status Superseded — Jun 18, 2026. The declarative-rule-pack deterministic mapper and the regex-based planner postprocessing overrides described in the original decision have been removed. See "Decision (current): LLM is the sole tool selector" below. The original decision is retained for historical context. ## Context The interactive-shell action policy had grown through layered heuristics in single modules: a regex/keyword deterministic mapper inferred tools from free-form text, and planner postprocessing rewrote the model's chosen actions with more regex. These heuristics competed with the LLM and caused misclassifications (e.g. "investigate a sample test alert?" being treated as an informational question instead of running the sample alert), and they were a recurring source of precedence drift. ## Decision (current): The shell action agent is the sole tool selector 1. There is no regex/keyword intent inference. Non-command turns are selected entirely by the shell action agent via native tool-calling. 2. Tool selection is driven by the action-agent system prompt (`core/agent_harness/prompts/action/assemble.py`) and the per-tool descriptions in the tool catalog (`tools/interactive_shell/*`). Keep both precise — they are the only selection signal. 3. The action path does not post-hoc rewrite the model's tool calls. Tool calls execute as first-class `AgentTool`s through the shared `core` tool-calling loop; argument shape and availability are enforced by the AgentTool runtime contract and per-tool gates. 4. When the action-agent prompt overflows the context window, the turn falls through to a conversational reply rather than guessing an action. When the action-agent LLM itself is unavailable, the REPL renders and persists a failed assistant turn so `/resume` can show the outage. 5. Literal `/slash` command text the user types verbatim is dispatched deterministically, without the action-agent LLM (see the "Deterministic literal-`/slash` dispatch" addendum below). This is an explicit-command bypass, not natural-language intent inference: free-form text is still selected entirely by the action agent. The runtime's literal-`/slash` detection in `runtime/input_policy._literal_slash_command_text` remains terminal-UI policy (spinner suppression and exclusive-stdin gating); the execution-side deterministic dispatch lives in `core/agent_harness/turns/action_driver.py`. ## What this means for changes * To change how a phrasing maps to a tool, edit the action-agent system prompt and/or the relevant tool description — never add a regex. * To add a new tool, add it to the tool catalog with a clear, self-describing `description` and `input_schema`; the action agent selects it from that text and receives it as an AgentTool. * Live turn scenarios under `tests/core/agent/scenarios/` are the regression surface for action-agent behavior. Deterministic scenarios (`intent_class: deterministic`) assert literal command dispatch only. ## Original decision (historical, superseded) 1. Deterministic mapping was split into declarative rule packs with one explicit precedence table. 2. Rule matching windows were named typed strategies instead of inline numeric slices. 3. Planner postprocessing ran as pure transforms over a typed `PlannerState`. 4. Fail-closed policy transforms and normalization transforms were registered separately and executed in one ordered list. 5. Legacy planner-result tuple compatibility was collapsed behind a single adapter. 6. Planner contracts included policy-trace artifacts to detect silent precedence drift. ## Integration awareness and LLM-driven read-only discovery Addendum — Jun 18, 2026. Factual questions about live state (for example "is sentry installed?") are answered without adding keyword/regex rules. Two complementary mechanisms: 1. Context grounding (not action planning). At REPL boot, `run_repl_async` (`surfaces/interactive_shell/main.py`) hydrates `session.configured_integrations` from the shared `configured_integration_services()` helper in `integrations/catalog.py` (the same source the welcome banner uses, so they never diverge). The agent prompt lists the configured set as facts, letting the model answer directly when state is already known. 2. LLM-driven discovery. The agent system prompt (`core/agent_harness/prompts/action/assemble.py`) lets the model, at its own discretion, emit a read-only discovery action (for example `slash_invoke("/integrations", ["list"])` or `["verify"]`) to discover the answer instead of deflecting. There is no keyword mapping for this — the LLM decides. Under the alpha allow-all policy every discovery action runs without confirmation (`execution_policy.allow_tool("slash")` returns `allow`); the former `ExecutionTier`/`resolve_slash_execution_tier` classification was removed because it gated nothing. No fail-closed regex rule is involved; the agent decides whether to emit a discovery action. ### Observe→answer summary loop Addendum — Jun 18, 2026. When the agent runs a read-only discovery command, the tool result is appended to the same ReAct conversation. The same model then writes the user-facing answer. There is no second assistant pass or deterministic pre-agent dispatch. Discovery commands also no longer dump validator stack traces into the REPL: a vendor/config failure during verification (for example a GitHub MCP `401`) is logged as a one-line warning instead of a full traceback, because `report_validation_failure` now defaults to `include_traceback=False` while still capturing the exception to Sentry. ### Auto-launching interactive setup ("can you configure X?") Addendum — Jun 18, 2026. When the user asks to configure, connect, set up, or add an integration ("can you configure sentry?", "connect datadog"), the action agent does not just hand off to the conversational assistant — it launches the setup wizard for them. The action agent emits a `slash_invoke` tool call for `/integrations setup ` or `/mcp connect `. The model chooses the service; there is no per-vendor hardcoding. The setup wizard is a child process that needs exclusive stdin, so it cannot run inline mid-turn (the live prompt is competing for stdin). Instead `tools/interactive_shell/actions/slash.py` queues the command via `session.queue_auto_command(...)`, which prefills the next prompt and marks it for auto-submit. The prompt refresh hook (`wire_prompt_refresh` in `surfaces/interactive_shell/ui/input_prompt/refresh.py`) then submits it, so the command flows through the normal exclusive-stdin turn path of the REPL (`turn_needs_exclusive_stdin` recognizes `/integrations setup`) — the only place an interactive child process gets clean stdin. In a non-TTY/scripted context (no prompt to submit into), the slash command path degrades to normal non-interactive slash behavior. ### Removal of the planning-stage fail-closed safeguard (v0.1) Addendum — Jun 18, 2026. The action agent does not deny a turn. Previously, any clause the old planner could not map to an executable tool — flagged via the `mark_unhandled` tool, an `UNHANDLED:` text marker, or an unavailable tool call — collapsed the whole turn into a hard denial that printed *"I couldn't safely decide actions for that request."* In practice this fired on legitimate input (most often a conversational question that embedded a quoted, list-style directive such as *figure out why X is crashing by querying (a) sentry, (b) github, (c) posthog*), producing a dead end with no safety benefit. Every terminal action in v0.1 is **read-only**, so an unmatched, ambiguous, or chatty clause is not a safety risk. The action agent now: * runs every clause it *can* map to an executable action, and * lets everything else fall through to the conversational assistant (or simply drops a chatty clause in a compound request). Removed as part of this change: the `denied` field on `ActionPlanningDecision`, `enforce_plan_fail_closed_policy`, `normalize_terminal_plan`, `render_plan_denied`, the `mark_unhandled` planner tool, and the `UNHANDLED:` convention. The `fail_closed`, `has_unhandled_clause`, and `turn.expected_signals` fields were also removed from turn scenario fixtures, since the oracle never asserted on them; the fixture `policy` block now carries a single `executes_terminal_action` `boolean` (true only when a shell action AgentTool is expected to run). If write/mutating actions are introduced later, gate them with the execution-stage confirmation policy (`tools/interactive_shell/shared/execution_policy.py`), **not** an action-selection denial. ### Shell commands during alpha Addendum — Jun 27, 2026. **Behavior:** while OpenSRE is in **alpha**, the interactive REPL accepts every nonempty shell command. There is no command allowlist or hard deny floor. This is a deliberate trade-off: alpha prioritizes developer velocity over command sandboxing, and commands run on the developer's machine with their privileges. * Read-only and mutating commands, shell operators (`| && ; > <`), command substitution (`` ` ``/`$(...)`), redirects, and heredocs are all supported. Compact forms such as `cat README.md;echo done` work without spaces around the operator. The optional `!` prefix is still accepted. * The **only** remaining non-execution outcome is genuinely empty input (a bare `!` or whitespace), which is rejected as input validation, not as a guardrail. Each command starts a new shell in OpenSRE's working directory. A directory change therefore applies only within the same command; prefix each command that needs another directory with `cd path && command`. On POSIX, commands use non-interactive `/bin/sh` syntax rather than loading the configured interactive shell, its aliases, or its startup files. The existing confirmation flow remains available to **`/auto`** (Off/Low/Med) and `/trust`. At those stricter `/auto` levels, every shell command asks for approval, including commands that appear read-only. Plan-only also asks before running a shell command; its confirmation distinguishes allowing just that command from authorizing the rest of the plan. The shell interprets expansions after the approval decision, so OpenSRE does not exempt shell text based on a partial parser. ### `/auto` autonomy (tool-type confirmations) Addendum — Aug 2026. **Decision:** the REPL exposes `/auto off|low|med|high` as session-scoped tool-approval autonomy. Default is **high** (alpha: no confirmation). Lower levels promote a default-`allow` policy result to `ask` based on `tool_type` (`config/constants/repl_autonomy.py`), not a shell-command allowlist. | Level | Asks before | | - | - | | `high` | Nothing | | `med` | Mutating agent tools (`shell`, `code_agent`, `slash`, `cli_command`, `opensre_cli`, `switch_llm_provider`, `synthetic_test`, `sentry_issue_fix`, …) | | `low` | Same as `med` | | `off` | Every tool type | **Interaction with `/trust`:** `trust_mode` still short-circuits `ask` to allow (skip the prompt). Non-TTY `ask` remains fail-closed. **Still true:** there is no shell argv allowlist/deny floor under alpha. `/auto` only gates whether the existing confirmation UX runs before a tool launch. If a shell-command allowlist is reintroduced after alpha, gate it here at the execution stage — never with an action-selection denial in the planner. ### Deterministic literal-`/slash` dispatch (no LLM) Addendum — Jun 28, 2026. **Decision:** input the user types as a literal `/slash` command is dispatched deterministically, **without consulting the action-agent LLM**. This supersedes the earlier "the literal-`/slash` detection must never become an action execution shortcut" wording. **Why:** all REPL turns previously routed through the action-agent LLM, so when that LLM was unavailable (a provider with no credit, a failed auth, an outage) every slash command failed — including the exact commands needed to recover (`/login`, `/auth`, `/onboard`, `/model`). That is a deadlock: you could not log in because logging in required the LLM you were trying to fix. Typed commands should not depend on a funded LLM. **Scope — the line that keeps the original concern intact.** The original ADR removed regex/keyword heuristics because they *inferred intent from natural language* and competed with the LLM. This bypass does the opposite: it fires **only** when the message text itself is a literal `/command` the user typed verbatim. There is no inference. Free-form natural language ("log me in", "show my integrations") is still selected entirely by the action agent. The line is "explicit typed command" vs "natural language", identical to the line the terminal-UI policy (`_literal_slash_command_text`) already draws. **How it works.** `core/agent_harness/turns/action_driver.ActionTurnRunner` recognizes literal `/slash` input and emits a deterministic `slash_invoke` tool call through the same static-LLM path as the explicit `!cmd` shell escape (`_StaticToolCallLLM`). Execution then flows through the normal `slash_invoke` AgentTool → `dispatch_slash`, so recording, execution policy, exclusive-stdin gating, pickers, and exit behavior are identical to an LLM-selected slash call — the only difference is the tool *selection* is deterministic instead of LLM-driven. The bypass no-ops (falls back to the LLM path) when `slash_invoke` is not an available tool that turn. **Consequences.** * Literal slash commands are faster, free, and reliable even with no LLM credit. * Compound requests that *start with* a literal slash are dispatched as that one command (a single `slash_invoke`); compound phrasing that does not start with a slash (e.g. `run /health and then investigate …`) is unaffected and still LLM-routed. No turn scenario uses a prompt beginning with a literal `/`. * A literal discovery command (e.g. `/integrations list`) shows its raw output directly; the optional LLM summary pass only runs if the action-agent LLM is available, and degrades cleanly (no summary) when it is not. **Still forbidden:** regex/keyword/fuzzy intent routing for natural language, post-hoc rewriting of LLM-selected tool calls, and any deterministic mapping from non-`/`-prefixed text to an action. Those compete with the LLM and were removed for good reason. ### Plan guards (host-enforced) The live plan (`update_plan`) is checked by the host, not trusted from the model: * A step is `completed` only when a non-bookkeeping tool returned while it was `in_progress`; a plan cannot be born or bulk-ticked complete (`core/agent_harness/task_plan/completion.py`). * The step marked `verifies: true` is the only one labelled `(verify)`. It is never exempt from that rule, and a text-only closing step closes for free only after such a step completed. Without one the closing step is reset and the tool result says so; the model adds a check or marks the step `blocked`, and the result is reported as unverified. * A step newly marked `blocked` does not end the turn: the conclusion is rejected until the model has asked the user how to resolve it (`core/agent_harness/task_plan/conclusion.py`), and the "Plan ended" breakdown is not printed while that question is queued. * The onboarding menu's answer turn that loads the chosen demo skill and does nothing else is rejected once, with a nudge to write the plan and run the first step; the first demo stalled that way live. * The second work tool of a turn is refused while no plan with open work is stored (`core/agent_harness/task_plan/required.py`). The refusal names the fix: re-send the tool in one response with the plan write listed before it. These are before/after tool hooks on the execution path, the same seam as the duplicate-call guard; they never route intent or rewrite a tool call. # Closed-Loop Learning Source: https://opensre.com/docs/platform/closed-loop-learning How the agent carries lessons from one session into the next # Closed-Loop Learning OpenSRE learns across sessions through [long-term memory](/docs/platform/memory). When a conversation surfaces a durable fact — a root cause worth remembering, a known-flaky service, a naming convention, a preference — the agent stores it as a local memory file, and the memory index grounds every future conversation. There is nothing to configure: memories accrue as you work. Review what the agent has learned with `/memory`, inspect one entry with `/memory show `, and correct course by deleting stale entries with `/memory forget ` or by simply telling the agent the updated fact. See [Long-Term Memory](/docs/platform/memory) for storage locations, gateway scoping, and how to turn memory off. # Scheduled Deliveries (Cron) Source: https://opensre.com/docs/platform/cron Cron-driven recurring reports to Telegram, Slack, Discord, Rocket.Chat, and the interactive shell # Scheduled Deliveries OpenSRE can deliver recurring reports to messaging providers on a cron schedule. Use it for recurring prompt loops, repository-scoped GitHub CI health reports, GitHub PR standups, PostHog metric reports, and work-item reminders — delivered automatically without manual CLI runs. For human task reminders and proactive work check-ins, use `opensre work add --remind-at ...` or `opensre work schedule-checkin ...`. Those commands create the right scheduler entries, support repeated `--target provider:chat_id` fan-out delivery, and keep task metadata in the work-item store. ## Demo picker on launch Every interactive launch (`opensre`) asks which demo to run. The recommended option analyzes CI/CD performance on a real repository. Option B schedules a CI/CD reliability agent; see [CI/CD analytics demo](/docs/cicd/cicd-analytics-demo). Escape skips onboarding; "Or type your own answer..." lets you submit a different request. Run `opensre --skip-onboarding` to open the shell without the startup picker. Type `/demo` to start it later. ## Quick Start ```bash theme={null} # Add a free-form recurring ops digest to Telegram at 09:00 IST on weekdays opensre cron add --kind manual_loop --cron "0 9 * * mon-fri" \ --name "Ops digest" --prompt "Check open incidents and summarize production risk" \ --tz Asia/Kolkata --provider telegram --chat-id # Same kind to Slack via incoming webhook (omit --chat-id; webhook is channel-bound) opensre cron add --kind manual_loop --cron "0 9 * * mon-fri" \ --prompt "Check open incidents and summarize production risk" \ --tz Europe/London --provider slack # Slack bot token: omit --chat-id when SLACK_DEFAULT_CHAT_ID is set opensre cron add --kind manual_loop --cron "0 9 * * mon-fri" \ --prompt "Check open incidents and summarize production risk" \ --tz Europe/London --provider slack # GitHub PR sweep standup digest → Slack opensre cron add --kind github_pr_sweep --cron "0 9 * * mon-fri" \ --tz Europe/London --provider slack --chat-id C0123ABCD # Read-only CI health for one repository and branch → Slack opensre cron add --kind recurring_skill --skill reporting-github-ci-failures \ --cron "0 8 * * mon-fri" \ --tz Europe/London --provider slack --owner Tracer-Cloud --repo opensre \ --branch main # Morning-report skill with its previewed city → local inbox opensre cron add --kind recurring_skill --skill delivering-morning-briefings \ --cron "0 8 * * mon-fri" --tz Asia/Kolkata \ --provider interactive_shell --city "New Delhi" # Local interactive-shell inbox delivery, useful for debugging opensre cron add --kind manual_loop --cron "0 9 * * mon-fri" \ --name "Local ops digest" --prompt "Check open incidents and summarize production risk" \ --provider interactive_shell # List configured tasks opensre cron list # Inspect durable backlog pressure opensre cron status # Run a task immediately (for debugging) opensre cron run # Start the scheduler daemon (or rely on `opensre gateway start`) opensre cron start ``` ## CLI Commands In the interactive shell, bare `/cron logs`, `/cron run`, and `/cron remove` open a searchable scheduled-task picker. Search by name, kind, channel, destination, schedule, timezone, or full task ID. Each choice is one exact scheduled task, even when `/cron list` groups several channels into one loop. Selecting logs opens its history; run requires **Run now**, and remove requires **Remove permanently**. Cancel leaves the task unchanged. Supplying an ID and any flags runs the command directly; scripts and gateway sessions still require an explicit task ID. ### `opensre cron add` Create a new scheduled delivery task. | Option | Required | Description | | - | - | - | | `--name` | No | Human-readable loop name shown by `opensre cron list` and `/loops` | | `--kind` | Yes | Task kind: `manual_loop`, `recurring_skill`, `github_pr_sweep`, `posthog_metric_report`, `work_item_reminder`, `work_item_checkin` | | `--cron` | Yes | Cron expression (5 fields: minute hour day month day\_of\_week; prepend a seconds field for sub-minute polling, see [Cron Syntax](#cron-syntax)) | | `--tz` | No | IANA timezone (default: `UTC`). Examples: `Europe/London`, `US/Eastern`, `Asia/Kolkata` | | `--provider` | Yes | Messaging provider: `telegram`, `slack`, `discord`, `rocketchat`, `interactive_shell` | | `--chat-id` | Cond. | Required for telegram/discord/rocketchat. Optional for slack when an incoming webhook or `SLACK_DEFAULT_CHAT_ID` is configured, and not needed for `interactive_shell`. | | `--window` | No | Lookback window in hours (default: `24`) | | `--prompt` | Cond. | Instruction to execute on every run; required for `manual_loop` (optional with `--mode agent --skill`) and rejected for other kinds | | `--skill` | Cond. | Required with `--kind recurring_skill`; names the revision-pinned skill to execute. With `--kind manual_loop --mode agent`, names the skill each run follows, or gives the path of an installed skill folder that holds a `SKILL.md` | | `--stateless` | No | With `--kind manual_loop --mode agent`: every run starts fresh, without earlier runs, a note for the next run, or long-term memory | | `--owner` | Cond. | GitHub repository owner; required for the `reporting-github-ci-failures` recurring skill | | `--repo` | Cond. | GitHub repository name; required for the `reporting-github-ci-failures` recurring skill | | `--branch` | No | Branch filter for `reporting-github-ci-failures`; cannot be combined with `--pr` | | `--pr` | No | Pull-request filter for `reporting-github-ci-failures`; cannot be combined with `--branch` | | `--city` | No | City for `delivering-morning-briefings`; omitted to use its default-location behavior | ### `opensre cron list` Scan scheduled tasks in a table on wide terminals or labeled rows on narrow terminals. IDs remain visible below each task; run times are UTC and each cron expression uses its displayed timezone. Use `opensre --json cron list` for complete configuration. ### `opensre cron remove ` Remove a scheduled task by its ID. Its run history remains available through `opensre cron logs `. ### `opensre cron status` Show the number of durable runs waiting for execution, including expired claims awaiting recovery. Age starts at admission for pending runs and at lease expiry for interrupted runs; live claims are excluded. Waiting work remains visible across scheduler restarts and while a task is paused or disabled. Deleted tasks are excluded. Use `--json` for structured output suitable for monitoring and runbooks. JSON output includes `status: ok`; if the task store or run database cannot be read (including storage access or lock failures), the command exits unsuccessfully and reports `status: unknown` with null metrics instead of returning a misleading empty backlog. The JSON `error` field identifies `task_store_unreadable` or `run_store_unreadable`. ### `opensre cron run ` Execute a task immediately (ad-hoc one-shot). Useful for debugging delivery without waiting for the next cron tick. By default this delivers to every configured destination again, even ones that already received the message — that is what you want to trigger a task on demand. To recover a partial failure instead, use `--failed-only`: it retries only the destinations the most recent run failed at, using the saved report without executing the task again. A missing saved report requires a deliberate full run. `--failed-only` never widens. If it cannot read per-target history for the task — no prior run, or one recorded before that history was tracked — it stops and tells you, rather than quietly delivering everywhere. Re-run without the flag when a full send is what you want. When the last run was a partial failure, a plain `cron run` warns which destinations it is about to re-deliver to before it sends. ### `opensre cron logs ` Show execution history and the newest retained report. Work status and delivery status are separate: `blocked` work can have a successfully delivered report. `unknown` work status means the older run did not retain completion evidence. Use `--run ` to read one historical attempt and `--json` for structured results. Reports remain available after removing the schedule. For a repository repair loop, use `--kind manual_loop --mode agent` with `--owner --repo ` and optionally `--pr ` or `--branch `. The repository target is saved with the task. CI repairs and security repairs that open a PR create their own matching checkouts when no explicit workspace is supplied. With `--pr` or `--branch`, the loop pauses when its target can never be repaired, such as a fork PR, a closed PR, or a merge that waits on a person's decision. Without them, the loop sweeps the repository: it skips those targets, keeps its schedule, and records a run that verifiably found nothing to repair as a no-op. A no-op run that replies with only its note delivers nothing. Add `--skill ` to run a skill instead of writing the instructions in `--prompt`. Each run follows the skill as currently installed, so fixes to the skill reach the loop without recreating it. For example, this loop merges the base branch into conflicting pull requests every hour and comments on the ones that need a person's decision: ```bash theme={null} opensre cron add --kind manual_loop --mode agent --skill fixing-github-merge-conflicts \ --owner --repo --cron "4 * * * *" --provider interactive_shell ``` A skill from a public registry works the same way: install it into a folder, then pass that folder's path. Keep the `--prompt` short and say what to pick, what never to do, and what to reply: ```bash theme={null} npx clawhub@latest install skills-sh:cursor/plugins/fix-ci --workdir ~ --dir skills opensre cron add --kind manual_loop --mode agent --stateless \ --skill ~/skills/fix-ci --owner Tracer-Cloud --repo opensre \ --cron "7 * * * *" --provider interactive_shell --name "PR CI fix" \ --prompt "Fix the oldest open PR whose latest checks failed. Never merge or force-push." ``` `opensre cron run ` prints the work result, delivery result, and report. A blocked or incomplete agent task exits unsuccessfully even if its report was delivered. Agent tasks need a structured completion result from their execution tools; report text alone is recorded as unverified work. ### `opensre cron start` Start the blocking scheduler daemon. Loads all enabled tasks and fires them according to their cron schedules. Blocks until `SIGINT` or `SIGTERM`. To keep it running without a terminal, `/loops service install` in the shell registers a per-user service (launchd on macOS, systemd on Linux) that runs `opensre cron start --service` at login; `/loops service remove` deletes it. ## Cron Syntax Standard 5-field cron expressions: ``` ┌───────────── minute (0-59) │ ┌───────────── hour (0-23) │ │ ┌───────────── day of month (1-31) │ │ │ ┌───────────── month (1-12) │ │ │ │ ┌───────────── day of week (0-7, Sunday is 0 or 7; or mon–sun) │ │ │ │ │ * * * * * ``` **Weekday numbering** is the Unix cron convention: `0` and `7` are Sunday, `1` is Monday and `6` is Saturday, so `1-5` runs Monday to Friday. Day names such as `mon-fri` work too. Earlier versions counted weekdays from `0` = Monday, so a loop saved with a numeric weekday now runs one day earlier than before: `1-5` moves from Tuesday–Saturday to Monday–Friday. If you wrote numbers for the old counting on purpose, for example `0` for Monday, check the loop with `/loops next `. To keep its old days, record its settings, stop it with `/loops stop `, recreate it through the original scheduling command with day names, such as `--cron "0 9 * * mon"`, and delete the old loop with `/loops delete ` once the replacement is verified. Examples: * `0 9 * * mon-fri` — weekdays at 09:00 * `0 8 * * mon` — Mondays at 08:00 * `*/30 * * * *` — every 30 minutes * `0 0 1 * *` — first day of each month at midnight A five-field expression fires at most once per minute. For polling loops that must react faster — the CI repair loop polls every 30 seconds — prepend a seconds field (`0-59`) to get a six-field expression: * `*/30 * * * * *` — every 30 seconds * `0 */5 * * * *` — every 5 minutes, on the minute (same as `*/5 * * * *`) A tick that fires while the previous tick of the same task is still running is skipped, so a slow tick never overlaps with the next one. Cron expressions are validated at `cron add` time using APScheduler's `CronTrigger`. Invalid expressions are rejected immediately. ## Timezone Behavior * All fire times are internally converted to UTC for dedup consistency * The `--tz` option accepts any IANA timezone (for example `Europe/London`, `US/Eastern`) * DST transitions are handled correctly — the UTC-normalized dedup key ensures no duplicate or missed deliveries across clock changes ## Dedup Semantics The scheduler uses a SQLite-backed claim store with a unique task, fire-time, and attempt key. Each attempt has a 2-minute lease and a random owner token. The worker renews that lease while it runs; two minutes is how long a worker that died mid-tick keeps the next tick from starting. 1. When a cron tick fires, `EVENT_JOB_SUBMITTED` captures `scheduled_run_times[0]` and the job uses that UTC-normalized `fire_time` for the claim key 2. The executor creates an attempt with a 2-minute lease and owner token 3. If the current attempt is still leased, another instance skips the tick 4. If the lease expired, the old attempt is marked `abandoned` and one new worker creates the next attempt 5. Completion is accepted only from the attempt's owner token, so an old worker cannot overwrite a reclaimed attempt The scheduler also runs a recovery sweep every minute, including after a restart. It finds expired running attempts and resubmits their original fire-time keys through the same claim path. Recovery preserves the destination restriction of a `--failed-only` retry, so it does not resend to destinations outside that retry's scope. Attempts created before delivery scopes were persisted cannot safely recover automatically. They appear as failed runs with a delivery-scope error; use `opensre cron run --failed-only` when readable per-target history exists, or explicitly rerun the task for all destinations after checking its logs. This prevents concurrent duplicate claims and recovers ticks after worker crashes. A crash after an external provider accepts a message but before the completion is recorded can still result in a duplicate on reclaim; preventing that case requires provider-specific idempotency support. ## Capacity and overload Each admitted cron tick is first written as a durable `pending` row in `scheduler.db`. It is not silently dropped because workers are busy. The default scheduler concurrency remains **2**; a scheduled agent turn also needs a permit from the shared process turn gate, so the effective agent-turn concurrency can be lower. The current scheduler bounds active workers, but its executor can still retain more submitted callbacks in memory while those workers are busy. A large burst therefore remains recoverable from SQLite, but is not yet a bounded in-memory queue. Use the baseline below to size an installation rather than raising the default to absorb a burst. A later scheduler change will move overflow to the durable pending store and reserve recovery control work from user execution. | Boundary | Owns | Does not own | | - | - | - | | Task store | Task definitions and enabled state | Execution history | | Run store | Durable admissions, claims, leases, recovery candidates, and completion fencing | Worker scheduling | | Scheduler runner | Cron submission, recovery sweep, and task lookup | LLM or provider delivery | | Scheduler executor | The currently submitted worker callbacks | Durable recovery state | | Shared turn gate | Process-wide concurrent agent turns | Scheduler admission or persistence | The capacity baseline records these terms consistently: * **Durable pending count** and **oldest pending age**: `pending` rows and the elapsed time since their durable admission. Read them with `opensre cron status`; the benchmark reads the same stores directly. * **In-memory callbacks**: callbacks submitted to the current worker executor that have not completed. This is the current overload signal, not a durable queue limit. * **Throughput and drain time**: successful fake deliveries per second and the elapsed time to finish admitted work. * **Recovery query latency and plan**: time and SQLite plan used to find pending or expired work. Completed history must not expand the candidate scan. * **SQLite lock failures** and **RSS**: lock errors plus current and peak process resident memory sampled by the benchmark host. Run the deterministic baseline before changing `OPENSRE_SCHEDULER_MAX_CONCURRENT_RUNS`: ```bash theme={null} uv run python -m tests.benchmarks.scheduler.capacity_benchmark ``` It makes no LLM, provider, or network calls. The checked-in JSON report at `docs/benchmarks/scheduler-capacity-baseline.json` records the exact workload, host metadata, SQLite version, and observed values. Restart recovery uses the production shape: one recovery callback iterates its durable candidates serially. ## Credential Resolution Credentials are resolved lazily at delivery time in this priority order: 1. **Task params** — credentials stored in the task definition (not recommended) 2. **Integration store** — `~/.opensre/integrations.json` (configured via `opensre integrations`) 3. **Environment variables** — `TELEGRAM_BOT_TOKEN`, `SLACK_BOT_TOKEN`, `DISCORD_BOT_TOKEN`, `ROCKETCHAT_*` You do not need to pass credentials at `cron add` time — they are picked up from your existing integration configuration. ## Task Kinds | Kind | Behavior | | - | - | | `manual_loop` | Runs a stored prompt as one headless agent turn and delivers the answer. This is what `/loops` creates. | | `recurring_skill` | Runs a named, revision-pinned recurring skill. `delivering-morning-briefings` preserves its optional city; `reporting-github-ci-failures` requires repository scope and can be narrowed to a branch or PR. | | `github_pr_sweep` | Headless GitHub PR standup (mergeable / stale / conflicted). Requires GitHub configured; posts via the chosen provider. | | `posthog_metric_report` | Headless PostHog metric report over the configured window. Requires PostHog configured. | | `work_item_reminder` | Reminder for a single work item at its `--remind-at` time. Created by `opensre work add --remind-at …`. | | `work_item_checkin` | Recurring check-in on open work items. Created by `opensre work schedule-checkin …`. | Sentry morning digests use a separate CLI (`opensre sentry digest schedule …`), not `opensre cron add`. See [Sentry](/docs/integrations/incidents/sentry#morning-digest-scheduled). ## What a loop remembers between runs A loop added with `--stateless` remembers nothing: each run sees only its prompt, its skill and what its tools find, and long-term memory is neither shown nor written. A stateless run that finds nothing to act on replies `NO_ACTION`: it delivers nothing and counts as a no-op. A stateless run that pushes a commit, or creates, comments on or edits a pull request or issue, counts as done work. Everything below applies to loops without `--stateless`. Every run of a `manual_loop` starts a fresh session. Before it starts, the loop sees its three most recent finished runs: when each ran, how its work and delivery ended, the changes it made with tools (for example a pull-request comment, a push, or a message it sent), and the start of its report. It uses them to avoid repeating work and to follow up on open items, and it still checks the current state with its tools before it acts. To carry something specific forward, a loop ends its reply with one line: ```text theme={null} NOTE FOR NEXT RUN: Commented on PR #6555 at head 1a2b3c4; it waits for a human decision. ``` The note, up to 300 characters, is saved with that run, removed from the delivered report, and shown to the next run. Only a note that ends the reply counts: one followed by more text, or quoted with `>`, stays in the report. Loops already know about this line; to make sure a loop uses it, ask for it in the prompt, for example "leave a note listing the PRs you commented on". The next run reads earlier runs, notes included, as a record of what happened, never as instructions, because reports can quote pull requests and issues. Put standing rules, such as "comment on a PR only once per head commit", in the loop's prompt. * Changes made with the GitHub CLI, shell commands and chat messages are listed; reads such as viewing a PR or `git status` are not. A command is listed by its leading words (`gh pr comment 6555 …`), never with option values such as a comment body. A change made through an integration that does not report it may be missing, so when an action must never repeat, have the loop name it in its note. * History belongs to one loop. A loop recreated with new settings starts without history. ## Persistence * **Task definitions** are stored in `~/.opensre/scheduler_tasks.json` (JSON + filelock) * **Execution history** is stored in `~/.opensre/scheduler.db` (SQLite with WAL mode) * **Run records** keep each task's newest 20 run attempts in `~/.opensre/scheduler_runs/.json`: the prompt sent, the report, delivery results, the changes made and the note for the next run. A loop's next run reads its history from here; prompts and reports are credential-redacted. * **Interactive-shell loop messages** are stored in `~/.opensre/scheduler_loop_messages.jsonl` * **Operational breadcrumbs** are appended to `~/.opensre/operations_log.jsonl` with loop lifecycle and run status metadata, not prompt or message bodies * Both survive process restarts — `opensre cron list` and `opensre cron logs` read from disk Set `OPENSRE_OPERATIONS_LOG_PATH` to write the operations log somewhere else, or `OPENSRE_OPERATIONS_LOG_DISABLED=1` to turn it off. Set `OPENSRE_OPERATIONS_LOG_MAX_BYTES` to change the rotation threshold. ## REPL Prefer `/loops` for user-facing recurring prompt loops: ```text theme={null} /loops add --name "Morning ops" --time 08:30 --prompt "Check open incidents and summarize production risk" --run-now /loops next /loops stop /loops delete /loops messages ``` You can also ask naturally: ```text theme={null} Set up a manual loop called Morning ops at 08:30 UTC to check open incidents and summarize production risk, and run it once now. ``` By default, `/loops add` sends to every configured default handle it can reach: Telegram when `TELEGRAM_BOT_TOKEN` and `TELEGRAM_DEFAULT_CHAT_ID` are configured, Slack when `SLACK_WEBHOOK_URL` is configured **or** when `SLACK_BOT_TOKEN` and `SLACK_DEFAULT_CHAT_ID` are configured, and the local interactive-shell inbox. When a scheduled loop or skill finishes, the scheduler delivers the report body to those channels without you prompting — you do not need to be in the REPL or Slack chat at fire time. Loops created before Slack was in your default channel list stay inbox-only until you recreate them with Slack in `--channels` (or rely on the updated defaults for new loops). The lower-level `/cron` slash command forwards to the CLI: ``` /cron list /cron add --kind manual_loop --cron "0 9 * * *" --prompt "Check open incidents" --provider telegram --chat-id /cron run ``` Run `/loops` to see named loops with active/draft state, execution time, channels, last run, and next fire time. Use `/loops stop ` to pause a loop without losing it, `/loops start ` to re-enable it, and `/loops delete ` to remove it. Onboarding seeds a few draft starter loops, such as a weekday morning report, so you have concrete examples before creating an active delivery. # Interactive Shell Privacy Source: https://opensre.com/docs/platform/interactive-shell-privacy How OpenSRE redacts secrets from command history and LLM prompt/response logs, and the controls you have over persistence and cloud telemetry. ## Overview The interactive shell saves every line you type to a history file so up-arrow recall and `/history` work across sessions. Each new prompt reloads the file when another running shell has changed it, so concurrent sessions share recent commands without a restart. Separately, it records each LLM prompt/response turn for local debugging and `/resume`. Incident prompts can include sensitive identifiers and tokens, so the shell: * Redacts known token shapes before each entry is written to disk * Supports disabling persistence entirely (memory-only mode) * Caps how many entries are kept (oldest pruned) * Offers `/history clear` to wipe the file on demand The history file is `~/.opensre/interactive_history`. See [Prompt and response logging](#prompt-and-response-logging) for the separate LLM turn log and remote product-analytics forwarding. ## Defaults | Setting | Default | Effect | | - | - | - | | Persistence | **on** | Lines you type are appended to the history file. | | Redaction | **on** | Known token shapes are replaced with `[REDACTED:]` before writing. | | Retention cap | **5000 entries** | Older entries are pruned when the cap is exceeded. | ## Redaction patterns The built-in patterns target token shapes that are distinctive enough to keep false positives on natural-language incident text low. Each match becomes a labeled placeholder. | Kind | Examples | | - | - | | `aws_key` | `AKIA…`, `ASIA…` | | `aws_secret` | `aws_secret_access_key=…` | | `github_pat` | `ghp_…`, `github_pat_…` | | `anthropic_key` | `sk-ant-…` | | `openai_key` | `sk-…` | | `slack_token` | `xoxb-…`, `xoxp-…`, `xoxa-…` | | `stripe_key` | `sk_live_…`, `sk_test_…` | | `bearer` | `Bearer ` headers | | `jwt` | `eyJ…` three-segment tokens | | `password` | `--password=…`, `password=…` | | `private_key` | PEM-encoded private keys | Redaction applies only to **persistent history**. The line you typed is still passed to OpenSRE’s normal pipeline as you typed it. ## Slash commands | Command | Effect | | - | - | | `/history` | Show all persisted entries. | | `/history clear` | Wipe the history file. Up-arrow recall resets on next launch. | | `/history off` | Pause persistence for this session. New entries are not written. | | `/history on` | Resume persistence for this session. | | `/history retention ` | Keep at most N entries on disk. Prunes immediately. | | `/privacy` | Show current persistence and redaction state, retention cap, and threat model. | ## Configuration Settings resolve from (highest wins): 1. Environment variables 2. The `interactive.history` block in `~/.opensre/config.yml` 3. Built-in defaults ### Environment variables | Variable | Default | Effect | | - | - | - | | `OPENSRE_HISTORY_ENABLED` | `1` | Set to `0` / `false` / `off` to skip persistence entirely (in-memory only). | | `OPENSRE_HISTORY_REDACT` | `1` | Set to `0` / `false` / `off` to disable redaction (raw `FileHistory`). | | `OPENSRE_HISTORY_MAX_ENTRIES` | `5000` | Non-negative integer. `0` disables the cap (unlimited). | ### Config file ```yaml theme={null} interactive: history: enabled: true redact: true max_entries: 5000 ``` ## Prompt and response logging Separately from typed-command history, OpenSRE records each agent turn — the submitted prompt and final response — in the interactive shell, `opensre ask`, gateway chat, and scheduled or embedded agent runs. Failures and cancellations retain the prompt and available error details. Each continuation is recorded separately. This log is richer than command history (it includes model output, not just what you typed) and is used for two purposes: 1. **Local debugging / `/resume`**: appended as JSON Lines to `~/.opensre/prompt_log.jsonl`, and folded into the session file so `/resume` can restore conversation context. Gateway logs use the organization and user conversation directory rather than the host-wide default. An explicit `OPENSRE_PROMPT_LOG_PATH` overrides that location. 2. **Product analytics**: forwarded through the first-party OpenSRE app endpoint as an `$ai_generation` event (model, provider, latency, token counts, and the prompt/response text) so usage and quality can be tracked. Failed turns carry structured error properties (`$ai_is_error`, `$ai_error`, `error_kind`). Turns handled entirely by terminal tools or slash commands report `$ai_model` / `$ai_provider` as `no_conversational_agent`; when a conversational reply was intended but the LLM provider failed (missing API key, model access, quota, auth), the event reports the attempted model (or `unknown`) plus an `ai_error_kind` bucket (`not_configured`, `quota`, `auth`, or `provider_error`). A turn that calls the model also records `model_blocks` in both places: the characters and estimated tokens of each prompt block, of the replayed history and of your message, and the number of tool schemas. It holds sizes and block names only, never prompt text. `/context` shows the same breakdown for your next message. ### Defaults | Setting | Default | Effect | | - | - | - | | Logging | **on** | Each LLM turn is recorded. | | Local JSONL file | **on** | Turns are appended to `~/.opensre/prompt_log.jsonl`. | | Remote forwarding | **on** | Turns are also sent as a product-analytics `$ai_generation` event. | | Redaction | **on** | Known token shapes (same patterns as command history) are stripped from the prompt and response before either sink. | ### Environment variables | Variable | Default | Effect | | - | - | - | | `OPENSRE_PROMPT_LOG_DISABLED` | `0` | Set to `1` to disable prompt/response logging entirely (both local file and remote analytics). | | `OPENSRE_PROMPT_LOG_LOCAL_DISABLED` | `0` | Set to `1` to skip the local JSONL file while leaving remote forwarding (if enabled) unaffected. | | `OPENSRE_PROMPT_LOG_REDACT` | `1` | Set to `0` to log/send raw prompt and response text without redaction. | | `OPENSRE_PROMPT_LOG_PATH` | `~/.opensre/prompt_log.jsonl` | Override the local JSONL file path. | The separate operations log at `~/.opensre/operations_log.jsonl` records agent and scheduler lifecycle breadcrumbs such as loop start, iteration outcome, loop creation, and delivery status. It does not store prompt or response bodies. Remote forwarding for this event also honors the global telemetry opt-outs: set **`OPENSRE_NO_TELEMETRY=1`**, `OPENSRE_ANALYTICS_DISABLED=1`, or `DO_NOT_TRACK=1` to stop all product analytics (including `$ai_generation`) without touching the local JSONL file. See [Environment variables](/docs/install/environment-variables#telemetry-monitoring). ### Config file ```yaml theme={null} interactive: prompt_log: posthog_enabled: true redact: true max_chars: 32000 path: ~/.opensre/prompt_log.jsonl ``` Redaction here uses the same built-in pattern set as command history (see [Redaction patterns](#redaction-patterns) above). It catches known secret shapes, not arbitrary sensitive content. Raw incident details, hostnames, or business context in a prompt are not redacted — only credential-shaped substrings. ## Threat model The history file is **plain text on local disk** at `~/.opensre/interactive_history`, with the user’s default file permissions. Built-in redaction targets common token shapes only — it is not a substitute for proper secret handling. Treat the file as confidential and be aware: * A determined attacker with read access to your home directory can still read entries written before redaction was enabled. * Redaction cannot detect tokens that look like normal text (for example a natural-language password). Do not paste secrets you would not put in a system log. * Custom redaction patterns are not supported in v1. To redact internal token shapes, use `/history off` for that session and run `/history clear` afterwards. The prompt/response log has the same caveats, plus one more: with remote forwarding on (the default), redacted prompt/response text leaves your machine. If you discuss confidential systems or data in the interactive shell, set `OPENSRE_NO_TELEMETRY=1` (or `OPENSRE_PROMPT_LOG_DISABLED=1` to also stop the local file) rather than relying on redaction alone. For a strong local posture: set `OPENSRE_HISTORY_ENABLED=0` and `OPENSRE_NO_TELEMETRY=1`. That stops command-history persistence and remote analytics. Up-arrow recall across sessions is lost. `/resume` conversation context also depends on session files and the prompt log — set `OPENSRE_PROMPT_LOG_DISABLED=1` (or clear session/prompt-log data) if you need that path off too. Otherwise rely on the in-memory ring for the current process. # LLM Providers Source: https://opensre.com/docs/platform/llm-providers Supported LLM APIs and CLIs, environment variables, and how to switch between them. The interactive shell uses your signed-in OpenSRE account by default. To select your own provider billing, run `opensre auth login` and choose API keys or Subscriptions. A successful login selects that connection for reasoning, classification, and tool calls while retaining your OpenSRE identity. Other hosts and account-independent CLI workflows can use the providers below. Choose one with `LLM_PROVIDER`; API providers use an API key stored in `~/.opensre/credentials.json` or the process environment. ChatGPT subscription credentials use separate storage. Default model IDs are defined in [`config/llm_models.py`](https://github.com/Tracer-Cloud/opensre/blob/main/config/llm_models.py). Routing is in [`core/llm/factory.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/factory.py). Each provider has two model roles: * **Reasoning** — used for diagnosis, claim checks, and multi-step analysis * **Toolcall** — a lighter model used for tool selection and routing ## Quick reference **API keys** | Provider | Config | Default models | | - | - | - | | Anthropic API key | `LLM_PROVIDER=anthropic` `ANTHROPIC_API_KEY` | Reasoning: `claude-opus-4-7` Toolcall: `claude-haiku-4-5-20251001` | | OpenAI API key | `LLM_PROVIDER=openai` `OPENAI_API_KEY` | Reasoning: `gpt-5.4-mini` Toolcall: `gpt-5.4-mini` | | OpenRouter | `LLM_PROVIDER=openrouter` `OPENROUTER_API_KEY` | Reasoning: `openrouter/auto` Toolcall: `openrouter/auto` | | TrustedRouter | `LLM_PROVIDER=trustedrouter` `TRUSTEDROUTER_API_KEY` | Reasoning: `trustedrouter/auto` Toolcall: `trustedrouter/auto` | | DeepSeek | `LLM_PROVIDER=deepseek` `DEEPSEEK_API_KEY` | Reasoning: `deepseek-v4-pro` Toolcall: `deepseek-v4-flash` | | Google Gemini API key | `LLM_PROVIDER=gemini` `GEMINI_API_KEY` | Reasoning: `gemini-3.1-pro-preview` Toolcall: `gemini-3.1-flash-lite-preview` | | NVIDIA NIM | `LLM_PROVIDER=nvidia` `NVIDIA_API_KEY` | Reasoning: `meta/llama-3.1-405b-instruct` Toolcall: `meta/llama-3.1-8b-instruct` | | MiniMax | `LLM_PROVIDER=minimax` `MINIMAX_API_KEY` | Reasoning: `MiniMax-M3` Toolcall: `MiniMax-M2.7-highspeed` | | Groq | `LLM_PROVIDER=groq` `GROQ_API_KEY` | Reasoning: `llama-3.3-70b-versatile` Toolcall: `llama-3.1-8b-instant` | | Azure OpenAI | `LLM_PROVIDER=azure-openai` `AZURE_OPENAI_API_KEY` + resource URL | Reasoning: `gpt-5.4-mini` (deployment name) Toolcall: `gpt-5.4-mini` (deployment name) | | Custom OpenAI-compatible | `LLM_PROVIDER=custom-openai` `CUSTOM_OPENAI_API_KEY` + `CUSTOM_OPENAI_BASE_URL` | Your `CUSTOM_OPENAI_MODEL` (both roles) | | Custom Anthropic-compatible | `LLM_PROVIDER=custom-anthropic` `CUSTOM_ANTHROPIC_API_KEY` + `CUSTOM_ANTHROPIC_BASE_URL` | Your `CUSTOM_ANTHROPIC_MODEL` (both roles) | **Subscriptions** | Provider | Setup | Model selection | | - | - | - | | ChatGPT | `opensre auth login chatgpt` | Models available to the signed-in account; `CHATGPT_MODEL` records the selected model | | Grok subscription (local testing) | `opensre auth login grok` | Browser login; no client-ID setup; `GROK_MODEL` records the selected account model | **Cloud & local** | Provider | Config | Default models | | - | - | - | | Amazon Bedrock | `LLM_PROVIDER=bedrock` AWS IAM (`AWS_REGION`) | Reasoning: `us.anthropic.claude-sonnet-4-6` Toolcall: `us.anthropic.claude-haiku-4-5-20251001-v1:0` | | Google Vertex AI | `LLM_PROVIDER=vertex-ai` Google ADC (`VERTEX_AI_PROJECT`) | Reasoning: `gemini-2.5-pro` Toolcall: `gemini-2.5-flash-lite` | | Ollama (local) | `LLM_PROVIDER=ollama` No API key (local daemon) | Reasoning: `llama3.2` Toolcall: `llama3.2` | **CLI** | Provider | Config | Default models | | - | - | - | | Google Gemini CLI | `LLM_PROVIDER=gemini-cli` `gemini` login or API key env | Gemini CLI default (both roles) | | Google Antigravity CLI | `LLM_PROVIDER=antigravity-cli` `agy` browser login | Antigravity CLI configured model (both roles) | | GitHub Copilot CLI | `LLM_PROVIDER=copilot` `copilot login` or `gh auth login` | Copilot CLI default (both roles) | | xAI Grok Build CLI | `LLM_PROVIDER=grok-cli` `grok login` | Grok Build CLI default (both roles) | | Cursor Agent CLI | `LLM_PROVIDER=cursor` `agent login` or `CURSOR_API_KEY` | Cursor CLI default (`CURSOR_MODEL` to override) | | OpenCode CLI | `LLM_PROVIDER=opencode` `opencode auth login` or provider API keys | OpenCode configured model (`OPENCODE_MODEL` to override) | | Kimi Code CLI | `LLM_PROVIDER=kimi` `kimi login` or `KIMI_API_KEY` | Kimi CLI default (`KIMI_MODEL` to override) | | Pi CLI (BYOK) | `LLM_PROVIDER=pi` Provider API key env or `pi` → `/login` | Pi configured model (`PI_MODEL` to override; both roles) | ## Selecting a provider Set `LLM_PROVIDER` (default: `anthropic`) in your environment or `.env` file: ```bash theme={null} export LLM_PROVIDER=openai export OPENAI_API_KEY=sk-... ``` Or use the onboarding wizard, which writes the same values to `.env`: ```bash theme={null} opensre onboard ``` Onboarding collects an API key for API providers. Choose ChatGPT subscription under **Other LLM provider** to sign in using your browser. Local provider configuration supports custom model IDs for OpenAI, OpenRouter, TrustedRouter, Gemini, NVIDIA, Bedrock, local CLIs, Ollama, and DeepSeek. When using OpenSRE hosted billing, set `OPENSRE_LLM_CONNECTION=local` alongside your provider settings to select your own billing instead: ```bash theme={null} export OPENSRE_LLM_CONNECTION=local export LLM_PROVIDER=openai export OPENAI_REASONING_MODEL=gpt-6.1-sol export OPENAI_TOOLCALL_MODEL=gpt-6.1-sol ``` `gpt-6.1-sol` is the hosted account default. On the native OpenAI SDK and Codex, unset reasoning effort sends extra high (`xhigh`) for that model. LiteLLM and OpenRouter leave effort unset unless you set it. Tool calls on the native OpenAI SDK use the Responses API. GPT-5.6 remains available as `gpt-5.6-sol`, `gpt-5.6-terra`, and `gpt-5.6-luna`. The bare `gpt-5.6` name maps to Sol. Override defaults with env vars: ```bash theme={null} export OPENAI_REASONING_MODEL=gpt-5.4-mini export OPENAI_TOOLCALL_MODEL=gpt-5.4-mini ``` `LLM_MAX_TOKENS` (default `25000`) sets the response token budget for all providers. ## LiteLLM transport You can send hosted API providers through [LiteLLM](https://docs.litellm.ai/) instead of the native vendor SDK. This is optional for most providers and **required** for Azure OpenAI. | Command / variable | What it does | | - | - | | `export OPENSRE_LLM_TRANSPORT=litellm` | Route API providers through LiteLLM | | unset or `export OPENSRE_LLM_TRANSPORT=sdk` | Use native SDK clients (default) | | `opensre onboard` → **Azure OpenAI** | Sets `OPENSRE_LLM_TRANSPORT=litellm` automatically | CLI providers (`codex`, `claude-code`, `copilot`, `pi`, `cursor`, `opencode`, `kimi`, and others) always run as a subprocess. LiteLLM does not apply to them. ### Providers supported via LiteLLM When `OPENSRE_LLM_TRANSPORT=litellm` (or `LLM_PROVIDER=azure-openai`), OpenSRE builds the same tool schemas as the SDK path and passes them to `litellm.completion(..., tools=..., tool_choice="auto")`. LiteLLM routes the request; OpenSRE still handles schema cleanup, retries, and message replay. | `LLM_PROVIDER` | Native SDK path (default) | LiteLLM path | Notes | | - | - | - | - | | `anthropic` | Anthropic SDK | `anthropic/` | Opt-in with `OPENSRE_LLM_TRANSPORT=litellm` | | `openai` | OpenAI SDK | `openai/` | Opt-in | | `bedrock` | boto3 / AnthropicBedrock / Converse | `bedrock/` | Opt-in; uses AWS credentials | | `openrouter` | OpenAI-compatible SDK | `openai/` + OpenRouter base URL | Opt-in | | `trustedrouter` | OpenAI-compatible SDK | `openai/` + TrustedRouter base URL | Opt-in | | `deepseek` | OpenAI-compatible SDK | `openai/` + DeepSeek base URL | Opt-in | | `gemini` | OpenAI-compatible SDK | `openai/` + Gemini base URL | Opt-in | | `nvidia` | OpenAI-compatible SDK | `openai/` + NVIDIA NIM base URL | Opt-in | | `minimax` | OpenAI-compatible SDK | `openai/` + MiniMax base URL | Opt-in | | `groq` | OpenAI-compatible SDK | `openai/` + Groq base URL | Opt-in | | `ollama` | OpenAI-compatible SDK | `openai/` + `${OLLAMA_HOST}/v1` | Opt-in | | `azure-openai` | — | `azure/` | **Always** via LiteLLM | | `vertex-ai` | — | `vertex_ai/` | **Always** via LiteLLM; uses Google ADC | For GPT-5.6 agent tool calls, the native OpenAI SDK uses the Responses API and replays reasoning and function-call items between steps. Older OpenAI models and OpenAI-compatible providers still use Chat Completions. LiteLLM supports [many more backends](https://docs.litellm.ai/docs/providers). OpenSRE only wires the providers listed above. Use one of those, or open an issue if you need another first-class provider. ## Login and secret storage Use `opensre auth login` to choose **API keys** (provider API billing) or **Subscriptions** (provider plan usage). OAuth subscriptions use browser authorization; API-key providers use a hidden paste prompt. | Command | What it does | | - | - | | `opensre auth` | Show API-key and subscription auth status and the active provider | | `opensre auth login deepseek` | Guide DeepSeek setup, validate `DEEPSEEK_API_KEY`, store it in `.env` and `~/.opensre/credentials.json`, and select DeepSeek | | `opensre auth verify deepseek` | Check DeepSeek credentials and refresh local metadata | | `opensre auth logout deepseek` | Remove OpenSRE-managed DeepSeek credentials and metadata | API-key login writes the key to `.env` and `~/.opensre/credentials.json`. `opensre auth` and `/auth status` are safe to show in prompts: they do not read stored secrets. For API-key providers they check environment variables and non-secret metadata in `~/.opensre/llm-auth.json`. If you delete a key from the credentials file directly, status may look stale until you run `opensre auth verify ` or make a request. Verification marks the provider `stale` when the secret is gone. For account-independent CLI workflows: ```bash theme={null} opensre auth login openai opensre auth login anthropic opensre auth login deepseek opensre auth status ``` ## ChatGPT subscription ```bash theme={null} opensre auth login chatgpt opensre auth verify chatgpt opensre auth status chatgpt opensre auth logout chatgpt ``` Continue with ChatGPT in your browser and authorize OpenSRE to use your plan. Choose a model offered by your account, or pass `--model `. ChatGPT plan access must be enabled during authorization; available models and usage limits depend on your account. Review usage and app access in [ChatGPT Settings → Usage](https://chatgpt.com/#settings/Usage). Verification checks credentials and model access; a completed inference request is still needed to confirm that plan usage is available. Use `--new-account` to add another account or workspace. Use `--profile ` to sign in to a saved registration listed by `opensre auth status chatgpt`. Registrations remain separate even when their email addresses match. Logout clears the selected account's tokens and retains its registration for later sign-in. Other saved accounts and API keys remain. Login selects `LLM_PROVIDER=chatgpt` and your own plan billing, while preserving your OpenSRE account identity. `--no-set-provider` saves credentials without changing the selected provider. To restore OpenSRE hosted billing for a signed-in account, set `OPENSRE_LLM_CONNECTION=account` in `.env`. To switch to API billing, run `opensre auth login openai` (or another API provider). Subscription credentials are stored separately in owner-only files under `~/.opensre/subscriptions/chatgpt/`; keep this directory private and out of source control. `OPENSRE_SUBSCRIPTION_AUTH_DIR` overrides the parent directory. Subscription login requires local credential storage and is unavailable when `OPENSRE_DISABLE_KEYRING` is set. Tokens renew automatically; an expired or revoked session requires another explicit login. ChatGPT requests use LiteLLM automatically, regardless of `OPENSRE_LLM_TRANSPORT`. For a terminal without an available browser, pass `--no-open-browser` and open the displayed URL yourself. The browser must reach the terminal's loopback callback; for SSH, forward local port 1455 to the remote port 1455 before login. If that port is busy, login displays the alternate callback port to forward. Provider API keys remain available through the separate API-key path. ## Grok subscription (local testing) Sign in with an eligible Grok subscription; no API key or client-ID setup is needed. **Live subscription inference and billing validation in OpenSRE remain pending.** ```bash theme={null} opensre auth login grok ``` OpenSRE uses the public Grok CLI OAuth registration also used by OpenCode, Kilo, and Hermes. If you have a separate xAI-approved registration, optionally set `OPENSRE_GROK_OAUTH_CLIENT_ID` before adding an account. Existing accounts keep the registration used at sign-in, even if this override changes. Login prints a verification URL containing your short user code and optionally opens it in a browser. For SSH/headless hosts, use `--no-open-browser` and open the URL on another device; no port forwarding is needed. Login polls until approval, denial, or expiry. Ctrl+C cancels without replacing your saved connection. Choose one of the account-visible models, or pass optional `--model `; `GROK_MODEL` defaults to `grok-4.3` when no selection has been saved. Models must be recognized by the installed LiteLLM version for native streaming; an unknown model fails explicitly instead of silently substituting a buffered request. Browser approval alone does not guarantee model access. If xAI denies access, check that the authorized account has an active eligible plan and remaining allowance. OpenSRE does not switch to a paid API key when subscription access fails. ```bash theme={null} opensre auth status grok opensre auth verify grok opensre auth login grok --new-account opensre auth login grok --profile opensre auth logout grok ``` `status` lists saved account IDs without reading tokens. `verify` checks renewal and model discovery; inference entitlement and quota are still enforced by xAI. `--no-set-provider` saves the account without changing the active model connection. Login otherwise selects `LLM_PROVIDER=grok` and `OPENSRE_LLM_CONNECTION=local`, keeping OpenSRE account identity separate from the chosen billing route. Tokens are stored under `~/.opensre/subscriptions/grok/`, or under `$OPENSRE_SUBSCRIPTION_AUTH_DIR/grok/` if that optional override is set. Storage must be enabled (`OPENSRE_DISABLE_KEYRING` must be unset). Refresh and reconnect retain the client ID that issued the session. Logout clears only the selected account's tokens and attempts remote revocation; it reports if remote revocation is unconfirmed. Requests use LiteLLM automatically, require a completed Responses event, and preserve live text, tool/results, encrypted reasoning replay, cancellation cleanup, and token usage. Grok requests allow a 900-second timeout for long reasoning generations, following [xAI’s guidance](https://docs.x.ai/developers/model-capabilities/text/best-practices). An interrupted stream is an error. Quota/entitlement/auth failures never switch to `XAI_API_KEY`, another account, or usage billing. Before relying on this connection, locally validate login, actual inference, tool/result replay, reconnect, live renewal, quota handling, subscription billing attribution, and logout with an eligible account. The automated tests use synthetic credentials. ## API providers Open a provider for environment variables and setup notes. > **Anthropic** > > ```bash theme={null} > export LLM_PROVIDER=anthropic > export ANTHROPIC_API_KEY=sk-ant-... > # Optional overrides: > export ANTHROPIC_REASONING_MODEL=claude-opus-4-7 > export ANTHROPIC_TOOLCALL_MODEL=claude-haiku-4-5-20251001 > ``` > > Default provider. Uses the Anthropic Python SDK. Get an API key at > [console.anthropic.com](https://console.anthropic.com/). > > Claude Sonnet 5.5 (`claude-sonnet-5-5`), Claude Opus 5.5 (`claude-opus-5-5`), > Claude Fable 5.1 (`claude-fable-5-1`), and Claude Fable 5 (`claude-fable-5`) are > also available (`/model set `, onboarding, or the Claude Code CLI provider). > Arbitrary unlisted model IDs are not automatically supported because direct > Anthropic validation is restricted to curated models; the default reasoning model > remains Claude Opus 4.7. > **OpenAI** > > ```bash theme={null} > export LLM_PROVIDER=openai > export OPENAI_API_KEY=sk-... > # Optional overrides: > export OPENAI_REASONING_MODEL=gpt-5.4-mini > export OPENAI_TOOLCALL_MODEL=gpt-5.4-mini > ``` > > Uses the OpenAI SDK. Reasoning models (`o1`, `o3`, `o4`, `gpt-5*`) use > `max_completion_tokens` instead of `max_tokens`. > **Azure OpenAI** > > Azure OpenAI always goes through LiteLLM. Model env vars are **deployment > names** from your Azure resource, not public OpenAI model IDs. > > ```bash theme={null} > export LLM_PROVIDER=azure-openai > export OPENSRE_LLM_TRANSPORT=litellm # set automatically by `opensre onboard` > > export AZURE_OPENAI_BASE_URL=https://your-resource.openai.azure.com > export AZURE_OPENAI_API_KEY=... > # Optional override; defaults to 2024-10-21 when unset: > # export AZURE_OPENAI_API_VERSION=2024-10-21 > > # Deployment names (must exist in your Azure resource): > export AZURE_OPENAI_REASONING_MODEL=gpt-5.4-mini > export AZURE_OPENAI_TOOLCALL_MODEL=gpt-5.4-mini > export AZURE_OPENAI_CLASSIFICATION_MODEL=gpt-5.4-mini > ``` > > Quick setup: > > ```bash theme={null} > opensre onboard # choose Azure OpenAI; paste resource URL, API key, deployment > ``` > > Onboarding asks for your **resource URL**, **API key**, then lists > **deployments** for you to pick. OpenSRE sets > `AZURE_OPENAI_API_VERSION=2024-10-21` and `OPENSRE_LLM_TRANSPORT=litellm` unless > you override them in `.env`. > > In the REPL: > > ```bash theme={null} > /model set azure-openai gpt-5.4-mini > /model set azure-openai gpt-5.4-mini --toolcall-model gpt-5.4-nano > ``` > > If deployment discovery fails during onboarding, enter the deployment name by > hand. It must match a deployment in your Azure resource, not a model ID from > `/openai/models`. > **Custom OpenAI / Anthropic endpoints** > > Point OpenSRE at an arbitrary base URL — a LiteLLM proxy, vLLM, LocalAI, or an > internal model gateway — when direct calls to the public APIs are not allowed. > > **`custom-openai`** reuses the OpenAI-compatible client: > > ```bash theme={null} > export LLM_PROVIDER=custom-openai > export CUSTOM_OPENAI_BASE_URL=http://localhost:4000/v1 # LiteLLM, vLLM, LocalAI, … > export CUSTOM_OPENAI_API_KEY=your-key > export CUSTOM_OPENAI_MODEL=gpt-5.4 # applies to every tier > # Optional per-tier overrides: > # export CUSTOM_OPENAI_TOOLCALL_MODEL=gpt-5.4-mini > ``` > > **`custom-anthropic`** uses the Anthropic SDK with a base-URL override: > > ```bash theme={null} > export LLM_PROVIDER=custom-anthropic > export CUSTOM_ANTHROPIC_BASE_URL=https://proxy.example.com > export CUSTOM_ANTHROPIC_API_KEY=your-key > export CUSTOM_ANTHROPIC_MODEL=claude-opus-4-7 > ``` > > Both require the base URL and a model — onboarding and startup fail if either is > missing, rather than reaching a wrong endpoint mid-run. > > * **The base URL is used verbatim.** Include the API path yourself (for example > `/v1` for OpenAI-compatible gateways). OpenSRE never appends a path. > * **`custom-anthropic` is SDK-only.** It ignores `OPENSRE_LLM_TRANSPORT=litellm` > and errors if you force it; use `custom-openai` for a LiteLLM-proxied > OpenAI-compatible endpoint. > * Run with `opensre --debug` (or set `TRACER_VERBOSE=1`) to print the resolved > provider, redacted base URL (host only — never a token), and model. > > Quick setup: > > ```bash theme={null} > opensre onboard # choose the custom provider; paste base URL, API key, model > ``` > **OpenRouter** > > ```bash theme={null} > export LLM_PROVIDER=openrouter > export OPENROUTER_API_KEY=sk-or-... > # Optional override (single value applies to both roles if set): > export OPENROUTER_MODEL=openrouter/auto > # Or per role: > export OPENROUTER_REASONING_MODEL=anthropic/claude-sonnet-4-6 > export OPENROUTER_TOOLCALL_MODEL=openai/gpt-4o-mini > ``` > > OpenAI-compatible proxy. Pick any model on > [openrouter.ai/models](https://openrouter.ai/models). Base URL: > `https://openrouter.ai/api/v1`. > **TrustedRouter** > > ```bash theme={null} > export LLM_PROVIDER=trustedrouter > export TRUSTEDROUTER_API_KEY=sk-tr-... > # Optional override (single value applies to all roles if set): > export TRUSTEDROUTER_MODEL=trustedrouter/auto > # Or per role: > export TRUSTEDROUTER_REASONING_MODEL=anthropic/claude-opus-4-7 > export TRUSTEDROUTER_TOOLCALL_MODEL=trustedrouter/fast > ``` > > OpenAI-compatible proxy. Base URL: `https://api.trustedrouter.com/v1`; keys at > [trustedrouter.com/keys](https://trustedrouter.com/keys). > > Model ids are namespaced (`anthropic/claude-opus-4-7`, `openai/gpt-5.4-mini`) — > a bare `gpt-5.4-mini` is rejected. Ids under `trustedrouter/` are routing > policies rather than single models: each picks an upstream per request and > fails over if it is down. > > | Route | Picks for | > | - | - | > | `trustedrouter/auto` | Capability, with failover (default) | > | `trustedrouter/fast` | Latency | > | `trustedrouter/cheap` | Cost | > | `trustedrouter/zdr` | Providers that retain no data | > | `trustedrouter/e2e` | Confidential-compute providers | > | `trustedrouter/eu` | EU-hosted providers | > > The full catalog is at > [api.trustedrouter.com/v1/models](https://api.trustedrouter.com/v1/models). > **DeepSeek** > > ```bash theme={null} > export LLM_PROVIDER=deepseek > export DEEPSEEK_API_KEY=sk-... > # Optional override (single value applies to all roles if set): > export DEEPSEEK_MODEL=deepseek-v4-pro > # Or per role: > export DEEPSEEK_REASONING_MODEL=deepseek-v4-pro > export DEEPSEEK_TOOLCALL_MODEL=deepseek-v4-flash > ``` > > Uses DeepSeek’s OpenAI-compatible API at `https://api.deepseek.com`. Run > `opensre auth login deepseek` for guided key setup and secure local storage. > **Google Gemini** > > ```bash theme={null} > export LLM_PROVIDER=gemini > export GEMINI_API_KEY=... > # Optional override: > export GEMINI_MODEL=gemini-3.1-pro-preview > # Or per role: > export GEMINI_REASONING_MODEL=gemini-3.1-pro-preview > export GEMINI_TOOLCALL_MODEL=gemini-3.1-flash-lite-preview > ``` > > Uses Google’s OpenAI-compatible endpoint at > `https://generativelanguage.googleapis.com/v1beta/openai/`. Get an API key at > [aistudio.google.com](https://aistudio.google.com/app/apikey). > **NVIDIA NIM** > > ```bash theme={null} > export LLM_PROVIDER=nvidia > export NVIDIA_API_KEY=nvapi-... > # Optional override: > export NVIDIA_MODEL=meta/llama-3.1-405b-instruct > # Or per role: > export NVIDIA_REASONING_MODEL=meta/llama-3.1-405b-instruct > export NVIDIA_TOOLCALL_MODEL=meta/llama-3.1-8b-instruct > ``` > > Uses NVIDIA’s OpenAI-compatible API at `https://integrate.api.nvidia.com/v1`. > Browse models on [build.nvidia.com](https://build.nvidia.com/). > **MiniMax** > > ```bash theme={null} > export LLM_PROVIDER=minimax > export MINIMAX_API_KEY=... > # Optional override (single value applies to both roles if set): > export MINIMAX_MODEL=MiniMax-M3 > # Or per role: > export MINIMAX_REASONING_MODEL=MiniMax-M3 > export MINIMAX_TOOLCALL_MODEL=MiniMax-M2.7-highspeed > ``` > > OpenAI-compatible endpoint at `https://api.minimax.io/v1`. Temperature is fixed > at `1.0` to match MiniMax guidance. > **Groq** > > ```bash theme={null} > export LLM_PROVIDER=groq > export GROQ_API_KEY=gsk_... > # Optional override: > export GROQ_MODEL=llama-3.3-70b-versatile > # Or per role: > export GROQ_REASONING_MODEL=llama-3.3-70b-versatile > export GROQ_TOOLCALL_MODEL=llama-3.1-8b-instant > ``` > > Uses Groq’s OpenAI-compatible API at `https://api.groq.com/openai/v1`. > **Amazon Bedrock** > > ```bash theme={null} > export LLM_PROVIDER=bedrock > export AWS_REGION=us-east-1 > # Optional overrides: > export BEDROCK_REASONING_MODEL=us.anthropic.claude-sonnet-4-6 > export BEDROCK_TOOLCALL_MODEL=us.anthropic.claude-haiku-4-5-20251001-v1:0 > ``` > > No API key. Auth uses the AWS credential chain (environment variables, shared > credentials file, or IAM role). Your principal needs permission to invoke the > model IDs you configure (for example Bedrock `InvokeModel` / Converse access for > those resources). > > **Model routing:** > > * **Anthropic Claude** on Bedrock (`anthropic.claude-*`, `us.anthropic.claude-*`, > and foundation-model ARNs that contain `anthropic.claude`) use the > **AnthropicBedrock** SDK path. > * **Other Bedrock foundation models** (for example Mistral, Meta Llama, or Amazon > Titan IDs enabled in your account) use the **Bedrock Converse** API via > `boto3`. You can set `BEDROCK_REASONING_MODEL` to a non-Claude model ID when > needed. > * **Application inference profile** ARNs > (`…:application-inference-profile/…`) do not encode the vendor in the ID. > Those always use **Converse**, which works for any model behind the profile. > > Defaults in `config/llm_models.py` are US cross-region inference profile IDs for > Anthropic Claude. Override with IDs or ARNs that are enabled for inference in > your account and region. > **Google Vertex AI** > > ```bash theme={null} > export LLM_PROVIDER=vertex-ai > export VERTEX_AI_PROJECT=my-gcp-project > # Optional (defaults to us-central1): > export VERTEX_AI_LOCATION=us-central1 > # Optional overrides: > export VERTEX_AI_REASONING_MODEL=gemini-2.5-pro > export VERTEX_AI_TOOLCALL_MODEL=gemini-2.5-flash-lite > ``` > > No API key. Auth uses Google Application Default Credentials (ADC): run > `gcloud auth application-default login`, set `GOOGLE_APPLICATION_CREDENTIALS` to > a service-account key file, or use the GCE/GKE metadata server. Your principal > needs the Vertex AI User role (or equivalent) on the project. > > Always routed through LiteLLM as `vertex_ai/` (same pattern as Azure > OpenAI). The wizard lists Gemini models; you can also type any other > Vertex-supported model ID (`allow_custom_models`). > > Defaults (`gemini-2.5-pro` / `-flash` / `-flash-lite`) are the GA Gemini > generation. Gemini 3.x (`gemini-3.1-pro-preview`, `gemini-3-flash-preview`, > `gemini-3.1-flash-lite-preview`) is selectable in the wizard but is Preview-only > in Vertex Model Garden as of mid-2026 — availability and pricing can change. > **Ollama (local)** > > ```bash theme={null} > export LLM_PROVIDER=ollama > # Optional overrides: > export OLLAMA_HOST=http://localhost:11434 > export OLLAMA_MODEL=llama3.2 > ``` > > Run any local model from an [Ollama](https://ollama.com/) daemon. No API key. > OpenSRE calls Ollama’s OpenAI-compatible endpoint at `${OLLAMA_HOST}/v1`. ## CLI providers (subprocess) CLI providers run a vendor CLI instead of calling an HTTP API. OpenSRE finds the binary on `PATH` (or via an explicit env var) and reuses the existing session. CLI providers authenticate with the vendor’s own login command. OpenSRE does not start that login. **Turn timeouts:** Each ReAct turn runs one full CLI subprocess with the system prompt, tool schemas, and conversation history. The default budget is **300 seconds** (Python adds a small buffer). Override per provider when needed, for example `GEMINI_CLI_TIMEOUT_SECONDS`, `CLAUDE_CODE_TIMEOUT_SECONDS`, or `ANTIGRAVITY_CLI_TIMEOUT_SECONDS` (clamped 30–600 where supported). Open a CLI provider for install, auth, and env overrides. > **GitHub Copilot** > > ```bash theme={null} > export LLM_PROVIDER=copilot > # Authenticate the Copilot CLI separately. Either flow works — OpenSRE detects > # both. The interactive `/login` command inside `copilot` writes to the platform > # credential store; `gh auth login` is an equivalent path that Copilot CLI can > # use automatically. > copilot login # OAuth device flow; preferred CLI-first onboarding > # or: > gh auth login # logs you into the gh CLI; Copilot will use that token > # Optional overrides (all blank by default): > export COPILOT_MODEL= > export COPILOT_BIN= > # Optional auth bypass for automation (only used when no CLI login is detected): > # export COPILOT_GITHUB_TOKEN= > # export GH_TOKEN= > # export GITHUB_TOKEN= > ``` > > Requires the > [GitHub Copilot CLI](https://docs.github.com/copilot/how-tos/use-copilot-agents/use-copilot-cli) > (`npm i -g @github/copilot`). Login with `/login` inside Copilot or > `copilot login`. > > OpenSRE checks auth in this order: > > 1. `COPILOT_GITHUB_TOKEN` / `GH_TOKEN` / `GITHUB_TOKEN` in the environment > 2. [`gh auth status`](https://docs.github.com/en/copilot/how-tos/copilot-cli/set-up-copilot-cli/authenticate-copilot-cli#authenticating-with-github-cli) > when `gh` is on `PATH` (including `✓ Logged in to github.com account …`, > `- Active account: true`, or a supported `- Token:` prefix: `gho_`, > `github_pat_`, `ghu_` — not `ghp_`). For a non-`github.com` host, it runs > `gh auth status --hostname …` when `COPILOT_GH_HOST` or `GH_HOST` is set. > > It does **not** read plaintext `$COPILOT_HOME/config.json` (keychain-backed > installs may omit that file; parsing arbitrary JSON risks false positives). If > nothing matches, detection reports `logged_in=None` and the runner checks again > at invoke time. If `COPILOT_MODEL` is unset, OpenSRE omits `--model`. > OpenSRE passes the expanded prompt through stdin, keeping the command line short > enough for Windows. It also prevents Copilot from waiting for user input. **BYOK > / `COPILOT_OFFLINE`:** GitHub auth may not be required; a `None` probe can still > be fine if Copilot is set up for offline or external providers only. > **Google Gemini CLI** > > ```bash theme={null} > export LLM_PROVIDER=gemini-cli > # Authenticate the Gemini CLI separately (interactive login or API key env): > gemini > # Optional overrides (all blank by default): > export GEMINI_CLI_MODEL= > export GEMINI_CLI_BIN= > export GEMINI_CLI_TIMEOUT_SECONDS=300 # default 300; clamped 30–600 > ``` > > Requires `@google/gemini-cli` (`npm i -g @google/gemini-cli`). If > `GEMINI_CLI_MODEL` is unset, OpenSRE omits `--model` and the CLI uses its > default. If `GEMINI_CLI_BIN` is unset, the binary is resolved from `PATH` and > known install locations. > > Google is moving Gemini CLI users to Antigravity CLI (see below). OpenSRE keeps > `gemini-cli` so existing setups still work; the probe may note deprecation. > Prefer `antigravity-cli` for new Google CLI setups unless you have a paid Gemini > Code Assist licence that keeps Gemini CLI available. > **Google Antigravity CLI** > > ```bash theme={null} > export LLM_PROVIDER=antigravity-cli > # Authenticate the Antigravity CLI separately (browser OAuth on first run): > agy # interactive launch triggers Google Sign-In; token cached by OS keyring > # Stay current — 1.0.0 had OAuth hangs (fixed in 1.0.1): > agy update > # Optional overrides (all blank by default): > export ANTIGRAVITY_CLI_BIN= > export ANTIGRAVITY_CLI_TIMEOUT_SECONDS=300 # default 300; clamped 30–600; maps to `--print-timeout {N}s` > # Note: ANTIGRAVITY_CLI_MODEL is registered for forward compatibility but is > # currently a no-op (agy v1.0.2 does not expose --model in headless `-p` mode). > # Each call uses the model saved in agy's local config; switch it with > # `/models` inside the `agy` REPL. The wizard model picker is also forward- > # compatible: when Google adds `--model` for headless mode, OpenSRE can forward > # the chosen value with a small adapter change. > ``` > > Antigravity CLI (`agy`) is Google’s successor to Gemini CLI. Install with > `curl -fsSL https://antigravity.google/cli/install.sh | bash`, then run > `agy install` to set your shell `PATH`. Minimum tested version is **1.0.1** — > older builds warn and point you to `agy update`. > > **Why two Google providers?** Google’s > [transition announcement](https://developers.googleblog.com/an-important-update-transitioning-gemini-cli-to-antigravity-cli/) > says that **on 2026-06-18** Gemini CLI stops serving Pro/Ultra and free users. > Paid Gemini Code Assist licences keep Gemini CLI. OpenSRE keeps both > `gemini-cli` (deprecated alias with a probe notice) and `antigravity-cli` so > either group can run. > > As a fallback, the probe treats `GEMINI_API_KEY` / `GOOGLE_API_KEY` / > `GOOGLE_APPLICATION_CREDENTIALS` as authenticated (same idea as the Gemini CLI > adapter), so you can keep env-based auth when migrating without repeating the > browser flow. > > Invocations run as `agy -p PROMPT --print-timeout {N}s`. The adapter never > passes `--continue` / `--conversation` / `--sandbox` / > `--dangerously-skip-permissions`, so each OpenSRE call stays ephemeral. > **Cursor Agent CLI** > > ```bash theme={null} > export LLM_PROVIDER=cursor > # Authenticate the Cursor Agent CLI separately: > agent login > # Headless / CI alternative: > # export CURSOR_API_KEY=... > # Optional overrides (all blank by default): > export CURSOR_MODEL= > export CURSOR_BIN= > ``` > > Requires the Cursor Agent CLI (`agent`). Install with > `curl https://cursor.com/install -fsS | bash`. If `CURSOR_MODEL` is unset, > OpenSRE omits the model flag and the CLI uses its default. If `CURSOR_BIN` is > unset, the binary is resolved from `PATH` and known install locations. > Invocations use non-interactive `agent --print`. > **OpenCode CLI** > > ```bash theme={null} > export LLM_PROVIDER=opencode > # Authenticate OpenCode separately: > opencode auth login > # …or export a provider API key OpenCode understands > # Optional overrides (all blank by default): > export OPENCODE_MODEL= > export OPENCODE_BIN= > ``` > > Requires the OpenCode CLI. Auth is checked with `opencode auth list` after > `--version` (file credentials and/or environment provider keys). If > `OPENCODE_MODEL` is unset, OpenSRE omits the model flag. If `OPENCODE_BIN` is > unset, the binary is resolved from `PATH` and known install locations. > **Kimi Code CLI** > > ```bash theme={null} > export LLM_PROVIDER=kimi > # Authenticate Kimi separately: > kimi login > # …or for API-key installs: > export KIMI_API_KEY=... > # Optional overrides (all blank by default): > export KIMI_MODEL= > export KIMI_BIN= > ``` > > Requires the Kimi Code CLI (`kimi`). Auth is checked with `kimi login status`, > then falls back to `KIMI_API_KEY` or keys in `~/.kimi/config.toml` (or > `KIMI_SHARE_DIR`). If `KIMI_MODEL` is unset, OpenSRE omits the model flag. > Invocations use non-interactive `kimi --print` / `-p` mode. > **xAI Grok Build CLI** > > ```bash theme={null} > export LLM_PROVIDER=grok-cli > # Authenticate the Grok Build CLI separately. Either path works: > grok login # OAuth with a SuperGrok / X Premium+ account > # ...or, for headless / CI runs, use an API key instead of a browser login: > export XAI_API_KEY=xai-... # get one from the xAI console > # Optional overrides (all blank by default): > export GROK_CLI_MODEL= # e.g. grok-build; unset → CLI configured default > export GROK_CLI_BIN= # explicit path to the `grok` binary > export GROK_CLI_TIMEOUT_SECONDS=300 # default 300; clamped 30-600 > ``` > > Requires the [xAI Grok Build CLI](https://x.ai/cli) (binary: `grok`). Install > with `curl -fsSL https://x.ai/cli/install.sh | bash` (macOS/Linux) or > `irm https://x.ai/cli/install.ps1 | iex` (Windows). If `GROK_CLI_MODEL` is > unset, OpenSRE omits `-m` and the CLI uses its default. The wizard loads models > from `grok models` at onboarding time so new models appear without an OpenSRE > update. > > Invocations run as `grok -p PROMPT --output-format plain` (one non-interactive > turn). The adapter does not pass `--always-approve`: OpenSRE owns its own tools, > so Grok is used as a text responder only and does not auto-run shell commands or > file edits. > > **Auth detection:** OpenSRE runs `grok models` (\~0.5 s, no LLM call). Success > output includes "You are logged in". `XAI_API_KEY` counts as authenticated for > headless / CI even when the probe is unclear. `XAI_API_KEY` is forwarded **only** > to the Grok subprocess (not via the shared CLI env allowlist), so it cannot leak > into other CLI adapters. > > > **Not the same as `groq`.** `grok-cli` is xAI’s Grok Build CLI. `groq` is the > > Groq HTTP API (a different company). > **Pi CLI** > > ```bash theme={null} > export LLM_PROVIDER=pi > # Authenticate Pi separately. Either path works — OpenSRE detects both: > pi # then run /login for an OAuth subscription or to store a key > # …or export a provider API key Pi understands (BYOK), e.g. for Gemini: > export GEMINI_API_KEY=... > > export PI_MODEL=google/gemini-2.5-flash-lite # provider/model; unset → Pi configured default > export PI_BIN= # explicit path to the `pi` binary (optional) > ``` > > Requires the [Pi CLI](https://pi.dev) > (`npm i -g @earendil-works/pi-coding-agent`). Pi is bring-your-own-key across > about 30 providers, so `PI_MODEL` uses the `provider/model` form (for example > `google/gemini-2.5-flash-lite`, `anthropic/claude-haiku-4-5`, > `openai/gpt-4o-mini`). Run `pi --list-models` for the full list. If `PI_MODEL` > is unset, OpenSRE omits `--model` and Pi uses its default. If `PI_BIN` is unset, > the binary is resolved from `PATH` and known install locations. > > Invocations run as `pi -p PROMPT` (non-interactive print mode). > > **Auth detection:** Pi has no non-interactive auth-status command, so OpenSRE > infers auth from state: > > 1. A supported provider API key in the environment (`GEMINI_API_KEY`, > `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, …) → authenticated > 2. Otherwise, credentials in `~/.pi/agent/auth.json` (from `pi` `/login`) → > authenticated > 3. Neither → not authenticated > > Provider API keys are forwarded **only** to the Pi subprocess, never via the > shared CLI env allowlist, so they cannot leak into other CLI adapters. > > See > [`integrations/llm_cli/AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/integrations/llm_cli/AGENTS.md) > for how to add new CLI providers. ## Reasoning effort (interactive shell) In the interactive shell (`opensre` with no subcommand), `/effort` sets a **session** preference for how hard reasoning models should think before answering. It applies only when `LLM_PROVIDER` is **`openai`** (HTTP API) or **`codex`** (Codex CLI). Other providers ignore it, and the shell tells you so. | Input | Sent to the model | | - | - | | `low`, `medium`, `high`, `xhigh` | same string | | `max` | `xhigh` | Run `/effort` alone to see the current value (or `(default)` when unset). `/new` starts a new session but **keeps** `/effort` (and trust mode), like other session preferences. Outside the REPL, set a default with: ```bash theme={null} export OPENSRE_REASONING_EFFORT=high # low | medium | high | xhigh ``` On the native OpenAI SDK and Codex, GPT-6.1 Sol sends `xhigh` (extra high) when this variable and `/effort` are both unset. LiteLLM and OpenRouter do not add an effort value. Session `/effort` overrides this for interactive runs. Implementation: [`config/llm_reasoning_effort.py`](https://github.com/Tracer-Cloud/opensre/blob/main/config/llm_reasoning_effort.py). ## Provider diagnostics OpenSRE does not silently switch providers when credentials are missing. It keeps the configured provider and reports missing or stale auth before LLM work starts. * **`opensre auth` and `/auth status`** show status from environment variables, provider metadata, CLI probes, or local config — without exposing secrets. * **`opensre auth verify `** checks credentials at request time and refreshes metadata. * **`opensre config` and `opensre doctor`** report the configured provider and credential status without resolving secrets. * **Provider errors** include the configured provider name: ``` [LLM provider: openai] Missing credential for LLM provider 'openai'. Set OPENAI_API_KEY or run `opensre auth login openai`. ``` If credentials are missing, set the provider API key, run `opensre auth login `, or change `LLM_PROVIDER` to a provider you have configured. ## Switching providers at runtime OpenSRE caches LLM clients on first use. To switch providers in the same process (tests, benchmarks), call `reset_llm_clients()` from `core.llm.factory` after updating env vars. A new process picks up the new `LLM_PROVIDER` automatically. ## Where this lives in the code * Provider names and settings: [`config/llm_settings.py`](https://github.com/Tracer-Cloud/opensre/blob/main/config/llm_settings.py) (`LLMProvider`, `LLMSettings`) * Model defaults: [`config/llm_models.py`](https://github.com/Tracer-Cloud/opensre/blob/main/config/llm_models.py) * Runtime routing: [`core/llm/factory.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/factory.py) (`resolve_llm_route`, `get_llm`) and client construction in [`core/llm/client_builders.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/client_builders.py) * LiteLLM routing (when enabled): [`core/llm/transports/litellm/routing.py`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/transports/litellm/routing.py) * API provider guide: [`core/llm/AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/core/llm/AGENTS.md) * CLI provider guide: [`integrations/llm_cli/AGENTS.md`](https://github.com/Tracer-Cloud/opensre/blob/main/integrations/llm_cli/AGENTS.md) # Masking Sensitive Identifiers Source: https://opensre.com/docs/platform/masking Reversible masking of pod, cluster, and account identifiers before external LLM calls. ## Overview OpenSRE can mask sensitive infrastructure identifiers (pod names, cluster names, hostnames, account IDs, service names, IP addresses, emails) **before** sending text to an external LLM, then restore the originals in user-facing output. Teams can use external models while keeping raw identifiers inside the agent runtime. Masking is **off by default**. Turn it on with environment variables — no code changes required. ## How it works 1. When masking is enabled, call sites that send text to an external model replace identifiers with stable placeholders such as ``, ``, or ``. The placeholder → original map can be stored in session state as `masking_map`. 2. The model sees masked text, so raw identifiers are not sent in that payload. 3. Where a map is present, OpenSRE restores real identifiers in downstream state and display output. 4. Message delivery (for example Slack) can run a final unmask pass before sending, as defence in depth. The same identifier always maps to the same placeholder within one session, so reasoning about `` stays consistent. Today, masking is applied at opt-in edges (for example selected GitHub / Sentry fix tools and CLI agent-exec paths) rather than as a single central step on every LLM call. Output still unmasks when a map is present. Env policy is read when masking runs — changes apply on the next run without a restart. ## Environment variables | Variable | Default | Description | | - | - | - | | `OPENSRE_MASK_ENABLED` | `false` | Master switch. Set to `true` / `1` / `yes` / `on` to activate masking. | | `OPENSRE_MASK_KINDS` | `pod,namespace,cluster,hostname,account_id,ip_address,email,service_name` | Comma-separated list of identifier kinds to mask. Unknown kinds are ignored with a warning. Empty value uses all defaults. | | `OPENSRE_MASK_EXTRA_REGEX` | *(empty)* | Optional JSON object mapping a label → regex for custom identifiers. Example: `'{"jira_key": "\\\\b[A-Z]+-\\\\d+\\\\b"}'`. Group 1 of the regex, if present, defines the span to mask. | ## Built-in identifier kinds | Kind | Example input | Example placeholder | | - | - | - | | `pod` | `etl-worker-7d9f8b-xkp2q` | `` | | `namespace` | `kube_namespace:tracer-test` | `kube_namespace:` | | `cluster` | `eks_cluster:prod-us-east-1` | `eks_cluster:` | | `service_name` | `service:checkout-api` | `service:` | | `hostname` | `kind-control-plane`, `ip-10-0-1-23.ec2.internal` | `` | | `account_id` | `123456789012` | `` | | `ip_address` | `192.168.1.50` | `` | | `email` | `alice@example.com` | `` | ## Round-trip guarantee For the built-in detectors and extra regex patterns, `mask → unmask` restores the original payload byte-for-byte. See `tests/masking/test_integration_with_k8s_fixture.py` for a worked example against a realistic Datadog Kubernetes alert. ## Relationship to guardrails Masking is complementary to the one-way `GuardrailEvaluator`. Guardrails handle hard-block rules (credit cards, API keys) and replace matches with `[REDACTED]` irreversibly. Masking handles infrastructure identifiers reversibly so they can be restored for user-facing output. Both can be active together: guardrails apply at the LLM client layer; masking applies at the call sites that opt in. ## Example ```bash theme={null} export OPENSRE_MASK_ENABLED=true export OPENSRE_MASK_KINDS=pod,namespace,cluster,hostname opensre ask "why is the checkout pod crash-looping?" ``` When masking runs on a call path, the LLM sees masked identifiers; output paths that unmask show the original pod, namespace, and cluster names in the final answer. # Long-Term Memory Source: https://opensre.com/docs/platform/memory The agent remembers who you are and how your infrastructure works — durable facts persist across sessions as plain local files you control. ## Commands at a glance | Command | What it does | | - | - | | `/memory` | List everything the agent remembers | | `/memory show ` | Print one memory in full | | `/memory forget ` | Delete one memory | | `/memory path` | Print the folder where memories are stored | | `opensre knowledge` | List your memories and the organization's workspace knowledge | | `opensre knowledge scan` | Scan connected integrations into workspace knowledge now (`--force` to refresh fresh ones) | | `opensre knowledge audit` | Check memory quality: duplicates, one-off status notes, missing evidence, stale entries | *** ## What the agent remembers OpenSRE keeps durable knowledge across sessions so you never repeat yourself: * **Who you are** — your name, role, and how you like to work * **Your infrastructure** — cluster names, naming conventions, known-flaky services * **Your repositories** — owner/name, purpose, default branch, CI behavior, and repo-specific conventions * **Your preferences** — report formats, notification choices, defaults * **Operational learnings** — failures, their causes, and the fix that worked Every conversation sees two things: * **An index of every memory** — one line each (name and summary), with your profile and preferences first. * **The full text of the memories that match your request** — up to five, picked by matching the words of your message, plus the active repository and your connected integrations. So a memory about `acme/payments` shows up in full when you ask about the checkout deploy, even if you saved ten other notes since. Any other memory is one `memory_recall` away; ask "what do you remember about the payments repo?" and the agent searches every memory for the words you used. ### Multiple repositories OpenSRE keeps one active repository per GitHub or GitLab context while retaining facts for many repositories. Naming a different repository makes it active for unqualified follow-up requests in that session; it does not delete or overwrite the repositories used earlier. Durable details are stored as separate `repository` memories, one per repository, and remain available in later sessions. A new session defaults the active target to the current workspace checkout, while the other repository memories stay available for recall. ## Saving memories Talk normally — no special phrasing: ``` our prod cluster is eks-prod-1 ``` ``` Tracer-Cloud/opensre is our SRE assistant repo; its default branch is main ``` ``` my name is Vaibhav, I'm on the platform team ``` When the agent judges a fact useful for future sessions, it stores it (and updates an existing entry instead of duplicating). Facts the agent verified with a tool — a CI job that fails only on Windows, the fix that made a run pass — can be remembered too, and the memory file records where each fact came from: ``` source: tool evidence: gh run view 18822 failed on windows-latest; retry passed verified: true ``` A background pass also reviews the conversation every six turns and once more when you exit the shell, so knowledge lands even if the agent did not save it mid-turn. That pass saves at most five memories, and it only keeps facts you stated or a tool showed — never something only the agent claimed about your systems. The background review skips demo and onboarding runs (`/demo` and the first-run picker), and memories about demo, sample, or synthetic repositories and runs are hidden from the agent and archived. ## Viewing and deleting memories ``` /memory ``` ``` Long-term memory name type description updated ───────────────────────────────────────────────────────────────────────────────────────── prod-cluster-conventions infrastructure Prod EKS clusters: eks-prod- 2026-07-09 user-profile user Vaibhav, platform team 2026-07-09 ``` Delete one with `/memory forget prod-cluster-conventions`, or ask the agent: "forget what you know about the prod cluster". Forgetting a memory also removes the consolidated summary, which is rewritten without it in the background as you keep chatting. Summaries of conversations that had started before you forgot it are not used to write the new one. ## Upkeep When you send a message, at most once every six hours, OpenSRE tidies the store in the background: * **Archives** demo, sample, and synthetic memories, and memories neither updated nor used for 120 days. Your profile and preferences never expire. * **Merges duplicates** — two repository memories about the same `owner/repo`, or two memories of one type that say the same thing ("prefers short replies" saved under two names), become one: the most recently updated is kept, the other's notes are added to it under a "Merged from" heading (lines it already has are not repeated), and the other's file is archived. When the notes would not fit in one memory, both memories stay as they are. The background review also folds a fact it already knows into the existing memory instead of saving it again under a new name. * **Writes `memory_summary.md`** — a short summary of who you are, your preferences, your repositories, and recurring failures and fixes, shown to the agent above the index. It is skipped when no LLM is configured. Nothing is deleted. To bring an archived memory back, move its file from `.archive/` into the memory folder. ## Where memories live On a laptop, memories are plain markdown files under `~/.opensre/memory/`: | Path | What it holds | | - | - | | `.md` | One memory per file | | `MEMORY.md` | Generated list of every memory | | `memory_summary.md` | The consolidated summary | | `.archive/` | Archived memories, never shown to the agent | | `sessions/` | Short summaries of recent sessions (the last 30), read only by upkeep | | `.usage.json` | How often each memory was shown to the agent | OpenSRE creates the folder (and an empty index) the first time memory is used in a chat or via `/memory`. Files are stored **locally and unencrypted**. Override the directory with `OPENSRE_MEMORY_DIR` when you need a custom path. On a Slack or Discord silo, each member gets a private memory folder inside the organization's mounted context volume (`/users//memory/`), not a shared host folder. Telegram turns are unbound and stay **host-global** under the flat `~/.opensre/memory/` layout. The index, the matching memories, and the summary are sent to your configured LLM provider with each turn. You can open, edit, or delete the files directly at any time; the index rebuilds on the next write. ## Workspace knowledge Besides your own memory, the agent keeps knowledge about your organization that every member's conversations share: * **Organization** — the company, what it does, the team (names and titles), and the busiest channels. * **One entry per connected integration** — for Slack, every public channel and what it is for. It is written by a background scan when an integration is connected and refreshed weekly; you never write it by talking to the agent. The Slack scan reads only what any full member can: public channels, the member directory (never guests, bots, emails or phone numbers), and recent messages from public channels the agent was invited to, excluding channels shared with another organization. It never reads private channels, group DMs or DMs. Credentials in sampled messages are redacted before the summary model sees them. Only full members of the workspace see it in their conversations: Slack guests and people from another organization in a shared channel never do. A scan that cannot read the member or channel list keeps the previous knowledge and tries again within the hour. Run a scan yourself with `opensre knowledge scan`; on a hosted gateway the Knowledge page in the OpenSRE app shows both tiers, and each member sees only their own memories there. | Environment variable | Effect | | - | - | | `OPENSRE_KNOWLEDGE_DISABLED=1` | Turn workspace knowledge off (scans, prompt, Knowledge page sync) | | `OPENSRE_KNOWLEDGE_SCAN_MESSAGES=0` | Scan workspace metadata only; never read a channel message | | `OPENSRE_KNOWLEDGE_DIR` | Override the workspace knowledge directory | ## Turning memory off | Environment variable | Effect | | - | - | | `OPENSRE_MEMORY_DISABLED=1` | Disable long-term memory entirely | | `OPENSRE_MEMORY_AUTOEXTRACT_DISABLED=1` | Keep memory, but skip the background review and the upkeep | | `OPENSRE_MEMORY_GATEWAY_ENABLED=1` | Opt in on Slack/Telegram gateway (off by default — see below) | | `OPENSRE_MEMORY_DIR` | Override the memory storage directory | The background review makes one lightweight LLM call every six turns and one on exit; the upkeep makes at most one call every six hours. On shell exit the review runs to completion so durable facts are not dropped. Never share secrets or credentials expecting them to be remembered — the memory ingestion path blocks obvious tokens, passwords, private keys, and credential-shaped values, and redacts matching spans before anything is sent to the LLM. ## Gateway (Slack / Telegram / Discord / Buzz) On Slack, Discord, and Buzz, when org scope is bound, memory is **per-user scoped** — each member's facts live under their own `users//memory/` folder inside the org volume, so one user's memories cannot appear in another's prompts or channel replies. Telegram turns are unbound and stay **host-global**. Because Telegram is host-global (and to keep memory off by default on shared hosts), the **Slack and Telegram** gateway path is **off by default**; opt in with `OPENSRE_MEMORY_GATEWAY_ENABLED=1`. Discord and Buzz use the same actor-scoped storage layout when scoped; they are not gated by that opt-in flag today — use `OPENSRE_MEMORY_DISABLED=1` if you need memory off there. # Multi-instance integrations Source: https://opensre.com/docs/platform/multi-instance-integrations Configure multiple accounts, regions, or clusters for the same provider. Use multiple **named instances** when you have more than one of the same provider — for example prod and staging Grafana, two AWS accounts, or several Kubernetes clusters. Each instance can carry **tags** for filtering. Single-instance setups keep working. You do not need to change them. ## Configure instances ### Environment variable Set `_INSTANCES` to a JSON array: ```bash theme={null} export GRAFANA_INSTANCES='[ {"name":"prod", "tags":{"env":"prod"}, "endpoint":"https://prod.grafana.net", "api_key":"..."}, {"name":"staging", "tags":{"env":"staging"}, "endpoint":"https://staging.grafana.net", "api_key":"..."} ]' ``` Supported variables: | Variable | Provider | | - | - | | `GRAFANA_INSTANCES` | Grafana | | `DD_INSTANCES` | Datadog | | `GROUNDCOVER_INSTANCES` | Groundcover | | `HONEYCOMB_INSTANCES` | Honeycomb | | `CORALOGIX_INSTANCES` | Coralogix | | `AWS_INSTANCES` | AWS | | `ARGOCD_INSTANCES` | Argo CD | | `SPLUNK_INSTANCES` | Splunk | When `_INSTANCES` is set, the legacy single-instance variables for that service (for example `GRAFANA_INSTANCE_URL`, `GRAFANA_READ_TOKEN`) are ignored. If the JSON is invalid, OpenSRE logs a warning and falls back to the legacy variables. ### Store file You can also define instances in `~/.opensre/integrations.json` (schema v2): ```json theme={null} { "version": 2, "integrations": [ { "id": "grafana-prod-staging", "service": "grafana", "status": "active", "instances": [ {"name": "prod", "tags": {"env": "prod"}, "credentials": {"endpoint": "...", "api_key": "..."}}, {"name": "staging", "tags": {"env": "staging"}, "credentials": {"endpoint": "...", "api_key": "..."}} ] } ] } ``` Older v1 store files are migrated automatically on first load. If both the store and environment variables configure the same service, the **store takes precedence**. To use multi-instance env vars, remove that service from the store or add the instances in the store instead. ## Select an instance during a chat turn ### Alert hint (Grafana) Grafana alerts can include a `grafana_instance` field at the top level or in `annotations`: ```json theme={null} { "alert_source": "grafana", "grafana_instance": "staging" } ``` OpenSRE uses the matching instance when the hint is set. If it is missing or unknown, the default (first) instance is used. ### Default for other providers Other multi-instance providers use the default (first) instance unless a selector is applied in code. Extending alert-based selection beyond Grafana is planned. ## Provider support | Provider | `*_INSTANCES` env | Alert-based selection | | - | - | - | | Grafana | Yes | Yes (`grafana_instance`) | | Datadog | Yes | Default instance | | Groundcover | Yes | Default instance | | Honeycomb | Yes | Default instance | | Coralogix | Yes | Default instance | | AWS | Yes | Default instance | | Argo CD | Yes | Default instance | | Splunk | Yes | Default instance | | Others | — | Default instance | ## Limitations * Only Grafana honors an alert-provided instance hint in this release. * The CLI setup wizard is not instance-aware yet. Configure multi-instance via env vars or by editing the store JSON. * `opensre integrations verify` validates only the default instance of a multi-instance record. # Principal-scoped storage (Slack and Discord) Source: https://opensre.com/docs/platform/principal-scoped-storage How OpenSRE gives each messaging user laptop-like session context while sharing org integrations on a multi-tenant gateway. # Principal-scoped storage (Slack and Discord) For a **Slack or Discord user** on a team install, OpenSRE resolves one **principal** (the organization this deployment serves) and one **actor** (the messaging user). Team integrations and billing follow the principal. Conversation sessions follow the actor — the same private context a laptop CLI user keeps under `~/.opensre`. Local CLI and Telegram stay on the flat host home when no org scope is bound. ## Who owns the turn | Context | Principal | Actor | | - | - | - | | Slack | Organization from `ORGANIZATION_ID` | Slack user id | | Discord | Organization from `ORGANIZATION_ID` | Discord user id | | Other surfaces (CLI, Telegram, …) | Unbound — flat host home | — | With no organization configured, or a workspace outside the allowlist (`OPENSRE_SILO_TEAM_IDS` for Slack, `DISCORD_SILO_GUILD_IDS` for Discord), the turn is **refused** rather than billed or attributed to the wrong owner. Discord DMs use a sentinel guild id and skip the guild allowlist; they still require `ORGANIZATION_ID`. ## On-disk layout A laptop user keeps everything in `~/.opensre`. A Slack or Discord user gets the same private conversation context, filed under their organization. Deployed, the organization root is the mounted volume named by `OPENSRE_CONTEXT_ROOT` (for example `/workspace/memories`). The infrastructure chroots that mount to one organization through a per-org access point, so OpenSRE adds no org segment of its own: ```text theme={null} /workspace/memories/ # = organizations// integrations.json # shared by every member of the org users// # that user's own context sessions/.jsonl memory/ ``` Without the mount, a bound org principal nests under the host home so several organizations can be exercised on one machine: ```text theme={null} ~/.opensre/ gateway/… # host: pidfile, logs integrations.json # laptop / unbound: unchanged sessions/ memory/ orgs// integrations.json users// sessions/… memory/ ``` Integrations sit at the org root because credentials belong to the team: whoever connected Datadog, every member investigates with it. Sessions and memory sit under the user because conversation history is personal, the same way it is on a laptop. Personal connections are different: the OpenSRE app authorizes them against the signed-in Clerk user and returns them only to that user. They are never materialized into an organization's shared `integrations.json`. A deployed silo can also point integrations at ephemeral storage with `OPENSRE_INTEGRATIONS_STORE_PATH` (credentials re-hydrated at boot rather than kept on the shared mount). With no scope bound — a terminal run — every path resolves to the flat `~/.opensre` layout, unchanged. ## What happens on a scoped turn 1. Resolve `StorageScope` (`principal` + `actor`) at the start of the turn. 2. Bind the scope for the turn (`bound_storage_scope`). 3. Resolve paths: org home for integrations, member home for sessions. 4. Look up / create session bindings with `(platform, chat_id, principal_id, actor_id)`. 5. Consume credits against `principal.id` (the org). Alice and Bob in the same Slack thread or Discord channel get different session files and different binding rows. They share the org's integrations store. ## Other surfaces All four chat transports — Slack, Discord, Telegram, Buzz — resolve a scope and meter the turn. `principal` and `actor` stay optional on the binding store and resolver: the unbound CLI path omits them and keys bindings by empty principal/actor ids. ## Which organization a deployment serves There is no install catalog. The workspace → organization mapping is control-plane data the webapp already owns, so OpenSRE does not keep a second copy that could drift. A deployment is told which organization it serves: ```bash theme={null} export ORGANIZATION_ID=org_... export OPENSRE_SILO_TEAM_IDS=T0123 # Slack — recommended in production export DISCORD_SILO_GUILD_IDS=G0123 # Discord — recommended with real credentials ``` With the Slack or Discord allowlist set, only those workspaces/servers are served and every other one is refused. Unset, any workspace that installed the app (or any Discord server the bot is in) is served from the configured organization — convenient for dogfood, and logged as a warning because an uninvited workspace would inherit that organization's credentials. **Consequence:** one process serves one organization. A gateway fronting several workspaces would need the workspace → organization lookup back, reading the webapp rather than a local catalog. ## Upgrading an existing deployment Session bindings moved out of the host SQLite database into a per-organization `bindings.json` beside that organization's context. SQLite is gone from the live path: the context root is an NFS-backed mount, where its advisory locking is unreliable, while a write-temp-then-rename is atomic. Opening the JSON file for the first time adopts rows from any old SQLite index — including one written before scoping existed, which has neither `principal_id` nor `actor_id`; those adopt as unscoped rows and keep working for Telegram and the CLI. A deployment whose context moved onto a mounted volume also adopts from the host database. Scoped members still get a **new session on their first turn after upgrade**: an adopted row carries an empty actor id, so it no longer matches a member-scoped lookup. The transcript survives and remains readable. Session transcripts and `integrations.json` are not copied automatically. To keep continuity on a silo, copy them once: ```bash theme={null} mkdir -p ~/.opensre/orgs/$ORGANIZATION_ID/users cp ~/.opensre/integrations.json ~/.opensre/orgs/$ORGANIZATION_ID/ ``` Telegram and other unbound surfaces keep matching their migrated empty-actor rows. ## Related env vars | Env | Purpose | | - | - | | `ORGANIZATION_ID` | The organization this deployment serves. Required for Slack and Discord turns, and required to use `OPENSRE_CONTEXT_ROOT` | | `OPENSRE_CONTEXT_ROOT` | Mounted, org-scoped volume holding this organization's context (for example `/workspace/memories`). Requires `ORGANIZATION_ID` to say who owns it | | `OPENSRE_SILO_TEAM_IDS` | Comma-separated Slack team ids this deployment serves. Unset, any workspace is served (with a warning); set, every other workspace is refused | | `DISCORD_SILO_GUILD_IDS` | Comma-separated Discord server ids allowed to use this deployment's organization. Unset serves every server the bot is in (with a warning) | | `OPENSRE_INTEGRATIONS_STORE_PATH` | Optional override for the integrations store path (common on deployed silos) | ## Non-goals * CLI emulation of a Slack or Discord member * Materializing personal webapp connections into a shared organization file * Nesting `opensre.json`, sessions, or REPL history under member homes * Changing local CLI individual home layout * A database of any kind: bindings are a JSON file * A workspace → organization catalog, so one process serves one organization # Session History Source: https://opensre.com/docs/platform/sessions Browse recent sessions, resume a conversation, compact long context, or start fresh. ## Commands at a glance | Command | What it does | | - | - | | `/sessions` | Browse recent sessions and resume the highlighted one | | `/resume ` | Restore a past session so you can continue asking questions | | `/resume :` | Restore a specific branch point by entry ID prefix | | `/compact` | Summarize older context into a replayable compaction entry | | `/context` | Show what the next model call will contain and how much of the history budget is used | | `/new` | Start a fresh session file while keeping the current conversation context | *** ## Browsing past sessions — `/sessions` ``` /sessions ``` In a terminal, **↑/↓** moves the highlight, **Enter** resumes that session, and **Esc** closes the picker. It shows up to 20 sessions with relative times. The status line shows the selected session's ID, start time, duration, and turn count. The current session is marked, and the most recent previous session is highlighted by default when one exists. When output is piped, `/sessions` prints a table with IDs, start times, durations, and turn counts. Use the displayed ID with `/resume `. The current session updates its elapsed time live. *** ## Resuming a past session — `/resume` Use `/resume` with the first characters of the session ID from `/sessions`: ``` /resume 9b2e4f7a ``` From outside the REPL, pass the same ID or name substring to the CLI launcher (for example `opensre --resume 9b2e4f7a`, or `o --resume OOM` when `o` is your shell alias): ```bash theme={null} opensre --resume ``` In a TTY, bare `/resume` opens an interactive picker of recent sessions. New turns after a resume append to that same session file — `/resume` does not create a duplicate entry in `/sessions`. To resume a specific branch point, append an entry ID prefix after the session ID: ``` /resume 9b2e4f7a:abc123 ``` When the branch contains a compaction entry, replay inserts the saved summary before the kept messages so the assistant has the compacted context. OpenSRE restores the conversation, shows the past exchanges, then lets you continue: ``` resumed session 9b2e4f7a · why is CPU spiking on prod-api (14 messages) ─── conversation history ──────────────────────────────────── ❯ why is CPU spiking on prod-api? ● assistant The root cause is a connection pool leak in the orders service... ───────────────────────────────────────────────────────────── ``` **What gets restored:** | | Restored by `/resume` | | - | - | | Full conversation context (what you asked, what the assistant said) | Yes | | Infra context (service name, cluster, region learned mid-session) | Yes | | Trust mode, reasoning effort | No — these are per-session preferences | You can also search by name instead of ID: ``` /resume redis # resumes the session whose name contains "redis" /resume cpu spike # matches by name substring ``` If the prefix matches more than one session, OpenSRE asks you to be more specific. *** ## When you exit When you leave the REPL, OpenSRE prints your session ID so you can resume later: | Exit path | Behavior | | - | - | | `/exit` or `/quit` | Prints `Resume this session with:`, `/resume `, and `opensre --resume `, then goodbye | | **Ctrl+C** twice within 2 seconds | Same resume hint, then exits | | **Ctrl+D** (EOF) | Same resume hint when no dispatch is running | Example: ``` Resume this session with: /resume 3f8a1c2d-… opensre --resume 3f8a1c2d-… Goodbye! ``` Copy the `/resume …` or `opensre --resume …` line from the terminal, or find the session later with `/sessions`. *** ## Starting fresh with context — `/new` `/new` closes the current session and opens a new one, but carries the conversation forward: ``` /new ``` ``` new session started — conversation context carried forward. 14 messages in context · type to continue ``` Use `/new` after a long session to keep `/sessions` tidy without losing your place in the conversation. *** ## Compacting long sessions — `/compact` OpenSRE replays earlier turns of the session to the model in order, including the tool calls it made and what they returned (long tool output is shortened to its start and end). A follow-up such as "rerun the one that failed" can use a run ID from an earlier answer without fetching it again. When that history grows past its token budget, OpenSRE compacts it before the next turn: the model writes a summary of the older turns, and the most recent turns stay word for word with their tool results. `/compact` does the same on demand and keeps only your latest exchange: ``` /compact ``` The compaction is saved in the session file, so `/resume` restores exactly the summary and the turns that were kept. Run `/context` to see how much of the budget the history uses and what else the next model call carries. | Variable | Default | Effect | | - | - | - | | `OPENSRE_HISTORY_TOKEN_BUDGET` | `48000` | Estimated tokens of history before automatic compaction | | `OPENSRE_HISTORY_TOOL_RESULT_CHARS` | `8000` | Characters of each tool result kept in history | | `OPENSRE_LLM_COMPACTION` | `1` | Set to `0` to summarize without a model call | | `OPENSRE_STRUCTURED_HISTORY` | `1` | Set to `0` to send earlier turns as a single text block (the previous behavior) | **`/new` vs `/clear`:** | Command | What it does | | - | - | | `/clear` | Clears the terminal screen only — session and context unchanged | | `/new` | Opens a new session file; conversation context carried forward | *** ## What OpenSRE saves Each session records: * When it started and ended, and how long it ran * Every message you sent and every response from the assistant * Tool calls, tool updates, tool results, model/tool changes, labels, custom messages, and compaction summaries * Infra context discovered while working (service names, cluster names, regions) * Turn counts Sessions are stored as files under `~/.opensre/sessions/`. Each file is plain text and human-readable. ## Inspecting a turn's decisions Local session traces record the model's original closing, the reason a stop was accepted or rejected, the plan and pending questions, and the displayed and saved answers separately. These diagnostic records stay out of resumed conversation context. Known credential patterns are redacted; ordinary conversation text is retained. Run `/sessions` to find the session ID, then use `jq` in your terminal. Replace `3f8a1c2d` with the ID prefix shown for your session: ```bash theme={null} jq 'select(.type == "trace_span" and .span_kind == "decision")' \ ~/.opensre/sessions/3f8a1c2d*.jsonl ``` Look for `goal_review` and `conclusion` to understand stopping, `agent_finished` for the original closing, `response_composition` for display filtering, and `response_displayed` / `assistant_persisted` for the two outputs. Rejected closing attempts appear as `model_conclusion` records. The records include the prompt turn ID when available; `assistant_persisted.message_id` identifies the saved message. These records are added on new runs; older traces cannot recover text that was never recorded. *** ## Privacy and what is not saved * **Secrets or credentials** — known token shapes are redacted from the prompt log by default (set `OPENSRE_PROMPT_LOG_REDACT=0` to store raw text instead) * **Token usage** — tracked in `~/.opensre/prompt_log.jsonl` when prompt logging is enabled See [Interactive Shell Privacy](/docs/platform/interactive-shell-privacy) for redaction controls. *** ## Related * [Interactive Shell Privacy](/docs/platform/interactive-shell-privacy) — command history, prompt/response logging, and redaction settings * [Environment variables](/docs/install/environment-variables) — full prompt/response logging configuration # Work Management Source: https://opensre.com/docs/platform/work-management Track human tasks, choose what to do next, and schedule reminders or check-ins. ## Commands at a glance | Command | What it does | | - | - | | `/work` | List open work items | | `/work add ` | Add a durable task | | `/work edit [id-or-title]` | Choose a task and edit its current details | | `/work done <id-or-title>` | Mark one or more tasks completed | | `/work next` | Rank open tasks by priority, due date, and age | | `/work path` | Print the local work-item store path | | `opensre work ...` | Script the same workflow from the terminal | Use `/tasks` for OpenSRE runtime jobs. Use `/work` for human todos, hackathon work, and follow-ups. `/work add` does not schedule reminder delivery; use the destination-aware CLI form below or ask from a connected chat. ## Natural language You can ask normally: ```text theme={null} Add task "finish Slack reminder smoke test" for the hackathon, priority high. ``` ```text theme={null} Give me an overview of all tasks that still need to be completed. ``` ```text theme={null} What should I focus on next? ``` ```text theme={null} Mark the docs and CLI tasks as completed. ``` The agent stores these as work items and uses the same store when ranking or listing them later. ## Reminders and check-ins One-off reminders: ```bash theme={null} opensre work add "ping owners about demo blockers" \ --project hackathon --priority high \ --remind-at 2026-07-30T09:30 \ --provider slack --chat-id C0123ABCD ``` Recurring check-ins: ```bash theme={null} opensre work schedule-checkin \ --cron "0 9 * * mon-fri" \ --tz Europe/London \ --target slack:C0123ABCD \ --target telegram:-100123456 \ --project hackathon ``` Slack, Telegram, and Discord gateway turns can default reminders and recurring check-ins to the current chat. Terminal commands require either `--provider` and `--chat-id`, or one or more `--target` flags. Use repeated `--target provider:chat_id` flags to send the same reminder or check-in to more than one service. Slack webhook delivery may use `--target slack` without a channel id; Telegram, Discord, and Rocket.Chat need explicit chat/channel ids. ## Storage Work items are stored locally as JSON under `~/.opensre/work_items/` by default. Set `OPENSRE_WORK_ITEMS_DIR` to choose another folder. Existing entries from the old Slack captured-task JSONL store are imported once into the new work-item store. Recurring check-ins are stored in OpenSRE's scheduler store and run whenever the gateway scheduler or `opensre cron start` is active. # Surface boot chain Source: https://opensre.com/docs/surface-boot-chain # How surfaces boot OpenSRE can be reached in several ways — the `opensre` CLI, the interactive shell, the gateway daemon (chat channels), the gateway's web app, scheduled `cron` commands, and even a plain Python script embedding the agent. Each of them starts up by running the same shared setup — `configure_process(<profile>)` in [`bootstrap/process.py`](https://github.com/Tracer-Cloud/opensre/blob/main/bootstrap/process.py) — rather than assembling its own startup sequence. A surface's own code is responsible only for its channel and UX (rendering to a terminal, replying in chat, serving an HTTP route, parsing CLI arguments) plus any prerequisites specific to that surface; some surfaces do a little of their own setup before or after the shared steps run (see "Where each surface calls it" below). ## What the shared setup does Startup is a short checklist of setup work that has to happen in the same order every time: 1. **Load the environment** — read configuration and secrets. 2. **Start error reporting** — so failures later in startup are still captured when the selected profile owns Sentry. 3. **Register adapters** — connect the agent to the integrations and tools it can use. 4. **Register scheduled-delivery adapters** — bind outbound delivery for scheduled work. Agent runners are constructed by the scheduler host at startup rather than stored globally. 5. **Log capability warnings** — flag anything the sandbox can't do in this environment. 6. **Preload the LLM client modules** — so a long-running process doesn't end up mixing old and new versions of those modules after a later code change. Not every surface needs every step. A profile is just a name for "the subset of this checklist a given surface needs" — the CLI needs far less than the gateway daemon does. Running the shared setup with a profile performs exactly those steps, in the order above, and running it again for a surface that's already started is a harmless no-op. ## The six profiles | Profile | What it sets up | Used by | | - | - | - | | CLI | Environment. | The `opensre` command | | Gateway | Environment, error reporting, adapters, capability warnings, LLM preload. | The gateway daemon (chat channels) | | Web | Environment, error reporting, adapters. | The gateway's standalone web app | | Scheduler worker | Environment, error reporting, adapters, scheduled-delivery adapters. | The `opensre cron start` daemon | | Scheduled command | Environment, adapters, scheduled-delivery adapters. | One-off CLI commands that create, run, or dispatch scheduled work | | Embedded | Environment, adapters. | Driving the agent from another Python program | ## Where each surface calls it * **CLI** — [`surfaces/cli/startup.py`](https://github.com/Tracer-Cloud/opensre/blob/main/surfaces/cli/startup.py) runs the CLI profile first, then handles CLI-only setup: its own error reporting (tolerant of a missing dependency during `opensre update`), terminal output styling, and keyboard-interrupt handling. [`main.py`](https://github.com/Tracer-Cloud/opensre/blob/main/main.py), the module-level entry point (`python main.py`), delegates straight into this same CLI path — it does not use the embedded profile. (The embedded profile only appears in `main.py`'s docstring, as an illustrative example for someone embedding the agent in their own script.) The interactive shell runs inside this same already-started CLI process, so it doesn't start up again. * **Interactive shell's saved loops** — when a saved prompt loop needs its own background scheduler, it starts up with the scheduled-command profile in [`surfaces/interactive_shell/runtime/loop_scheduler.py`](https://github.com/Tracer-Cloud/opensre/blob/main/surfaces/interactive_shell/runtime/loop_scheduler.py) and [`surfaces/interactive_shell/command_registry/loops_cmds.py`](https://github.com/Tracer-Cloud/opensre/blob/main/surfaces/interactive_shell/command_registry/loops_cmds.py). * **Gateway daemon** — [`gateway/core/lifecycle/controller.py`](https://github.com/Tracer-Cloud/opensre/blob/main/gateway/core/lifecycle/controller.py) sets up its own logging, readiness state, and credentials first, then runs the gateway profile before connecting chat channels. When it also hosts the scheduler, the controller installs the scheduled-delivery adapters and constructs the gated scheduler runners at that later lifecycle stage. * **Gateway web app** — [`gateway/web/webapp.py`](https://github.com/Tracer-Cloud/opensre/blob/main/gateway/web/webapp.py) runs the web profile as soon as the module loads, so both the in-process gateway and a standalone web server have everything they need before handling a request. * **Scheduled/cron CLI commands** — one-off commands like `cron run` in [`surfaces/cli/commands/cron.py`](https://github.com/Tracer-Cloud/opensre/blob/main/surfaces/cli/commands/cron.py) and [`sentry_digest.py`](https://github.com/Tracer-Cloud/opensre/blob/main/surfaces/cli/commands/sentry_digest.py) run the scheduled-command profile. [`posthog_report.py`](https://github.com/Tracer-Cloud/opensre/blob/main/surfaces/cli/commands/posthog_report.py) checks that the PostHog integration is configured first, then runs the same profile. The long-running `cron start` daemon uses the scheduler-worker profile instead, since it needs its own error reporting and only needs to register task runners once, not on every run. * **Embedded Python usage** — [`bootstrap/embedded.py`](https://github.com/Tracer-Cloud/opensre/blob/main/bootstrap/embedded.py) runs the embedded profile. This is also the pattern to follow if you're driving the agent from your own Python script: run `configure_process(EMBEDDED_PROFILE)` before your first request. ## Why it works this way Keeping one shared startup checklist means every surface gets the same guarantees — nothing is skipped, and nothing runs out of order — no matter how it's launched. It also means adding or changing a startup step only needs to happen in one place. For how this fits into OpenSRE's broader package layout, see [`docs/ARCHITECTURE.md`](ARCHITECTURE.md). # Terminal metrics runbook Source: https://opensre.com/docs/terminal_metrics_runbook # Terminal Metrics Runbook This runbook defines how to interpret the interactive-terminal analytics emitted by the CLI. ## Event Groups * Interactive terminal behavior: `terminal_actions_planned`, `terminal_actions_executed`, `terminal_turn_summarized` * Agent-loop reliability: `react_turn_completed`, `$ai_generation` * Gateway reliability: `gateway_turn_started`, `gateway_turn_completed`, `gateway_turn_failed` ## Core KPIs * `terminal_action_execution_success_rate`: successful deterministic action executions * `terminal_fallback_rate`: share of turns that required LLM fallback ## Operational Guidance * High `terminal_fallback_rate` with low `planned_count` indicates missing deterministic action coverage; improve action recognizers before changing LLM prompts. * High `planned_count` but low execution success suggests command execution reliability issues (shell failures, missing dependencies, timeout thresholds). ## Data Contract Source of Truth * Event enum: `infrastructure/analytics/events.py` * Capture helpers and KPI query specs: `infrastructure/analytics/capture.py` * Event property builders: `infrastructure/analytics/event_properties.py` * Provider type constraints and coercion: `infrastructure/analytics/provider.py` # Tool placement policy Source: https://opensre.com/docs/tool-placement-policy # Tool placement policy (T-20) Where a piece of agent-callable capability lives is decided by **how many vendor integrations its domain logic touches**, not by convenience or file size. This is the decision rule referenced from [ARCHITECTURE.md](ARCHITECTURE.md#tier-3--tools-and-integrations). ## Decision rule Ask: does this tool's *purpose* depend on a specific external vendor or SaaS backend (an `integrations/<vendor>/` package)? 1. **Single vendor** → `integrations/<vendor>/tools/`. The tool only makes sense in terms of one vendor's domain (a Datadog query, a GitHub mutation, a Sentry issue lookup). This is the default, most common case — see "Adding a Tool" in `AGENTS.md`. 2. **No vendor at all** → `tools/system/`. The tool's domain purpose has nothing to do with an external vendor — local process introspection, sandboxed code execution, RAG-style guidance retrieval. An *incidental* import of a vendor client for a side concern (e.g. resolving one of several possible credential sources, or picking a notification channel for delivery) does not make a tool vendor-specific; the test is the tool's reason to exist, not every import in the file. 3. **Genuinely spans 2+ vendor integrations** → `tools/cross_vendor/`. The tool's logic itself correlates or orchestrates across multiple `integrations/<vendor>/` packages — e.g. `fix_sentry_issue` reads from `integrations.sentry` and hands the fix to `integrations.pi`. This is the narrow bucket; don't reach for it just because a tool happens to format its output for a second vendor (e.g. "Slack-ready" report text is still single-vendor logic, not cross-vendor). 4. **True surface-level (CLI + REPL) duplication, not tool logic at all** → `surfaces/shared/` (see T-21). This is a different axis entirely — it's about presentation code two *surfaces* both need, not about where an agent-callable tool's business logic lives. `tools/` also holds framework subsystems that aren't individual tools — `tools/interactive_shell` (REPL action tools), `tools/registry.py` (the tool registry itself). These stay at the top level of `tools/`; the `system` / `cross_vendor` split only applies to individual tool packages. ## Current state (as of T-19) Applied to the pre-existing top-level `tools/` packages: | Package | Placement | Why | | - | - | - | | `tools/system/python_execution_tool/` | system | Generic sandboxed Python execution; the GitHub token import is one of several optional credential sources, not the tool's purpose. | | `tools/system/sre_guidance_tool/` | system | Local knowledge-base retrieval; no vendor. | | `tools/cross_vendor/fix_sentry_issue/` | cross\_vendor | Reads a Sentry issue and hands the fix to the Pi coding agent — two `integrations/` packages in one tool's logic. | | `tools/cross_vendor/resolve_merge_conflicts/` | cross\_vendor | Reads the conflicts of a local git merge and hands them to the coding agent, then commits the merge — `integrations/git` and `integrations/coding_agent` in one tool's logic. | **Migrated to their vendor packages** — every single-vendor tool now lives under `integrations/<vendor>/tools/`, so rule 1 has no exceptions left: * `integrations/github/tools/` — `architecture_issue_tool`, `community_followup_tool`, `git_deploy_timeline_tool`, `github_cli`, `work_status_report_tool`. * `integrations/pi/tools/pi_coding_tool/` — Pi-only. * `integrations/slack/tools/slack_send_message_tool/` — Slack-only. A vendor's tool package is walked once its dotted path is listed in `INTEGRATION_TOOL_PACKAGES` (`tools/registry_discovery.py`); nested tool packages under it are discovered without further wiring. ## Registry mechanics `tools/system/` and `tools/cross_vendor/` are ordinary packages discovered by `tools/registry.py`'s top-level walk of `tools/`. Each declares a `TOOL_MODULES` tuple in its `__init__.py` (the same manifest mechanism `integrations/slack/tools/slack_send_message_tool/__init__.py` already uses for its own `tool` submodule) listing the tool packages nested one level inside it. Adding a new system or cross-vendor tool means adding its package name to the relevant `TOOL_MODULES` tuple — no registry code changes required.