> ## Documentation Index
> Fetch the complete documentation index at: https://opensre.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Better Stack Telemetry

> Connect Better Stack so OpenSRE can pull log evidence from your Telemetry sources

## Overview

OpenSRE uses Better Stack's ClickHouse SQL Query API to read log evidence. It queries the configured source via `remote(<source>_logs)` for recent rows and `s3Cluster(primary, <source>_s3)` for historical rows, bounded by the incident time window.

## Prerequisites

* A Better Stack account with at least one **Telemetry source** collecting logs
* A **ClickHouse HTTP client** credential pair (username + password) generated from the dashboard
* The **region-specific query endpoint** for your workspace (for example `https://eu-nbg-2-connect.betterstackdata.com`)
* Network access from the OpenSRE environment to that endpoint over HTTPS

## Setup

### Option 1: Interactive CLI

```bash theme={null}
opensre integrations setup betterstack
```

You will be prompted for the query endpoint, username, password, and an optional comma-separated list of source IDs (planner hint).

### Option 2: Environment variables

Add to your `.env`:

```bash theme={null}
BETTERSTACK_QUERY_ENDPOINT=https://eu-nbg-2-connect.betterstackdata.com
BETTERSTACK_USERNAME=<clickhouse-http-username>
BETTERSTACK_PASSWORD=<clickhouse-http-password>
BETTERSTACK_SOURCES=t123456_myapp,t123456_gateway
```

| Variable                     | Default   | Description                                                                                                                                                                                                                                        |
| ---------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BETTERSTACK_QUERY_ENDPOINT` | —         | **Required.** Region-specific SQL API host (for example `https://eu-nbg-2-connect.betterstackdata.com`)                                                                                                                                            |
| `BETTERSTACK_USERNAME`       | —         | **Required.** Username from **Connect ClickHouse HTTP client**                                                                                                                                                                                     |
| `BETTERSTACK_PASSWORD`       | —         | **Required.** Password from the same dashboard flow                                                                                                                                                                                                |
| `BETTERSTACK_SOURCES`        | *(empty)* | Optional comma-separated list of **base source IDs** (for example `t123456_myapp`). The integration appends `_logs` / `_s3` internally. If omitted, the planner must derive the source from alert metadata (via a `betterstack_source` annotation) |

### Option 3: Persistent store

Credentials are persisted to `~/.opensre/integrations.json` with `0o600` permissions:

```json theme={null}
{
  "version": 1,
  "integrations": [
    {
      "id": "betterstack-prod",
      "service": "betterstack",
      "status": "active",
      "credentials": {
        "query_endpoint": "https://eu-nbg-2-connect.betterstackdata.com",
        "username": "<clickhouse-http-username>",
        "password": "<clickhouse-http-password>",
        "sources": ["t123456_myapp"]
      }
    }
  ]
}
```

## Credentials

From the Better Stack dashboard:

1. Open **Telemetry** and pick the source you want OpenSRE to query.
2. In the source sidebar, open **Integrations** → **Connect ClickHouse HTTP client**.
3. Copy the generated **username**, **password**, and **query endpoint**. The endpoint's subdomain encodes the region (for example `eu-nbg-2-connect`, `us-connect`).
4. The **base source ID** (for example `t123456_myapp`) is shown above the integration panel and is the value to supply for `BETTERSTACK_SOURCES`. Use the base name only — OpenSRE appends `_logs` and `_s3` internally.

## Tools

OpenSRE exposes one tool against a Better Stack source:

### `query_betterstack_logs`

Returns `(dt, raw)` pairs by UNIONing:

* Recent rows from `remote(<source>_logs)`
* Historical rows from `s3Cluster(primary, <source>_s3) WHERE _row_type = 1`

Arguments the planner supplies:

| Argument          | Default                          | Description                                                                             |
| ----------------- | -------------------------------- | --------------------------------------------------------------------------------------- |
| `source`          | first configured `sources` entry | Base identifier (for example `t123456_myapp`)                                           |
| `since` / `until` | —                                | ISO-8601 timestamps that bound the `dt` column; typically derived from the alert window |
| `limit`           | `500`                            | Row cap                                                                                 |

All queries run with `FORMAT JSONEachRow` and `output_format_pretty_row_numbers=0`. Source names are validated against `^[A-Za-z0-9_]+$` to prevent identifier injection.

## Verify

```bash theme={null}
opensre integrations verify betterstack
```

Expected output:

```
SERVICE       SOURCE      STATUS   DETAIL
betterstack   store       passed   Connected to Better Stack SQL API at https://eu-nbg-2-connect.betterstackdata.com
```

The verify step issues a cheap probe (`SELECT 1 FORMAT JSONEachRow`) against the configured endpoint using the stored credentials.

## Troubleshooting

| Symptom                                        | Fix                                                                                                                                                                                    |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Authentication failed (401)**                | Regenerate credentials via the dashboard's **Connect ClickHouse HTTP client** flow; confirm `BETTERSTACK_USERNAME` / `BETTERSTACK_PASSWORD` match exactly.                             |
| **Endpoint not found / DNS error**             | The region subdomain is wrong. Copy the endpoint directly from the dashboard (for example `eu-nbg-2-connect`, `us-connect`, `eu-fsn-3-connect`).                                       |
| **`invalid source name` error on query**       | The `source` argument contains characters outside `[A-Za-z0-9_]`. Use the base ID shown in the dashboard — no dashes, no quotes, no whitespace.                                        |
| **Empty result set for a known-busy source**   | Check that the alert window (`since` / `until`) actually overlaps with the source's `dt` range; historical rows older than recent retention live in `s3Cluster(primary, <source>_s3)`. |
| **`planner did not configure a source` error** | Either set `BETTERSTACK_SOURCES`, or ensure the alert payload includes a `betterstack_source` annotation.                                                                              |

## Security

* Use a **dedicated ClickHouse HTTP client credential** for OpenSRE — not your personal dashboard login.
* Keep the credential pair out of source control — use `.env` or the persistent store (`~/.opensre/integrations.json`).
* The integration is **read-only**: OpenSRE only issues `SELECT` statements against `remote(...)` and `s3Cluster(...)` table functions.
* Source identifiers are allowlisted against `^[A-Za-z0-9_]+$` before being interpolated into SQL, preventing identifier-injection attacks from the alert payload.
* Rotate credentials periodically via the Better Stack dashboard.