> ## Documentation Index
> Fetch the complete documentation index at: https://opensre.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Alertmanager

> Connect Alertmanager so OpenSRE can surface firing alerts and active silences

## Overview

OpenSRE queries Alertmanager for firing, silenced, and inhibited alerts so the agent
can correlate the alert you ask about with other concurrent signals.

## Prerequisites

* Alertmanager v0.20+ reachable from the machine running OpenSRE
* The Alertmanager URL (for example `http://alertmanager.monitoring.svc:9093`)
* Credentials if your instance uses authentication (bearer token **or** basic auth — not both)

## Setup

### Option 1: Interactive CLI

```bash theme={null}
opensre integrations setup alertmanager
```

The wizard asks for:

1. **Alertmanager URL** — base URL of your Alertmanager instance
2. **Authentication method** — one of:
   * **None** — unauthenticated instances on an internal network
   * **Bearer token** — reverse proxy that accepts a token
   * **Basic auth** — username and password

### Option 2: Environment variables

```bash theme={null}
ALERTMANAGER_URL=http://alertmanager.monitoring.svc:9093

# Bearer token auth (optional)
ALERTMANAGER_BEARER_TOKEN=your-token

# Basic auth (optional)
ALERTMANAGER_USERNAME=admin
ALERTMANAGER_PASSWORD=secret
```

| Variable                    | Default | Description                                          |
| --------------------------- | ------- | ---------------------------------------------------- |
| `ALERTMANAGER_URL`          | —       | **Required.** Base URL of your Alertmanager instance |
| `ALERTMANAGER_BEARER_TOKEN` | —       | Bearer token for reverse-proxy auth                  |
| `ALERTMANAGER_USERNAME`     | —       | Basic auth username                                  |
| `ALERTMANAGER_PASSWORD`     | —       | Basic auth password                                  |

Use **at most one** auth method. Setting both a bearer token and basic auth credentials
is rejected.

### Option 3: Persistent store

```json theme={null}
{
  "version": 1,
  "integrations": [
    {
      "id": "alertmanager-prod",
      "service": "alertmanager",
      "status": "active",
      "credentials": {
        "base_url": "http://alertmanager.monitoring.svc:9093",
        "bearer_token": "",
        "username": "",
        "password": ""
      }
    }
  ]
}
```

## Credentials

Alertmanager itself often has no built-in API key UI. Credentials usually come from the
reverse proxy or basic auth in front of the instance. Prefer a **read-only** token or
user — OpenSRE only reads alerts and silences.

## Tools

| Tool                    | What it does                                                        |
| ----------------------- | ------------------------------------------------------------------- |
| `alertmanager_alerts`   | Queries `/api/v2/alerts` for firing, silenced, and inhibited alerts |
| `alertmanager_silences` | Queries `/api/v2/silences` for active silences                      |

### `alertmanager_alerts`

Typical uses:

* Discover other alerts firing at the same time as the triggering alert
* Check whether the triggering alert is already silenced or inhibited
* Understand blast radius from active alert labels
* Correlate Prometheus alerts (OOM, latency, error-rate) into one timeline

Name the alert in your question — the agent filters on the `alertname` label so
results stay scoped to the incident.

### `alertmanager_silences`

Typical uses:

* See whether a noisy alert was silenced on purpose
* Surface maintenance windows that overlap the incident
* Avoid false root-cause conclusions from suppressed alerts

## Verify

```bash theme={null}
opensre integrations verify alertmanager
```

Expected output:

```
Service: alertmanager
Status:  passed
Detail:  Connected to Alertmanager at http://alertmanager.monitoring.svc:9093; cluster status: ready.
```

### Local Docker verification

Start a disposable Alertmanager without external credentials:

```bash theme={null}
docker run --rm --detach --name opensre-alertmanager \
  --publish 127.0.0.1:9093:9093 \
  prom/alertmanager:v0.28.1 || exit 1
for attempt in $(seq 1 30); do
  curl --fail --silent http://127.0.0.1:9093/-/ready && break
  if [ "$attempt" -eq 30 ]; then
    docker logs opensre-alertmanager
    exit 1
  fi
  sleep 1
done
```

Seed one firing alert and one active silence so both tools return
real data:

```bash theme={null}
curl --fail --silent --request POST \
  --header "Content-Type: application/json" \
  --data '[{"labels":{"alertname":"OpenSREHighLatency","service":"checkout","severity":"critical"},"annotations":{"summary":"Checkout p99 latency exceeds 2 seconds"}}]' \
  http://127.0.0.1:9093/api/v2/alerts

starts_at=$(uv run python -c 'from datetime import UTC, datetime; print(datetime.now(UTC).isoformat())')
ends_at=$(uv run python -c 'from datetime import UTC, datetime, timedelta; print((datetime.now(UTC) + timedelta(hours=1)).isoformat())')
curl --fail --silent --request POST \
  --header "Content-Type: application/json" \
  --data "{\"matchers\":[{\"name\":\"alertname\",\"value\":\"OpenSREMaintenance\",\"isRegex\":false}],\"startsAt\":\"${starts_at}\",\"endsAt\":\"${ends_at}\",\"createdBy\":\"opensre-local-verification\",\"comment\":\"Disposable local verification silence\"}" \
  http://127.0.0.1:9093/api/v2/silences
```

From a source checkout, run `uv run opensre integrations setup alertmanager`.
Enter `http://127.0.0.1:9093` as the URL and select **None** for authentication,
then verify the connection:

```bash theme={null}
uv run opensre integrations verify alertmanager
```

The verifier should report cluster status `ready`. Exercise both tools through
an agent turn:

```bash theme={null}
uv run opensre ask \
  --allowed-tool alertmanager_alerts \
  --allowed-tool alertmanager_silences \
  "Report the firing Alertmanager alert and the active silence."
```

The result should include the `OpenSREHighLatency` alert and the
`OpenSREMaintenance` silence. Stop the disposable instance when finished:

```bash theme={null}
docker stop opensre-alertmanager
```

This unauthenticated instance binds only to loopback and is for local
verification only.

## Troubleshooting

| Symptom                   | Fix                                                                     |
| ------------------------- | ----------------------------------------------------------------------- |
| **Status: missing**       | Set `ALERTMANAGER_URL` or run `opensre integrations setup alertmanager` |
| **Connection refused**    | Confirm the URL is reachable from this host; check firewall rules       |
| **401 Unauthorized**      | Supply a bearer token **or** basic auth credentials (not both)          |
| **Both auth methods set** | Clear either `ALERTMANAGER_BEARER_TOKEN` or the basic-auth pair         |
| **SSL error**             | Trust the CA cert, or use an `http://` URL for internal instances       |
| **Empty alert list**      | Normal when nothing is firing — check the Alertmanager UI               |

## Security

* Prefer a **read-only** reverse-proxy token — OpenSRE never writes to Alertmanager.
* Store credentials in `.env` or your secret manager — not in source control.
* For internal Kubernetes deployments, prefer network isolation over exposing credentials.