Multi-instance integrations

Overview

Real deployments have multiple clusters, regions, teams, and accounts for the same provider — a prod and staging Grafana, two AWS accounts, three Kubernetes clusters. OpenSRE’s integration model now supports multiple named instances per provider with tags for filtering, while remaining fully backward-compatible with existing single-instance configurations.

Configuring multiple instances

There are two ways to configure multi-instance integrations.

1. Environment variable (JSON array)

Set <SERVICE>_INSTANCES to a JSON array. Each entry can use either a nested credentials object or a flat shape.

export GRAFANA_INSTANCES='[
  {"name":"prod", "tags":{"env":"prod"}, "endpoint":"https://prod.grafana.net", "api_key":"..."},
  {"name":"staging", "tags":{"env":"staging"}, "endpoint":"https://staging.grafana.net", "api_key":"..."}
]'

Supported env vars: GRAFANA_INSTANCES, DD_INSTANCES, HONEYCOMB_INSTANCES, CORALOGIX_INSTANCES, AWS_INSTANCES, ARGOCD_INSTANCES. When <SERVICE>_INSTANCES is set, the legacy single-instance vars for that service (e.g. GRAFANA_INSTANCE_URL, GRAFANA_READ_TOKEN) are ignored. If the JSON is invalid the loader logs a warning and falls back to the legacy vars.

2. Store file (`~/.opensre/integrations.json`)

The store uses a v2 schema with multiple instances per record:

{
  "version": 2,
  "integrations": [
    {
      "id": "grafana-prod-staging",
      "service": "grafana",
      "status": "active",
      "instances": [
        {"name": "prod", "tags": {"env": "prod"}, "credentials": {"endpoint": "...", "api_key": "..."}},
        {"name": "staging", "tags": {"env": "staging"}, "credentials": {"endpoint": "...", "api_key": "..."}}
      ]
    }
  ]
}

v1 stores are migrated automatically on first load — no manual action needed.

Selecting a specific instance during an investigation

By alert hint (Grafana, shipping now)

Alerts can carry a grafana_instance hint — either at the top level of the raw alert payload, or inside annotations:

{
  "alert_source": "grafana",
  "grafana_instance": "staging",
  ...
}

When set, OpenSRE selects the matching instance. If the hint is absent or unknown, the default (first) instance is used.

Programmatic selectors

from integrations.selectors import (
    get_default_instance,
    get_instance_by_name,
    get_instances_by_tag,
    select_instance,
)

# Flat default (backward-compat shape)
default = get_default_instance(resolved_integrations, "grafana")

# By name
prod = get_instance_by_name(resolved_integrations, "grafana", "prod")

# By tag
prod_cluster = get_instances_by_tag(resolved_integrations, "grafana", "env", "prod")

# Either
picked = select_instance(resolved_integrations, "grafana", name="prod")
picked = select_instance(resolved_integrations, "grafana", tags={"env": "staging"})

Backward compatibility

v1 store files are migrated on load; version bumped from 1 to 2; structural fields (id, service, status) preserved at the top level
Legacy env vars (GRAFANA_INSTANCE_URL, DD_API_KEY, etc.) continue to work unchanged
resolved_integrations[<service>] still returns the flat config dict of the default (first) instance — no existing consumer code changes
A sibling key _all_<service>_instances is published only when multiple instances exist (or an instance has a non-default name)
Existing single-instance tests continue to pass without modification

Current end-to-end provider support

Provider	`<SERVICE>_INSTANCES` env	Classifier multi-instance	`detect_sources` selection
Grafana	✅	✅	✅ (via `grafana_instance` hint)
Datadog	✅	✅	Default instance only
AWS	✅	✅	Default instance only
Honeycomb	✅	✅	Default instance only
Coralogix	✅	✅	Default instance only
Argo CD	✅	✅	Default instance only
Others	—	Default instance only	Default instance only

Providers without end-to-end selection fall back to the default (first) instance — identical behavior to before this feature.

Known limitations

Only Grafana honors an alert-provided grafana_instance hint in this release; extending per-provider selection is a follow-up.
Operators must configure multi-instance via env vars or direct JSON edit; the CLI wizard is not yet instance-aware.
verify_integrations currently validates only the default instance of a multi-instance record.
When both the store and env vars configure the same service, the store still wins (existing precedence). To use multi-instance env vars, either remove the store entry for that service or add instances via the store directly.

Overview

LLM providers

Observability and incidents

Cloud, code, and collaboration

Messaging

Data and workflow systems

Overview

Configuring multiple instances

1. Environment variable (JSON array)

2. Store file (`~/.opensre/integrations.json`)

Selecting a specific instance during an investigation

By alert hint (Grafana, shipping now)

Programmatic selectors

Backward compatibility

Current end-to-end provider support

Known limitations

​Overview

​Configuring multiple instances

​1. Environment variable (JSON array)

​2. Store file (~/.opensre/integrations.json)

​Selecting a specific instance during an investigation

​By alert hint (Grafana, shipping now)

​Programmatic selectors

​Backward compatibility

​Current end-to-end provider support

​Known limitations

Overview

Configuring multiple instances

1. Environment variable (JSON array)

2. Store file (`~/.opensre/integrations.json`)

Selecting a specific instance during an investigation

By alert hint (Grafana, shipping now)

Programmatic selectors

Backward compatibility

Current end-to-end provider support

Known limitations