Kafka - OpenSRE Documentation

Overview

OpenSRE queries Kafka to retrieve topic partition health, consumer group lag, and broker metadata — helping diagnose lag spikes, under-replicated partitions, and consumer group failures during incidents. All operations are read-only.

Prerequisites

Apache Kafka cluster (2.x or later)
Network access from the OpenSRE environment to the Kafka brokers
confluent_kafka available in the OpenSRE environment (installed with the project)

Setup

There is no dedicated opensre integrations setup kafka wizard today. Configure via environment variables or the persistent store, then verify.

Option 1: Environment variables

KAFKA_BOOTSTRAP_SERVERS=broker1:9092,broker2:9092
KAFKA_SECURITY_PROTOCOL=PLAINTEXT    # or SASL_SSL, SSL, SASL_PLAINTEXT
KAFKA_SASL_MECHANISM=PLAIN           # optional, for SASL
KAFKA_SASL_USERNAME=your-username    # optional
KAFKA_SASL_PASSWORD=your-password    # optional

Variable	Default	Description
`KAFKA_BOOTSTRAP_SERVERS`	—	Required. Comma-separated broker addresses
`KAFKA_SECURITY_PROTOCOL`	`PLAINTEXT`	`PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL`
`KAFKA_SASL_MECHANISM`	—	`PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, or `AWS_MSK_IAM` where supported
`KAFKA_SASL_USERNAME`	—	SASL username
`KAFKA_SASL_PASSWORD`	—	SASL password

Option 2: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "kafka-prod",
      "service": "kafka",
      "status": "active",
      "credentials": {
        "bootstrap_servers": "broker1:9092,broker2:9092",
        "security_protocol": "SASL_SSL",
        "sasl_mechanism": "PLAIN",
        "sasl_username": "your-username",
        "sasl_password": "your-password"
      }
    }
  ]
}

Credentials

Common configurations

MSK (AWS Managed Kafka) with IAM:

KAFKA_BOOTSTRAP_SERVERS=b-1.your-cluster.kafka.us-east-1.amazonaws.com:9098
KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=AWS_MSK_IAM

Confluent Cloud:

KAFKA_BOOTSTRAP_SERVERS=pkc-xxxxx.us-east-1.aws.confluent.cloud:9092
KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=PLAIN
KAFKA_SASL_USERNAME=your-api-key
KAFKA_SASL_PASSWORD=your-api-secret

Investigation tools

Tool	What it does
`get_kafka_topic_health`	Partition metadata: leader, replicas, ISR, under-replicated partitions
`get_kafka_consumer_group_lag`	Committed offsets vs high watermarks per partition for a consumer group

Verify

opensre integrations verify kafka

Expected output:

Service: kafka
Status: passed
Detail: Connected to Kafka cluster with 3 broker(s) and 42 topic(s)

Verify uses a live AdminClient.list_topics call.

Troubleshooting

Symptom	Fix
Connection timeout	Check broker hostnames, ports, and firewall rules
Authentication failed	Verify SASL credentials and mechanism match the broker config
SSL handshake error	Ensure the broker’s TLS certificate is trusted or configure a CA cert
Leader not available	Broker may be restarting — wait and retry

Security

Use SASL_SSL in production — avoid PLAINTEXT outside local development.
Create a dedicated Kafka user with Describe permissions only — no produce or consume.
Store credentials in .env or the integration store, not in source code.

​Overview

​Prerequisites

​Setup

​Option 1: Environment variables

​Option 2: Persistent store

​Credentials

​Common configurations

​Investigation tools

​Verify

​Troubleshooting

​Security