AWS S3 - OpenSRE Documentation

Overview

OpenSRE uses AWS S3 to read object metadata and content when an alert points at data in a bucket — inspecting the files a pipeline produced, sampling an object to spot a schema change, pulling an audit payload, or confirming a run finished by checking for a completion marker. All S3 access is read-only and routed through the shared AWS client, so the integration cannot mutate or delete your objects.

Prerequisites

AWS credentials with S3 read access, configured per the AWS integration
An S3 bucket you want OpenSRE to read from
IAM permissions for the two S3 actions listed below

Setup

S3 has no credentials of its own — it uses the same AWS credentials as the rest of the AWS integration. The S3 tools build their client from the ambient AWS credential chain: environment keys, a shared credentials profile, or an attached instance/task role — plus the region.

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_REGION=us-east-1

AWS_ROLE_ARN is assumed only by opensre integrations verify aws — the S3 tools do not assume it. The credentials above must have S3 read access themselves.

The bucket, key, and prefix OpenSRE reads are not environment variables — they come from the alert’s resolved sources. Each S3 tool turns on for the planner only when the sources carry the fields it needs:

Source key	Fields	Turns on
`s3`	`bucket`, `key`, `prefix`	`list_s3_objects` (needs `bucket`), `inspect_s3_object` / `get_s3_object` (need `bucket` + `key`), `check_s3_marker` (needs `bucket` + `prefix`)
`s3_audit`	`bucket`, `key`	`get_s3_object` — fetch an audit payload referenced by an alert
`s3_processed`	`bucket` (`prefix` optional)	`check_s3_marker` — check a pipeline’s output location

Credentials

IAM permissions

The integration only needs two read-only S3 actions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": "*"
    }
  ]
}

Attach this to the same IAM role or user the S3 tools actually use — the ambient credential chain (env keys or instance/task role), not only a role that the verifier assumes. If you already use the AWS managed ReadOnlyAccess policy, both actions are covered.

Tools

Tool	IAM action	What it returns
`list_s3_objects`	`s3:ListBucket`	Object keys under an optional prefix, each with size, last-modified, ETag, and storage class (up to 100 keys).
`inspect_s3_object`	`s3:GetObject`	Object metadata — size, content type, ETag, version ID, user metadata — plus a decoded text sample of the first few KB.
`get_s3_object`	`s3:GetObject`	The full object body (up to 1 MB) as text, for audit payloads, config, or manifest files.
`check_s3_marker`	`s3:ListBucket`	Whether a completion marker is present under a prefix, plus the file list — a signal that a pipeline run finished.

Use cases

Listing what a pipeline wrote to a bucket, or confirming an expected file is present
Sampling an object to detect a schema or format change in upstream input data
Reading an audit payload or manifest to trace an external vendor interaction
Confirming a batch job completed by checking for its _SUCCESS marker

Verify

S3 shares the AWS integration’s verification. Verify credentials and region with:

opensre integrations verify aws

Expected output:

Service: aws
Status: passed
Detail: Connected to AWS STS via static-creds in us-east-1; caller identity account=123456789012 arn=arn:aws:iam::123456789012:user/opensre.

This confirms the credentials S3 rides on. It does not test bucket-level access — that is exercised the first time a tool runs against a bucket.

Troubleshooting

Symptom	Fix
S3 tools never run	The alert’s sources have no `s3` entry, or it lacks the required field — object-level tools need both `bucket` and `key`, not just `bucket`.
AccessDenied on `GetObject` / `ListBucket`	Attach the IAM policy above to the credentials the S3 tools actually use — the ambient credential chain (env keys or instance/task role), not only a role that the verifier assumes.
`check_s3_marker` reports no marker when the run finished	It only counts files whose key contains `_SUCCESS` or `marker` (case-insensitive). A completion file named anything else won’t be detected.
A binary object returns metadata but no readable text	The object isn’t UTF-8 (for example Parquet or gzip); you still get size, type, and ETag, but there is no text sample to read.

Security

All S3 access is read-only — the integration cannot mutate or delete objects.
Prefer least-privilege IAM with only s3:ListBucket and s3:GetObject on the buckets you want OpenSRE to read.
Store AWS credentials in .env or your secret manager — not in source control.

​Overview

​Prerequisites

​Setup

​Credentials

​IAM permissions

​Tools

​Use cases

​Verify

​Troubleshooting

​Security

Overview

Prerequisites

Setup

Credentials

IAM permissions

Tools

Use cases

Verify

Troubleshooting

Security