Server Monitoring and Alerting Setup

Observability Design: checkout-platform (us-east-1)

Status: Draft v1. This is a complete design, but several numbers depend on answers only you have. Every assumption is tagged [A#], and the open questions are at the end. Items that change page routing or automation are marked ⚠.

0. Corrections to the Current Stack

These affect everything below, so they come first.

#	Issue	Why it matters	Fix
C1	"node_exporter + postgres_exporter + redis_exporter on all hosts" is not possible for RDS and ElastiCache. They are managed services with no shell access.	RDS and ElastiCache host metrics (CPU, storage, IOPS, replica lag, evictions) are only available from CloudWatch.	Run YACE (yet-another-cloudwatch-exporter) on a monitoring host. Run postgres_exporter and redis_exporter centrally in multi-target mode, pointed at the RDS and ElastiCache endpoints.
C2	Promtail reached EOL on 2026-03-02. It no longer receives fixes, including security fixes.	It is an unpatched agent on 39 EC2 hosts, including payments hosts.	Migrate to Grafana Alloy (`alloy convert --source-format=promtail`). This is Phase 2 unless your security policy requires it sooner.
C3	Prometheus HA replicas each carry a distinct `replica` external label.	Alertmanager sees two different alerts and pages twice.	Add `alert_relabel_configs: [{action: labeldrop, regex: replica}]` in both replicas. Point both replicas at all clustered Alertmanagers.
C4	CloudWatch metrics arrive 1–5 minutes late and are pre-aggregated.	Tight `for:` windows on CloudWatch-sourced alerts either flap or fire late.	Use `for:` of at least 5m on YACE-sourced alerts. Use app-side metrics as the primary SLI wherever possible.
C5	SQS `NumberOfMessagesSent` does not count messages that redrive moves into a DLQ.	A "DLQ growth" alert built on it will never fire.	Alert on `ApproximateNumberOfMessagesVisible` on the DLQ instead.

1. Golden Signals → Concrete Metrics

Metric names for the Go services assume standard prometheus/client_golang instrumentation [A1].

Service	Latency	Traffic	Errors	Saturation
checkout-api	`http_request_duration_seconds_bucket` (must include an `le="0.3"` bucket), ALB `TargetResponseTime`	`http_requests_total` rate by route/code/AZ/color, ALB `RequestCount`	5xx ratio (app), ALB `HTTPCode_ELB_5XX_Count` (errors that never reached a target), `TargetConnectionErrorCount`	CPU, `go_goroutines`, `go_sched_latencies_seconds`, DB pool `sql_db_wait_duration`/`in_use` vs max, ALB `RejectedConnectionCount`, healthy host count per AZ
orders-postgres	`pg_stat_statements` mean/max exec time, CW `ReadLatency`/`WriteLatency`	`xact_commit` rate, CW `ReadIOPS`/`WriteIOPS`	`pg_up`, deadlocks, rollback ratio, failed connections from app pool	`DatabaseConnections` vs `max_connections`, CPU, `FreeableMemory`, `DiskQueueDepth`, `FreeStorageSpace`, `MaximumUsedTransactionIDs` (wraparound), `ReplicaLag`, `OldestReplicationSlotLag`
redis-cache	Client-side go-redis histogram, e.g. `redis_client_cmd_duration_seconds{cmd="get"}` [A2]; server-side `redis_latency_percentiles_usec` (Redis 7 `INFO latencystats`)	`redis_commands_processed_total`, ops by command	`rejected_connections`, client-side error/timeout count, keyspace miss ratio (a quality signal)	`EngineCPUUtilization` (Redis is single-threaded, so host CPU% misleads), `DatabaseMemoryUsagePercentage`, `Evictions`, `CurrConnections`, `ReplicationLag`
payments-worker	End-to-end job latency histogram measured from SQS `SentTimestamp` to ack, e.g. `payments_job_e2e_seconds_bucket` with `le="60"` [A3]	jobs processed/s, SQS `NumberOfMessagesReceived`/`Deleted`	job failure rate, retries, DLQ `ApproximateNumberOfMessagesVisible`	`ApproximateAgeOfOldestMessage`, backlog-per-worker (`visible / healthy workers`), in-flight count vs concurrency limit

The SLIs used for alerting and error budgets are listed below. Health-check routes are excluded from all of them.

checkout availability: non-5xx / total, with a 99.9% target.
checkout latency: requests under 300ms / total, with a 99% target. This is how "p99 < 300ms" becomes an error budget.
Postgres: a synthetic probe success rate from postgres_exporter (pg_up plus a SELECT 1 on the writer) at 15s intervals, with a 99.95% target (21.6 min/month budget).
Redis: client-side GET latency under 10ms at 99%, and availability at 99.9%.
payments: jobs completed within 60s / total, with a 95% target. DLQ growth is treated as a hard invariant rather than a budget.

2. System, Process, and Log Monitoring

System (node_exporter on the 39 EC2 hosts):

CPU: utilisation and node_pressure_cpu_waiting_seconds_total (PSI). PSI is a better contention signal than CPU%.
Memory: MemAvailable, PSI memory, and OOM kills (node_vmstat_oom_kill).
Disk: fill forecast via predict_linear, inode usage, and node_disk_io_time_seconds_total saturation. c6i.large EBS bandwidth is burstable, so also watch the CloudWatch EBSIOBalance%/EBSByteBalance% for the payments-worker fleet.
Network: throughput, retransmits, conntrack usage, and ENA allowance counters via the ethtool collector (bw_in/out_allowance_exceeded, pps_allowance_exceeded, conntrack_allowance_exceeded). These counters catch silent packet drops that normal network graphs miss.
Clock: node_timex_offset_seconds and sync status. Clock skew matters for payment idempotency and token expiry.

Service/process health:

Enable node_exporter --collector.systemd --collector.systemd.unit-include="(checkout-api|payments-worker|alloy|node_exporter)\.service".
Alert on node_systemd_unit_state{state="failed"} and on restart loops, measured as changes(process_start_time_seconds[15m]).
Use Prometheus up via EC2 service discovery on tags, not static targets. Instances terminated during blue/green then disappear cleanly instead of firing TargetDown.
Run blackbox_exporter probes of the public checkout health and a synthetic checkout flow. Also run one probe from outside AWS (Grafana Synthetic Monitoring or a small external VM), because in-region probes cannot see edge, DNS, or ACM problems.

Log/event monitoring (Loki):

Labels: keep them low-cardinality: service, env, az, color, level, host. Never use request_id, user_id, or order_id as labels. Those stay in the log line and are queried with | json.
Scrubbing (⚠ must-have if logs can contain card data): add a pipeline replace stage for PAN, CVV, and Authorization headers in the agent before logs are shipped.
Loki ruler alerts (warning tier): panic/fatal rate, context deadline exceeded bursts, pq: too many connections, READONLY/MOVED storms from Redis, and payment-provider 4xx/5xx spikes.
Events: annotate Grafana from GitHub Actions for deploy start/cutover/finish. Pull RDS and ElastiCache events (failover, maintenance) and AWS Health via EventBridge → SNS → Slack. RDS failover events should also feed an inhibition label; see §4.

3. Alerting Philosophy (Avoiding Fatigue and Flapping)

Page on symptoms, notify on causes. Only SLO burn and hard invariants page, such as DLQ growth, writer down, wraparound, or monitoring being blind. High CPU on one of 18 stateless hosts is a Slack message, not a 3am call.
Use multi-window, multi-burn-rate alerts (SRE Workbook method) for every SLO. The short window provides fast reset. The long window prevents paging on blips.
Add a minimum-traffic guard to burn-rate alerts so that 1 error in 10 requests at 4am does not page.
Use for: plus keep_firing_for: (Prometheus ≥2.42) for hysteresis. An alert must be bad for N minutes to fire and good for M minutes to resolve. This removes resolve/refire flapping.
Inhibit downstream noise. An RDS failover suppresses checkout DB-connection warnings. HostDown suppresses that host's process and disk alerts. An AZ-level alert suppresses per-host alerts in that AZ. A page-level alert suppresses warnings for the same service.
Group by alertname, service, az, so that 18 hosts misbehaving produce one notification.
Every alert must carry a runbook_url and a dashboard_url. If there is no clear action, the alert does not ship.
Deploys: do not blanket-silence during blue/green. SLO alerts must stay live because a bad deploy is exactly when you need them. Only TargetDown for the draining color is silenced, and the silence is scoped by the color label.
Measure alert quality. Aim for fewer than 2 pages per on-call day. Review every page weekly. If an alert was not actionable, fix it or delete it.

4. Metric + Alert Table

Tiers used below:

P = page (PagerDuty)
W = Slack warning
T = ticket or Slack follow-up, no page

Burn-rate multipliers are applied to the error budget (the "Budget" noted in each row).

SLO / Symptom Alerts

Metric	Source	Warning	Critical	Action
checkout availability burn (5xx ratio, budget 0.1%)	app `http_requests_total`	T: 3× over 1d & 2h, or 1× over 3d & 6h	P: 14.4× over 1h & 5m (>1.44%), or 6× over 6h & 30m (>0.6%). Guard: >1 req/s	Check the deploy timeline first. Roll back the color if the burn started at cutover.
checkout latency burn (<300ms, budget 1%)	app histogram `le="0.3"`	T: 3× over 1d & 2h	P: 14.4× over 1h & 5m (>14.4% slow), or 6× over 6h & 30m	Split latency by route and AZ. Check DB pool wait and Redis latency.
ALB edge 5xx (errors before reaching a target)	YACE `HTTPCode_ELB_5XX_Count` / `RequestCount`	>0.1% for 10m	>1% for 5m → P	This means no healthy targets or a capacity problem. Check target health per AZ.
Healthy targets per AZ	YACE `HealthyHostCount` by AZ	<4 of 6 in any AZ for 5m	<2 in any AZ, or <12 total, for 3m → P	Check the ASG replacement status. Check whether an AZ event is under way.
External synthetic checkout	blackbox (external)	2 of 3 locations failing for 3m	All locations failing for 3m → P (`customer_impact=true`)	This is the customer-view truth. Start the incident.
Postgres writer availability (budget 0.05%)	postgres_exporter `pg_up` + probe query	Any probe failure for 1m → W	Down 2m or 14.4× burn → P	Check whether RDS failover is in progress. Leave RDS Multi-AZ to act and do not intervene manually.
Replication lag	YACE `ReplicaLag` + exporter	>5s for 5m (SLO line)	>30s for 5m or replica stalled → P	Check for long transactions or vacuum on the writer. Consider pulling the replica from the read pool.
Redis GET p99 (client-side)	app histogram	p99 >10ms for 10m	p99 >25ms for 5m, or 14.4× burn → P	Look for a hot shard, big keys, or a slow command (`SLOWLOG`).
Redis availability	client error ratio + `redis_up`	Node down 2m (the replica covers it)	Shard with no primary, or client errors >1% for 3m → P	Check ElastiCache failover. Confirm clients refresh the cluster topology.
payments e2e ≤60s (budget 5%)	app `payments_job_e2e_seconds`	3× burn over 6h & 30m (>15% slow)	6× over 1h & 5m (>30% slow) → P	Scale the workers. Check the downstream payment provider.
payments oldest message age	YACE `ApproximateAgeOfOldestMessage`	>60s for 5m	>300s for 5m → P	Workers are stuck or under capacity. Check crash loops and the provider.
payments DLQ	YACE DLQ `ApproximateNumberOfMessagesVisible`	—	Any increase (`delta` over 15m > 0) → P	⚠ Never auto-redrive. Inspect the payload, then fix and redrive manually after checking idempotency.

Cause / Saturation Alerts (Slack Only Unless Noted)

Metric	Source	Warning	Critical	Action
Host CPU	node_exporter	>85% for 15m (per-fleet average >70% for 30m = capacity T)	Fleet average >90% for 10m → W (the SLO alert pages if users are affected)	Scale out. Profile with pprof.
Host memory / OOM	node_exporter	MemAvailable <10% for 10m	Any OOM kill → W; repeated on the fleet → P	Check for a leak (goroutines, heap). Recycle the instance.
Disk fill	node_exporter `predict_linear`	Full within 24h (6h lookback)	Full within 4h, or >95% → P only on monitoring hosts, W elsewhere	logrotate/journald vacuum. Grow the gp3 volume.
ENA allowance exceeded	node_exporter ethtool	Rate >0 for 10m	Sustained, with retransmits climbing → W	Upsize the instance or spread the load.
Clock offset	node_exporter timex	>50ms for 10m	>500ms or unsynchronised → W	Restart chrony. Check the Amazon Time Sync endpoint.
systemd unit failed / restart loop	node_exporter systemd	>2 restarts in 15m	Unit failed on >20% of the fleet → P	Probably a bad deploy. Roll back.
RDS connections	YACE / exporter	>70% of `max_connections`	>90% → P	Check for a pool leak or an idle-in-transaction storm.
RDS storage	YACE `FreeStorageSpace`	<20% or full within 7d	<10% or full within 24h → P	Enable or verify storage autoscaling. Check WAL retained by slots.
Txn ID wraparound	YACE `MaximumUsedTransactionIDs`	>1.0B	>1.5B → P	Find what blocks autovacuum: long transactions or abandoned slots.
Long transactions / idle-in-txn	postgres_exporter	>5 min	>30 min → W	Kill with `pg_terminate_backend` (manual only).
Replication slot lag	YACE `OldestReplicationSlotLag`	>5 GB	>20 GB → P	Fix or drop the abandoned slot.
Redis memory	YACE `DatabaseMemoryUsagePercentage`	>75%	>90%, or evictions >0 on a no-eviction key space → W/P per [A4]	Check TTLs and big keys. Add a shard (online resharding).
Redis engine CPU	YACE `EngineCPUUtilization`	>70% for 15m	>90% for 10m → W	Look for a hot shard or O(N) commands.
Monitoring health	`Watchdog`, `prometheus_notifications_dropped_total`, rule eval failures, `alertmanager_cluster_members`	Rule failures >0 or one replica down	Dead-man's switch missed → P (via the external heartbeat)	Check the monitoring hosts. Until fixed, assume you are blind.
Log error spike	Loki ruler	panic/fatal >0, or error rate 5× baseline	— (the SLO alerts page)	Correlate the logs with the deploy annotation.

5. Routing and Escalation

Tier	Destination	Timing
page	PagerDuty `checkout-platform-oncall`, with a copy to Slack	`group_wait` 30s, `repeat_interval` 1h. Primary on-call. Escalates to secondary at 10 min if unacked. Proposed third level at 20 min: SRE manager/eng lead [A5].
warning	Slack `#alerts-checkout-platform` (`send_resolved: true`)	`group_interval` 5m, `repeat_interval` 4h
ticket	Slack thread, plus a Jira/Linear issue if you have one [A6]	Re-notify every 24h. Reviewed in the weekly ops meeting.
digest / capacity	Email to sre-team@company.com	Daily 08:00 ET. ⚠ Alertmanager cannot produce digests, and scheduled reports are Grafana Enterprise/Cloud only. On OSS, use a GitHub Actions cron that queries the Prometheus API, renders an HTML summary, and emails it.
status page	status.company.com via PagerDuty	⚠ Recommendation: trigger it only from alerts labelled `customer_impact="true"` (external synthetic, checkout fast burn, ALB edge 5xx), and prefer incident-commander confirmation over auto-posting. Auto-posting from infra alerts publishes false outages.

Error-budget freeze automation:

Recording rules track the rolling 30d SLI per service.
A GitHub Actions deploy gate queries Prometheus before non-critical deploys. It blocks when the breach count for the window exceeds 2, and deploys labelled critical-fix bypass it. The definition of "breach" is needed; see Q5.

6. Dashboards

Dashboard	Key panels	Question it answers
1. SLO Overview (the home page)	Per service: 30d SLI, budget remaining %, current burn rate (1h/6h), breach count in 30d, deploy-freeze status	"Are customers OK and can we ship?"
2. checkout-api RED	RPS by route/code, latency heatmap + p50/p95/p99 vs the 300ms line, error ratio by AZ and by blue/green color, DB pool wait, Redis client latency	"Is it the app, one AZ, or the new color?"
3. Deploy Comparison	Blue vs green side by side: error ratio, p99, CPU, restarts; deploy annotations	"Is this deploy safe to finish or roll back?"
4. AZ Health	Healthy targets, error and latency per AZ, ENA drops per AZ	"Is this a zonal failure?"
5. Postgres	Connections vs max, TPS, top-10 `pg_stat_statements` by total time, replica lag, locks/deadlocks, xid age, storage forecast, IOPS vs provisioned	"Is the DB the bottleneck, and which query?"
6. Redis	Per-shard ops, engine CPU, memory, evictions, hit ratio, client-side p99 per shard	"Is there a hot shard or memory pressure?"
7. Payments Pipeline	Queue depth, oldest age vs 60s, throughput in/out, e2e histogram, retries, DLQ depth, backlog per worker	"Are payments flowing on time?"
8. Fleet USE	Per-role CPU/mem/disk/net utilisation, saturation (PSI), errors (OOM, drops)	"Which hosts are unhealthy?"
9. Logs	Loki error rate by service/level, top error signatures (`pattern` parser), with a link-out from dashboard 2	"What is the error message?"
10. Capacity Planning	90d trends, peak RPS per instance vs the load-tested ceiling, `predict_linear` for RDS storage, Redis memory, and connections, headroom per tier, weeks-to-exhaustion	"When do we need to scale or buy?"
11. Meta-monitoring	Scrape durations, active series, TSDB size, rule eval lag, notifications sent/failed, Loki ingestion/drops	"Can we trust the monitoring?"

Retention and sizing (rough estimate; validate against prometheus_tsdb_head_series):

Series: about 150–250k active series in total (node_exporter ~1.5k/host × 39, apps and exporters the rest).
Prometheus ingest: a 15s scrape gives roughly 1.5–2.5 GB/day per replica.
Prometheus retention: set --storage.tsdb.retention.time=45d, which gives full 30d SLO windows plus margin, on a gp3 volume of about 150 GB.
Long-term (Phase 3): Thanos sidecar → S3 with downsampling, keeping 5m resolution for 13 months for year-over-year capacity work.
Loki: 30d in S3 as the default. Payments logs may need longer retention for PCI/audit [A7]; use per-stream retention for that.

7. Automated Responses (Safe Only)

Automation	Trigger	Why it is safe	Guardrail
systemd `Restart=on-failure`, `RestartSec=5`, `StartLimitBurst=5/10min`	Process crash	Stateless, and the crash is already happening	The restart-loop alert catches cases where restarting does not help
ASG health checks with `HealthCheckType=ELB` [A8]	ALB target unhealthy	Replaces the instance instead of patching it	Grace period of at least 300s. Alert if replacements exceed 3 in 30m.
checkout-api target tracking	ALB `RequestCountPerTarget` (from the load-tested ceiling) or CPU 60%	Scale-out is low risk	Min 6 per AZ. Max capped for cost. Scale-in cooldown of at least 10m.
payments-worker step scaling	SQS backlog-per-instance (AWS-recommended pattern)	Consumers are idempotent [A9]	Max capped to protect the payment provider rate limits and RDS connections
Blue/green auto-rollback	Fast-burn or 5xx ratio on the new color within 15 min of cutover	The old color is still running, so the rollback is only an ALB weight shift	Only during the bake window. Posts to Slack. Requires a human to re-promote.
Disk hygiene	logrotate + `journald SystemMaxUse`	Preventive	None beyond the settings themselves

Explicitly not automated:

RDS or Redis restarts and failovers (the managed Multi-AZ mechanism already handles them)
DLQ redrive (risk of double-charging)
Killing Postgres sessions
Scale-in based on errors
Auto-rollback outside the deploy bake window

8. Rollout Plan

Phase 1: Must-Have (Weeks 1–2)

This phase provides correct paging on customer pain.

Fixes C1, C3, C5:
- EC2 service discovery by tags
- YACE for ALB/RDS/ElastiCache/SQS
- Central multi-target postgres_exporter and redis_exporter
- Alertmanager HA deduplication
Instrumentation checks:
- The checkout histogram has an le="0.3" bucket.
- payments-worker records the e2e histogram and the go-redis client histogram.
SLO recording rules plus burn-rate pages for all 4 services, and the DLQ and oldest-age pages.
Alertmanager routing (PagerDuty, Slack), inhibitions, and grouping. Add runbook_url to every page.
Dead-man's switch: Watchdog → external heartbeat (e.g. healthchecks.io, which pages PagerDuty when the heartbeat is missed). Add the external synthetic probe.
PAN/secret scrubbing in the log pipeline.
Dashboards 1, 2, 7. systemd restart policies.

Phase 2: Should-Have (Weeks 3–6)

This phase adds diagnosis, capacity, and safer deploys.

Every cause/saturation alert in the Slack tier. Loki ruler alerts.
Dashboards 3–6, 8–11. Deploy annotations from GitHub Actions.
Blue/green auto-rollback. Error-budget deploy gate. Daily digest email job.
Migrate Promtail → Alloy (C2). Tune ENA, PSI, and pg_stat_statements coverage.
Weekly alert-review ritual. Page-count tracking.

Phase 3: Nice-to-Have (Quarter)

This phase adds long-term data and depth.

Thanos + S3 for 13-month retention and capacity forecasting.
OpenTelemetry tracing → Tempo, with exemplars linking latency spikes to traces.
Evaluate an upgrade to Prometheus 3.x (native histograms, UTF-8 labels).
Game days: AZ evacuation, RDS failover, Redis shard failover. Verify the alerts and inhibitions behave as designed.
Cost dashboards (per-service spend vs traffic).

9. Config Snippet (Prometheus Rules + Alertmanager Routing)

# ============================================================
# FILE: /etc/prometheus/rules/checkout-slo.yml
# Assumes: http_requests_total{job,code,route} and
#          http_request_duration_seconds_bucket with le="0.3"
# Validate: promtool check rules checkout-slo.yml
# ============================================================
groups:
  - name: checkout-api-sli
    interval: 30s
    rules:
      # --- Availability SLI: error ratio over multiple windows ---
      - record: slo:checkout_errors:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz",code=~"5.."}[5m]))
          / sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz"}[5m]))
      - record: slo:checkout_errors:ratio_rate30m
        expr: |
          sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz",code=~"5.."}[30m]))
          / sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz"}[30m]))
      - record: slo:checkout_errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz",code=~"5.."}[1h]))
          / sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz"}[1h]))
      - record: slo:checkout_errors:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz",code=~"5.."}[6h]))
          / sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz"}[6h]))
      # (Add 2h / 1d / 3d the same way for the ticket-tier slow-burn alert.)

      # --- Latency SLI: fraction of requests slower than 300ms ---
      - record: slo:checkout_slow:ratio_rate5m
        expr: |
          1 - (
            sum(rate(http_request_duration_seconds_bucket{job="checkout-api",route!~"/healthz|/readyz",le="0.3"}[5m]))
            / sum(rate(http_request_duration_seconds_count{job="checkout-api",route!~"/healthz|/readyz"}[5m]))
          )
      - record: slo:checkout_slow:ratio_rate1h
        expr: |
          1 - (
            sum(rate(http_request_duration_seconds_bucket{job="checkout-api",route!~"/healthz|/readyz",le="0.3"}[1h]))
            / sum(rate(http_request_duration_seconds_count{job="checkout-api",route!~"/healthz|/readyz"}[1h]))
          )
      - record: slo:checkout_traffic:rate5m
        expr: sum(rate(http_requests_total{job="checkout-api",route!~"/healthz|/readyz"}[5m]))

  - name: checkout-api-slo-alerts
    rules:
      - alert: CheckoutAvailabilityBudgetBurnFast
        # 14.4x = 2% of monthly budget in 1h; 6x = 5% in 6h. Budget = 0.001.
        expr: |
          (
            (slo:checkout_errors:ratio_rate1h > (14.4 * 0.001)
              and slo:checkout_errors:ratio_rate5m > (14.4 * 0.001))
            or
            (slo:checkout_errors:ratio_rate6h > (6 * 0.001)
              and slo:checkout_errors:ratio_rate30m > (6 * 0.001))
          )
          and slo:checkout_traffic:rate5m > 1        # low-traffic guard
        for: 2m
        keep_firing_for: 10m                         # hysteresis: no flapping
        labels:
          severity: page
          service: checkout-api
          slo: availability
          customer_impact: "true"
        annotations:
          summary: "checkout-api burning error budget fast ({{ $value | humanizePercentage }} errors)"
          runbook_url: "https://runbooks.company.com/checkout-api/availability-burn"
          dashboard_url: "https://grafana.company.com/d/checkout-red"

      - alert: CheckoutLatencyBudgetBurnFast
        # Budget = 1% of requests may exceed 300ms.
        expr: |
          slo:checkout_slow:ratio_rate1h > (14.4 * 0.01)
          and slo:checkout_slow:ratio_rate5m > (14.4 * 0.01)
          and slo:checkout_traffic:rate5m > 1
        for: 2m
        keep_firing_for: 10m
        labels:
          severity: page
          service: checkout-api
          slo: latency
        annotations:
          summary: "checkout-api: {{ $value | humanizePercentage }} of requests >300ms"
          runbook_url: "https://runbooks.company.com/checkout-api/latency-burn"

  - name: payments-invariants
    rules:
      - alert: PaymentsDLQGrowing
        # Uses VISIBLE count: NumberOfMessagesSent does NOT count redrive-to-DLQ.
        expr: |
          delta(aws_sqs_approximate_number_of_messages_visible_maximum{dimension_QueueName="payments-dlq"}[15m]) > 0
        for: 0m
        keep_firing_for: 15m
        labels:
          severity: page
          service: payments-worker
        annotations:
          summary: "payments DLQ grew by {{ $value }} messages. DO NOT auto-redrive."
          runbook_url: "https://runbooks.company.com/payments/dlq"

      - alert: PaymentsQueueStale
        expr: aws_sqs_approximate_age_of_oldest_message_maximum{dimension_QueueName="payments"} > 300
        for: 5m                                       # CloudWatch lag: keep >=5m
        labels:
          severity: page
          service: payments-worker
        annotations:
          summary: "Oldest payment job is {{ $value | humanizeDuration }} old (SLO 60s)"
          runbook_url: "https://runbooks.company.com/payments/queue-age"

  - name: meta
    rules:
      - alert: Watchdog                               # always firing -> external heartbeat
        expr: vector(1)
        labels:
          severity: heartbeat

# ============================================================
# FILE: /etc/alertmanager/alertmanager.yml   (Alertmanager 0.27)
# Validate: amtool check-config alertmanager.yml
# REQUIRED in prometheus.yml on BOTH replicas:
#   alerting:
#     alert_relabel_configs: [{action: labeldrop, regex: replica}]
# ============================================================
route:
  receiver: slack-warnings
  group_by: ['alertname', 'service', 'az']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="heartbeat"']
      receiver: deadmans-switch
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m
    - matchers: ['severity="page"']
      receiver: pagerduty-oncall
      repeat_interval: 1h
      continue: true                  # also mirror pages into Slack
    - matchers: ['severity=~"page|warning"']
      receiver: slack-warnings
    - matchers: ['severity="ticket"']
      receiver: slack-tickets
      repeat_interval: 24h

inhibit_rules:
  # A page for a service suppresses that service's warnings.
  - source_matchers: ['severity="page"']
    target_matchers: ['severity=~"warning|ticket"']
    equal: ['service']
  # Host down suppresses everything else from that host.
  - source_matchers: ['alertname="HostDown"']
    target_matchers: ['alertname!="HostDown"']
    equal: ['instance']
  # AZ-wide degradation suppresses per-host noise in that AZ.
  - source_matchers: ['alertname="AZDegraded"']
    target_matchers: ['severity=~"warning|ticket"']
    equal: ['az']
  # RDS failover in progress suppresses app-side DB connection warnings.
  - source_matchers: ['alertname="RDSFailoverInProgress"']
    target_matchers: ['alertname=~"CheckoutDBPool.*|PostgresConnectionErrors"']

receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-checkout-platform-oncall
        severity: critical
        description: '{{ .CommonAnnotations.summary }}'
        details:
          runbook: '{{ .CommonAnnotations.runbook_url }}'
          dashboard: '{{ .CommonAnnotations.dashboard_url }}'
          customer_impact: '{{ .CommonLabels.customer_impact }}'
          firing: '{{ .Alerts.Firing | len }}'
  - name: slack-warnings
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: '#alerts-checkout-platform'
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} ({{ .CommonLabels.service }})'
        text: '{{ .CommonAnnotations.summary }}\nRunbook: {{ .CommonAnnotations.runbook_url }}'
  - name: slack-tickets
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: '#alerts-checkout-platform'
        title: '[TICKET] {{ .CommonLabels.alertname }}'
        text: '{{ .CommonAnnotations.summary }}'
  - name: deadmans-switch
    webhook_configs:
      - url_file: /etc/alertmanager/secrets/heartbeat-url   # e.g. healthchecks.io ping -> pages PD if missed
        send_resolved: false

Notes on the snippet:

Metric names from YACE depend on your YACE version and config. Confirm them with curl :5000/metrics | grep sqs.
Run amtool check-config against your exact 0.27 build to confirm the *_file secret fields.
Keep the webhook URLs and routing keys in the secrets files, managed via Ansible Vault, never in Git.

10. Common Pitfalls and How This Design Avoids Them

Pitfall	Avoided by
Duplicate pages from HA Prometheus	`labeldrop replica` plus a clustered Alertmanager (C3)
Paging on causes (CPU, one host) → alert fatigue	Only SLO burn and hard invariants page. Causes go to Slack.
Static thresholds on SLOs, which are either noisy or late	Multi-window burn rates with a low-traffic guard
Flapping alerts	`for:` plus `keep_firing_for:`, and longer windows on CloudWatch data
Blue/green churn firing `TargetDown`	EC2 SD by tag, and silences scoped to the draining `color` only
Silencing everything during deploys	SLO alerts stay live, and auto-rollback depends on them
Host CPU% used for Redis	`EngineCPUUtilization` plus client-side latency
DLQ alert on `NumberOfMessagesSent` never fires	Alert on DLQ visible count delta (C5)
Prometheus quietly dead = silent on-call	External dead-man's switch plus meta-monitoring dashboard
Loki cardinality explosion	Label whitelist. IDs stay in the log body.
Card data in logs (PCI)	Scrubbing stage in the agent before logs are shipped
30d SLO computed with 15d retention	45d retention, with recording rules keeping queries cheap
Unpatched EOL agent	Promtail → Alloy migration (C2)
Auto-remediation making incidents worse	Automation limited to stateless or reversible actions. No DLQ redrive, no DB actions.

11. Open Questions (Please Answer Before I Finalize)

[A8] ASGs: Are checkout-api and payments-worker in Auto Scaling Groups, or fixed Terraform instances? This determines whether §7's self-healing and scaling apply.
RDS topology: Is it a Multi-AZ DB instance with 2 separate read replicas, or a Multi-AZ DB cluster (1 writer + 2 readable standbys = 3 instances)? Lag metrics, failover behaviour, and the availability SLI differ between the two.
Redis topology: 6 nodes as "3 shards × 2 replicas" would actually be 9 nodes. Is it 3 shards × (1 primary + 1 replica)? [A4] Also, is Redis purely a cache (evictions acceptable), or does it hold state like sessions or locks?
[A1][A2][A3] Instrumentation: Do the services already expose Prometheus metrics? What are the histogram buckets? Does the worker have access to SQS SentTimestamp for e2e latency?
Freeze policy: What counts as an "SLO breach"? For example: budget exhausted, a fast-burn page, or a monthly SLI miss. This determines the deploy-gate query.
[A5] Escalation: Is there a third escalation level after the secondary? Should warnings be business-hours-only in Slack?
[A6] Which ticketing system should the ticket tier use (Jira, Linear, other)?
[A7] Compliance: Are you in PCI-DSS scope? Is there a required log retention (for example 1 year for audit)?
Status page: Keep auto-posting, or switch to incident-commander-confirmed posting?
Traffic: What are the typical and peak RPS for checkout, and the jobs/min for payments? This lets me validate the low-traffic guards and burn-rate sensitivity.
[A9] Are payments-worker jobs idempotent, so that safe retries and scale-out without double-processing are possible?
Grafana edition: OSS or Enterprise/Cloud? This decides whether native scheduled reports or the cron-based digest is used.

Once you answer these, I'll finalize the thresholds, deliver the complete rule files (all services and all windows), the YACE and exporter configs, and the Alloy migration config.

Example run

Inputs used