Kubernetes Resource Diagnostics

Verdict: The pods are being OOMKilled because the in-process cache (LocalCacheManager) grows with request volume. At 3x traffic it no longer fits in a 220 MiB heap. The container's 256 MiB limit also leaves only about 36 MiB for all non-heap JVM memory. The liveness failures and the 502/504s are downstream symptoms of the process dying.

Confidence: High for the kill mechanism. Medium for why only 4 of 6 pods crash. Unknown whether the cache is bounded at all, and that decides whether adding memory fixes the problem or only delays it. The outputs I need to close these gaps are listed at the end.

1. Triage

Failing object: Deployment payments-api → ReplicaSet payments-api-7d4f9c8b6 → 4 of 6 pods in CrashLoopBackOff.

I ruled out each failure class against the evidence:

Failure class: crash loop caused by exceeding the memory limit.

2. Root cause

2a. The kill mechanism is proven

• Reason: OOMKilled, Exit Code: 137 (128 + SIGKILL 9). containerd records this reason from the kernel cgroup OOM event, so it is authoritative for the most recent termination. A liveness-probe kill would show Reason: Error, not OOMKilled.
• Started 15:03:40 → Finished 15:05:12 is about 92 s of lifetime, matching the "restart every ~90 s" symptom.
• Grafana shows memory climbing to the ceiling, then dropping to zero, which is consistent with this.
• kubectl top: the crashing pods sit at 248–253 Mi against a 256 Mi limit.

2b. Why memory hits the ceiling

Two problems stack on top of each other.

(i) The cache grows with traffic. From --previous logs:

Handling batch of 340 concurrent requests
ERROR c.a.p.cache.LocalCacheManager - Cache size exceeded soft limit, entries=48210
java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.resize(...)
    at c.a.p.cache.LocalCacheManager.put(LocalCacheManager.java:88)
    at c.a.p.controller.CheckoutController.processOrder(CheckoutController.java:112)

• A HashMap-backed cache is written on every checkout (processOrder → put).
• The "soft limit" only logs; it does not evict. The cache keeps growing past it.
• HashMap.resize allocates a new backing array about 2x the old one while the old one is still live. That is a sharp spike at exactly the moment the heap is fullest.
• More traffic means more entries and more concurrent in-flight requests (340 here), so more live heap. This is the link to the 3x marketing traffic, and it explains why there was no deploy.

(ii) The heap is sized too close to the container limit.

• -Xmx220m caps heap only. The JVM also needs metaspace (often 80–150 MiB for Spring Boot plus Hibernate), code cache, thread stacks (Tomcat threads × up to 1 MiB each), GC data structures and direct buffers.
• 256Mi − 220Mi ≈ 36Mi for all of that is not enough. At normal traffic the heap wasn't full, so total RSS stayed under 256 Mi. The healthy pod at 198 Mi shows the baseline.
• When the heap fills, the JVM throws OutOfMemoryError. A heap OOM does not kill the JVM by itself. Next, GC churn, HashMap.resize allocation and thread activity push total RSS past the cgroup limit, and the kernel kills it. That is the Killed line right after the stack trace.

2c. Symptoms, not causes

• Liveness probe failed: connection refused at 15:05:12 is the same second as the OOMKill. Nothing is listening because the process is dead.
• The earlier liveness kill (20m ago) points to a secondary issue. Either the JVM was thrashing in GC and not serving /healthz within timeout=1s, or startup took longer than 10s + 3×10s = 40s with no startupProbe (plausible for Spring Boot capped at 500m CPU). I need startup timing to tell which (see §3).
• 502/504 at the ingress: readiness keeps passing until the JVM dies, so nginx routes requests to a pod that is about to be killed. In-flight requests are then cut off (502) or hang (504).

2d. What the evidence does not explain yet

1. Why 2 pods survive. Under a purely load-proportional model, the survivors take more traffic while the others crash, yet they sit stable at around 198 Mi with 0 restarts. Hypotheses:

   • (a) Cold-start concurrency spike. A freshly started JVM has no JIT warmup, so requests queue up (340 concurrent), live heap spikes, and the pod dies before warming up. Warm pods process fast and hold less in flight.
   • (b) Uneven load balancing from nginx affinity or EWMA settings.
   • (c) The survivors are also growing, just more slowly.

   h2r6w is missing from your kubectl top output, so I can't compare both survivors.

2. Whether the cache has any eviction (TTL or size). The pods previously ran for days, so something probably bounds the cache: steady-state size ≈ rate × TTL. If so, the fix is enough memory for 3x+ steady state plus fewer requests per pod. If the cache is truly unbounded, more memory only moves the crash from ~90 s to some minutes later.

3. Why every pod is 42m old (started about 14:22) when the last rollout was 5 days ago. Something recreated all 6 pods at the start of the incident: a rollout restart, manual pod deletion, node replacement (auto-repair or autoscaler), or an HPA/GitOps action. The same pod-template hash doesn't rule this out without rollout history.

4. Garbled evidence. The Deployment status: block (Yaves:, Not) and one event (festored container) look corrupted by copy/paste. Please re-run the commands for clean output.

3. Diagnosis commands

3.1 Check that every crashing pod is OOMKilled

kubectl get pods -n payments -l app=payments-api -o custom-columns=\
'POD:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount,LAST_REASON:.status.containerStatuses[0].lastState.terminated.reason,EXIT:.status.containerStatuses[0].lastState.terminated.exitCode,FINISHED:.status.containerStatuses[0].lastState.terminated.finishedAt'

• Healthy: LAST_REASON <none>, RESTARTS 0.
• Broken (confirms the hypothesis): OOMKilled / 137 on all 4 crashers.
• If some show Error / 137, those were liveness kills, which strengthens the startup/probe secondary issue.

3.2 Check that the kernel did the kill (GKE node-problem-detector emits node events)

kubectl get events -A --field-selector reason=OOMKilling --sort-by='.lastTimestamp' | tail -20

• Broken: Memory cgroup out of memory: Killed process <pid> (java) on nodes a1, b2 and c3.
• Healthy: no events.

3.3 Check the heap OOM pattern and startup time on every pod

for p in $(kubectl get pods -n payments -l app=payments-api -o name); do
  echo "=== $p (previous)"
  kubectl logs -n payments "$p" --previous 2>/dev/null \
    | grep -E 'OutOfMemoryError|Cache size exceeded|Started PaymentsApiApplication in' | tail -5
  echo "=== $p (current)"
  kubectl logs -n payments "$p" 2>/dev/null \
    | grep -E 'Cache size exceeded|Started PaymentsApiApplication in' | tail -3
done

• Broken: crashers show OutOfMemoryError preceded by entries=4xxxx.
• Startup: Started PaymentsApiApplication in N seconds. If N is above about 35–40 s, the liveness probe can kill pods during startup.
• Survivors: are they logging Cache size exceeded too, and at what entry count?

3.4 Check what the JVM is actually running with (on a survivor)

JAVA_OPTS only takes effect if the image entrypoint passes it to java. If it doesn't, the JVM defaults to 25% of 256 Mi ≈ 64 MiB heap.

POD=payments-api-7d4f9c8b6-7mvqz
kubectl exec -n payments $POD -- sh -c 'cat /proc/1/cmdline | tr "\0" " "; echo'
kubectl exec -n payments $POD -- jcmd 1 VM.flags | tr ' ' '\n' | grep -E 'MaxHeapSize|MaxRAM|UseG1GC|UseSerialGC|ExitOnOutOfMemory'
kubectl exec -n payments $POD -- jcmd 1 GC.heap_info

• Expected: -XX:MaxHeapSize=230686720 (220 MiB).
• If jcmd isn't found (distroless or JRE image), tell me and I'll give you an ephemeral debug-container approach.
• If PID 1 isn't java (shell wrapper), run jcmd -l first.

3.5 Check cgroup memory from inside the container (GKE 1.28 is usually cgroup v2)

kubectl exec -n payments $POD -- sh -c '
  if [ -f /sys/fs/cgroup/memory.max ]; then
    echo "max=$(cat /sys/fs/cgroup/memory.max) current=$(cat /sys/fs/cgroup/memory.current)"
    cat /sys/fs/cgroup/memory.events
  else
    echo "max=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes) usage=$(cat /sys/fs/cgroup/memory/memory.usage_in_bytes)"
  fi'

• Healthy: max=268435456, current well below it, oom_kill 0.
• Warning: current within about 10% of max means this survivor is next.

3.6 Check what's in the heap (on a survivor)

⚠️ GC.class_histogram forces a full GC pause, and this pod is one of only 2 serving traffic. Run it after mitigation (§4), or accept a brief latency spike.

kubectl exec -n payments $POD -- jcmd 1 GC.class_histogram | head -25

• Confirms the hypothesis: HashMap$Node, byte[]/String, and your cache value class dominate the top rows.

3.7 Things that change the fix path (HPA, GitOps, rollout history, why pods are 42m old)

kubectl get hpa,pdb -n payments
kubectl rollout history deployment/payments-api -n payments
kubectl get rs -n payments -l app=payments-api -o wide
kubectl get deployment payments-api -n payments -o jsonpath='{.metadata.annotations}{"\n"}{.metadata.labels}{"\n"}{.spec.strategy}{"\n"}{.spec.template.metadata.annotations}{"\n"}{.status}{"\n"}'
kubectl get nodes -o wide   # check node AGE: were nodes replaced around 14:20?

• If you see an argocd/flux/managed-by label or annotation: a kubectl patch will be reverted. Make the change in Git instead.
• If an HPA exists: kubectl scale will be overridden. Patch the HPA's minReplicas instead.

3.8 Check capacity for bigger pods and per-pod load distribution

kubectl top nodes
kubectl top pods -n payments   # re-run; include h2r6w
for n in gke-prod-e2std4-pool-a1 gke-prod-e2std4-pool-b2 gke-prod-e2std4-pool-c3; do
  echo "== $n"; kubectl describe node $n | sed -n '/Allocated resources/,/Events/p'
done
kubectl get ingress -n payments -o yaml | grep -E 'nginx.ingress.kubernetes.io/(affinity|load-balance|upstream-hash-by|proxy-next-upstream)'

3.9 Metrics (PromQL, or the equivalents in Cloud Monitoring)

# Memory headroom per pod (broken: → 1.0 before every restart)
max by (pod) (container_memory_working_set_bytes{namespace="payments",container="payments-api"})
  / on (pod) max by (pod) (kube_pod_container_resource_limits{namespace="payments",container="payments-api",resource="memory"})

# CPU throttling (anything above ~25% slows JVM startup and GC)
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="payments",container="payments-api"}[5m]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="payments",container="payments-api"}[5m]))

4. Fix

Step 0 — Pre-flight checks (non-destructive; required)

kubectl get hpa -n payments
kubectl get deployment payments-api -n payments -o yaml | grep -iE 'argocd|fluxcd|managed-by'

• If GitOps manages this Deployment: apply the YAML from §5 through your repo. Don't patch live.
• If an HPA exists: skip the replicas change in step 3 and patch minReplicas instead.

Step 1 — Raise the memory limit and let the heap scale with it

This triggers a rolling update that also replaces the 2 healthy pods. The defaults are maxSurge=25% (2 pods) and maxUnavailable=25% (1 pod). The controller removes the unavailable old pods first, and new pods only take traffic after passing readiness. Expect a brief capacity dip, not an outage. It needs about 2 extra pods' worth of free node capacity; the cluster autoscaler will add nodes if needed.

kubectl patch deployment payments-api -n payments --type=strategic -p '
spec:
  template:
    spec:
      containers:
      - name: payments-api
        env:
        - name: JAVA_OPTS
          value: "-XX:MaxRAMPercentage=60.0 -XX:InitialRAMPercentage=60.0 -XX:+ExitOnOutOfMemoryError"
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
          limits:
            cpu: "1"
            memory: "1Gi"
        startupProbe:
          httpGet:
            path: /healthz
            port: 8080
          periodSeconds: 5
          failureThreshold: 24
          timeoutSeconds: 3
        livenessProbe:
          timeoutSeconds: 3
        readinessProbe:
          timeoutSeconds: 3
'

What each change does:

• 1 Gi limit with 60% heap: about 614 MiB heap (roughly 2.8x) plus about 410 MiB for metaspace, threads and code cache.
• MaxRAMPercentage replaces the hard-coded -Xmx. Don't keep -Xmx220m, because -Xmx overrides the percentage.
• ExitOnOutOfMemoryError: on a heap OOM the JVM exits immediately and cleanly instead of limping until the kernel kills it mid-request.
• CPU 1/1: keeps QoS Guaranteed and speeds JVM startup and GC. CPU was already about 40% of the old request.
• startupProbe: allows up to 120 s for startup before liveness takes over, so slow starts aren't killed.
• Probe timeouts 1 s → 3 s: avoids false failures during GC pauses.

Step 2 — Watch the rollout

kubectl rollout status deployment/payments-api -n payments --timeout=10m
kubectl get pods -n payments -l app=payments-api -w

If new pods are stuck in Pending, check kubectl describe pod <pod> -n payments and the autoscaler events. Nodes should arrive within a few minutes.

Step 3 — Scale out so each pod handles less traffic and holds a smaller cache (only if no HPA)

kubectl scale deployment/payments-api -n payments --replicas=10
# If an HPA exists instead:
# kubectl patch hpa <name> -n payments -p '{"spec":{"minReplicas":10}}'

If the cache is TTL-bounded, per-pod cache size is proportional to per-pod request rate, so this lever works directly.

Step 4 — Verify

# Flags actually applied (expect MaxHeapSize ≈ 644245094)
NEWPOD=$(kubectl get pods -n payments -l app=payments-api -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n payments $NEWPOD -- jcmd 1 VM.flags | tr ' ' '\n' | grep -E 'MaxHeapSize|ExitOnOutOfMemory'

# Restart counts stay at 0 for 15–30 min at peak traffic
kubectl get pods -n payments -l app=payments-api

# Memory should plateau, not climb monotonically
kubectl top pods -n payments

• Success: memory plateaus (the cache is bounded; you're fixed) and the ingress stops returning 502/504.
• Still climbing linearly: the cache is unbounded. You've bought time, not a fix. Escalate to the app team immediately for the cache change in §5.

Rollback, if new pods fail

kubectl rollout undo deployment/payments-api -n payments

⚠️ Step 5 — Reconcile payments (business-critical; do not skip)

Pods were SIGKILLed mid-processOrder, and HHH000010: On release of batch it still contained JDBC statements shows DB writes were in flight. The DB transaction should roll back, but calls to external payment providers may have succeeded. Have the payments team reconcile orders and charges between 14:20 UTC and the time of recovery against the PSP, looking for charged-but-not-recorded and duplicate charges from client retries.

5. Prevention

5.1 Corrected manifest (steady state)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
spec:
  replicas: 6            # remove if an HPA owns replicas
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0  # never drop below desired capacity during rollouts
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: payments-api
      containers:
      - name: payments-api
        image: gcr.io/acme-prod/payments-api:2.14.3
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_OPTS
          value: >-
            -XX:MaxRAMPercentage=60.0
            -XX:InitialRAMPercentage=60.0
            -XX:+ExitOnOutOfMemoryError
            -XX:+HeapDumpOnOutOfMemoryError
            -XX:HeapDumpPath=/dumps
        - name: SPRING_PROFILES_ACTIVE
          value: "prod"
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"   # validate via load test (see 5.3)
          limits:
            cpu: "1"
            memory: "1Gi"
        volumeMounts:
        - name: dumps
          mountPath: /dumps
        startupProbe:
          httpGet: { path: /healthz, port: 8080 }
          periodSeconds: 5
          failureThreshold: 24
          timeoutSeconds: 3
        livenessProbe:
          httpGet: { path: /healthz, port: 8080 }   # must NOT check DB or downstreams
          periodSeconds: 10
          failureThreshold: 3
          timeoutSeconds: 3
        readinessProbe:
          httpGet: { path: /ready, port: 8080 }
          periodSeconds: 5
          failureThreshold: 3
          timeoutSeconds: 3
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 10"]  # lets nginx drop the endpoint before shutdown
      volumes:
      - name: dumps
        emptyDir:
          sizeLimit: 2Gi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api
  namespace: payments
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: payments-api
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 6
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Notes:

• HPA uses CPU, not memory. A cache-driven heap doesn't shrink when you add pods quickly enough, so memory-based HPA tends to flap. A requests-per-second custom metric is better if you have one.
• Heap dumps land in emptyDir, which is lost on pod deletion but survives container restarts. Copy them out with kubectl cp after an incident.

5.2 Application fixes (the real root cause; owned by the app team)

• Replace the HashMap in LocalCacheManager with a bounded cache, such as Caffeine with maximumSize (or maximumWeight) plus expireAfterWrite. The "soft limit" must evict, not just log.
• Question what processOrder caches. Caching one entry per order is essentially unbounded by design. If the data must be shared across pods, move it to Memorystore/Redis.
• Add backpressure. Cap Tomcat server.tomcat.threads.max and accept-count so one pod can't take 340 concurrent requests during a cold start.
• Make checkout idempotent end to end with idempotency keys to the PSP, so crashes and retries can't double-charge.

5.3 Capacity and process

• Load-test at 3–5x peak before campaigns and size memory from observed working set × 1.3–1.5.
• Run VPA in recommendation-only mode (updateMode: "Off") to get data-driven request suggestions.
• Make pre-scaling part of the marketing-launch checklist: raise minReplicas before the campaign starts.

5.4 Alerting (you'd have had warning hours earlier)

• container_memory_working_set_bytes / limit > 0.85 for 5m → warning
• increase(kube_pod_container_status_restarts_total{namespace="payments"}[10m]) > 2 → page
• kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 → page
• An app metric for cache entry count, plus JVM heap-after-GC above 80%
• nginx upstream 5xx rate for payments-api-svc

Outputs I need before closing this out

1. §3.1 (termination reason for all pods): confirms all 4 crashers are OOMKilled and not a mix with liveness kills.
2. §3.7 (HPA, GitOps markers, rollout history, node ages): decides whether the §4 commands are safe to run as written, and explains why all pods are 42m old.
3. §3.4 (VM.flags and cmdline on a survivor): confirms -Xmx220m is actually applied.
4. Cache implementation: LocalCacheManager.java around line 88, plus any cache TTL or size properties in application-prod.*. This decides whether the memory bump is a fix or a delay.
5. §3.3 and a re-run of kubectl top pods including h2r6w: explains why 2 pods survive.
6. §3.8 (node allocatable): confirms room for 1 Gi / 1 CPU pods, or tells us how many nodes the autoscaler will add.
7. Clean re-runs of the Deployment status: block and the events. The pasted output contains corrupted fields.

If you need to stop the bleeding now, §4 Steps 0–3 are safe to run on the current evidence. Steps 0 and 4 will catch the cases where they wouldn't be enough.