Cloud Infrastructure Architect

VerdantConnect: Production Architecture on AWS (HIPAA / SOC 2)

Summary. The design runs in us-east-1 as the active region with a warm standby in us-west-2. It uses:

• ECS on Fargate (Graviton) for the microservices.
• Aurora PostgreSQL Global Database for PHI.
• Amazon Chime SDK as the managed SFU. It is covered by the AWS BAA, so it adds no new subprocessor.
• Cognito for customer identity.
• An immutable, Object-Locked audit archive in a dedicated Log Archive account.

Steady-state cost is about $11.3k/month with commitments, or about $12.5k on-demand during the first 2–3 months. That leaves roughly 37% headroom under the $18k ceiling today. At 12% month-over-month growth, the ceiling is reached around months 9–10. Section 7 covers this, and it is the most important business finding in this document.

1. Assumptions and Blocking Questions

Blocking questions

I have proceeded with the defaults shown in brackets. Please confirm or correct them.

1. The SLO and the regional RTO conflict. A 99.95% monthly target allows about 21.6–22.3 minutes of downtime. Your regional RTO is 30 minutes. A single regional failover therefore breaks the SLO for that month.
   • Do declared regional disasters count against the SLO?
   • [Default: the SLO covers AZ-level and component failures. Regional disasters are governed by the RTO/RPO and excluded from the SLO. Meeting 99.95% through a regional loss would require active-active, adding roughly $4–6k/month and substantial engineering work on data consistency.]
2. Current state. Is the platform greenfield, or are the 18k MAU running live somewhere today? If live, where, and on which database?
   • [Default: a live system exists on a single-region Postgres-compatible stack, and we migrate it using logical replication with a reverse-replication rollback path.]
3. Workforce identity. Do you have a corporate IdP (Okta, Entra ID, or Google Workspace) for support/admin staff? Do any partner clinics want clinicians to sign in through their own IdP?
   • [Default: staff federate from a corporate IdP. Clinicians use a Verdant-managed Cognito pool with passkeys or TOTP. Partner-clinic SAML/OIDC federation is added per clinic.]

Assumptions

2. Architecture Overview

2.1 Component and data-flow diagram

 Patients: Web (React) / iOS / Android     Clinicians: Web (React)      Support/Admin staff
        │                                          │                             │
        │ static assets only (no PHI)              │                             │ AWS Verified Access
        ▼                                          │                             │ (corp IdP + MFA + device posture)
 ┌─────────────────────┐                           │                             │
 │ CloudFront + S3     │  SPA bundles, geo-restricted to US                      │
 └─────────────────────┘                           │                             │
        │  api.verdantconnect.com  (Route 53 failover records, TTL 60s)          │
        ▼                                          ▼                             ▼
┌────────────────────────────── us-east-1  (ACTIVE, 3 AZs) ──────────────────────────────────┐
│  AWS WAF ──► Public ALB (TLS 1.3/1.2 policy, ACM)          Internal ALB (admin) ◄── VA     │
│                 │ HTTPS to targets                                  │                      │
│  ┌──────────────▼───────── ECS on Fargate (Graviton), private app subnets ─────────────┐   │
│  │ api-gateway / BFF (GraphQL + REST, session validation, authZ policy engine)          │   │
│  │     │  ECS Service Connect with TLS (AWS Private CA, short-lived certs)               │   │
│  │     ├── identity-svc ───────── Cognito: patient pool | clinician pool                │   │
│  │     ├── scheduling-svc ─────── Postgres exclusion constraints (no double-booking)    │   │
│  │     ├── clinical-svc (notes, Rx) ───► e-Rx vendor (Surescripts; mTLS via NAT EIPs)   │   │
│  │     ├── video-svc ──────────── Chime SDK Meetings API (control plane only)           │   │
│  │     ├── billing-svc ────────── Payment gateway (tokens only, SAQ A)                  │   │
│  │     ├── notify-svc ─────────── SMS/email vendor (BAA; minimum-necessary content)     │   │
│  │     ├── calendar-svc ───────── Google/Microsoft Graph (OAuth; events contain no PHI) │   │
│  │     ├── integration-svc ◄──── Step Functions (batched) ───► Partner EHR FHIR APIs    │   │
│  │     ├── admin-svc (reachable only via internal ALB + Verified Access)                │   │
│  │     └── workers (SQS consumers; Fargate + Fargate Spot): reminders, summaries, recon │   │
│  └──────────────────────────────────────────────────────────────────────────────────────┘   │
│        │TLS verify-full         │TLS              │TLS             │ VPC endpoints (HTTPS)   │
│  ┌─────▼──────────────┐  ┌──────▼──────────┐  ┌───▼───────────┐  ┌▼──────────────────────┐ │
│  │ Aurora PostgreSQL  │  │ ElastiCache     │  │ DynamoDB      │  │ S3 PHI bucket         │ │
│  │ writer + reader    │  │ Valkey (cache,  │  │ global table  │  │ (attachments, labs)   │ │
│  │ Multi-AZ, pgAudit  │  │ rate limits)    │  │ (sessions)    │  │ SSE-KMS, versioned    │ │
│  └─────┬──────────────┘  └─────────────────┘  └──────┬────────┘  └──────┬────────────────┘ │
│        │                                             │                  │                  │
│  App audit events ──► Firehose (Parquet) ──► S3 Object Lock (Log Archive acct, 7 yr) ──┐   │
│  CloudTrail (org) / VPC Flow Logs / pgAudit ─────────────────────────────────────────┘   │
└────────┼─────────────────────────────────────────────┼──────────────────┼────────────────┘
         │ Aurora Global (storage-level, <1s typical)   │ global table      │ S3 CRR + RTC
┌────────▼──────────────── us-west-2  (WARM STANDBY) ───▼──────────────────▼────────────────┐
│ Aurora secondary cluster (1 × r7g.2xlarge, promotable)   DynamoDB replica   S3 replica     │
│ ECS services at minimum capacity, ALB + WAF, Valkey (small), Cognito shadow pools          │
│ AWS Backup vault (Vault Lock) with cross-region snapshot copies; audit archive replica     │
└───────────────────────────────────────────────────────────────────────────────────────────┘

Media path (never enters our VPC):
 Patient app ⇄ DTLS-SRTP ⇄ Chime SDK media (MediaRegion pinned to us-east-1 or us-west-2) ⇄ Clinician

2.2 Narrative

• Edge. Static SPA bundles are served from CloudFront. API traffic deliberately bypasses CloudFront and resolves through Route 53 failover records to a regional ALB protected by WAF.
  • This keeps PHI-bearing traffic on US regional endpoints, so there are no edge-location questions for auditors.
  • The trade-off is losing edge-level DDoS absorption. That is covered by Shield Standard, WAF rate-based rules, and autoscaling.
• API layer.
  • A BFF/gateway service terminates user sessions. Web clients use an HttpOnly, Secure, SameSite=strict cookie; mobile clients use an opaque token.
  • The gateway enforces the 15-minute idle timeout server-side and runs authorization through an in-process Cedar/OPA policy engine.
  • It then calls domain services over Service Connect TLS.
• Sessions. Cognito handles authentication events only (sign-in, MFA, password reset). Our own session record lives in a DynamoDB global table, so active sessions survive a regional failover.
• Video.
  • video-svc creates Chime SDK meetings and attendees with pseudonymous IDs. No names, reasons for visit, or other PHI go into meeting metadata.
  • Clients connect directly to Chime media.
  • Nothing is recorded, and media pipelines/capture are disabled by IAM policy.
• Scheduling and reminders.
  • Appointments are stored in Postgres with a tstzrange exclusion constraint (btree_gist), so double-booking is impossible even under concurrency.
  • Reminders use a DB outbox table driven by a 1-minute EventBridge Scheduler tick. They do not use per-appointment schedules, which are regional resources and would complicate DR.
• Integration.
  • Daily FHIR batches (~15k records) run as one Step Functions execution per partner, with a Map state over chunks of about 500 records. Per-record executions would cost about 10× more.
  • Outbound traffic uses static NAT EIPs so partners and the e-Rx vendor can allowlist us.
• Audit. Every PHI read or write emits a structured audit event. Events go through Firehose to Parquet in an Object-Locked (COMPLIANCE mode) bucket in a separate Log Archive account, queryable with Athena.

3. Service Breakdown

4. Resilience and Disaster Recovery

4.1 Availability design (within the region)

• 3 AZs for ALB, ECS tasks (minimum one per AZ for critical services), NAT gateways, and endpoints.
• Aurora: writer plus a same-size reader in a different AZ (promotion tier 0), with auto-scaled readers 0–2 for peaks. Failover is typically 30–60 seconds, and clients reconnect using the cluster endpoint with retry and backoff.
• Dependency budget.
  • Several dependencies publish SLAs below 99.95%; Cognito's is 99.9%.
  • To reduce exposure, the BFF session design means existing users are unaffected by a Cognito brownout. Only new sign-ins are affected, and JWKS keys are cached.
  • Video clients have automatic rejoin. A PSTN dial-in fallback (Chime SDK PSTN audio) can be enabled later as a degraded-mode path for visits.
• Deploy safety: ECS deployment circuit breaker plus CloudWatch-alarm rollback on 5xx rate and p95 latency. Most real-world downtime comes from deploys, not AZ failures.

4.2 Autoscaling rules

4.3 Backups

• Aurora: 35-day PITR (continuous). AWS Backup takes daily snapshots (kept 35 days) and monthly snapshots (kept 1 year; longer if your BAA or partner medical-records retention requires it). Snapshots are copied cross-account into the Backup account and cross-region to us-west-2, both with Vault Lock in compliance mode.
• S3: versioning, CRR to us-west-2, and lifecycle rules that expire noncurrent versions after 90 days (PHI bucket only; audit bucket rules are in the Terraform).
• DynamoDB: PITR enabled.
• Restore testing: AWS Backup restore-testing plans run monthly with automated validation queries. The results are SOC 2 A1.2/A1.3 evidence.

4.4 RTO/RPO by component

Trade-off: S3 RPO gap. S3 Replication Time Control guarantees 99.99% of objects replicate within 15 minutes. That is outside the 5-minute RPO in the worst case, although it is typically seconds. Mitigations:

• Alarm on ReplicationLatency above 300s.
• A post-failover reconciliation job flags DB rows whose object is missing in the replica, and those uploads are re-requested.
• Dual-writing uploads to both regions would close the gap but doubles write complexity. I don't recommend it at this stage.

Trade-off: rds.global_db_rpo. This Aurora PostgreSQL parameter can enforce an RPO by pausing commits when secondary lag exceeds a threshold. I do not recommend enabling it. It couples primary write availability to the health of the secondary region, so a us-west-2 problem could stop writes in us-east-1. Use alarms instead.

4.5 Regional failover runbook (target: 30 minutes)

Failover is human-declared, then automated. Automatic regional failover risks flapping and split-brain, and it would cost more than the problem it solves at this stage.

Identity during regional DR (trade-off). Cognito user pools are regional and cannot export password hashes or MFA secrets.

• Existing sessions survive because they live in the DynamoDB global table.
• New sign-ins in us-west-2 go to shadow pools. These are kept in sync for profile attributes via EventBridge, but they carry no credentials.
  • Patients sign in with email or SMS OTP.
  • Clinicians re-verify with email OTP and then must re-enroll a passkey or TOTP before any PHI access.
  • Staff are unaffected because the corporate IdP is itself multi-region.
• If this degraded path is unacceptable, the alternative is a multi-region CIAM with a BAA, such as Okta Customer Identity (Auth0) at an enterprise tier. That is SaaS spend outside this budget.

DR testing: a quarterly game day. Run the full regional failover in staging each quarter and in production twice a year during a low-traffic window, such as Sunday 02:00 ET. Record measured RTO/RPO as SOC 2 evidence.

Fargate capacity during regional events: many customers fail over at the same time. The warm minimum capacity in us-west-2 and pre-created ALB, WAF, and target groups reduce reliance on burst launches.

5. Security by Design

5.1 Account structure and IAM least privilege

• Human access goes through IAM Identity Center from the corporate IdP, with phishing-resistant MFA (FIDO2).
  • There are no IAM users except two sealed break-glass users in the management account. Their hardware MFA keys are kept in separate safes, and any use of them triggers an alarm.
• Permission sets:
  • ReadOnly-Prod explicitly denies the PHI data plane: s3:GetObject on PHI buckets, secretsmanager:GetSecretValue, rds-db:connect, dynamodb:GetItem on sessions, and kms:Decrypt on the PHI key.
  • Deploy is used only by CI through OIDC.
  • PlatformAdmin-Prod is just-in-time, using AWS's open-source TEAM solution (Temporary Elevated Access Management). It requires an approver, a justification, and a session of at most 4 hours.
• Workload roles: one ECS task role per service, scoped to specific resource ARNs, with a permissions boundary on all roles that CI creates.
• Data perimeter:
  • PHI bucket policies allow only named task roles via aws:SourceVpce.
  • VPC endpoint policies restrict to aws:ResourceOrgID, which blocks exfiltration to buckets in other accounts.
  • KMS key policies grant use only to specific roles.
• SCPs (org-wide):
  • Deny regions other than us-east-1 and us-west-2 (data residency).
  • Deny disabling CloudTrail, GuardDuty, Config, or Security Hub.
  • Deny creating unencrypted RDS or EBS.
  • Deny IAM user and access-key creation in workload OUs.
  • Deny organizations:LeaveOrganization.
  • Deny non-HIPAA-eligible services in Prod via an allowlist.
• Database:
  • One Postgres role per service with schema-scoped grants.
  • Row-Level Security on PHI tables using SET LOCAL app.actor_id / app.role.
  • Human DB access is break-glass only: IAM DB authentication through SSM Session Manager port forwarding, fully logged.
  • ecs exec is disabled in prod.

5.2 Network segmentation

• A VPC per environment per region with three subnet tiers:
  • public: ALB and NAT only.
  • private-app: ECS tasks.
  • private-data: Aurora, Valkey, and interface endpoints. This tier has no route to the NAT gateway.
• Security groups reference other security groups: ALB SG → gateway SG → service SGs → data SG on specific ports only. NACLs on data subnets deny 0.0.0.0/0.
• Egress: NAT with static EIPs, plus Route 53 Resolver DNS Firewall in allowlist mode (AWS domains and named vendors: e-Rx, SMS/email, payment, Google/Microsoft calendar APIs, partner FHIR hosts).
  • Trade-off: DNS Firewall does not stop connections made directly to an IP address. AWS Network Firewall does, but costs about $870/month for 3 AZs. Revisit it at SOC 2 year 2 or if pen testing shows the gap.
• VPC Flow Logs go to S3 as Parquet in the Log Archive account.

5.3 Encryption

At rest. Customer-managed KMS keys use AES-256-GCM, with separate keys per domain (phi, secrets, logs, audit), annual rotation, and multi-Region replicas only to us-west-2. Encrypted with these keys:

• Aurora and Aurora snapshots
• S3 (SSE-KMS with bucket keys, which cut KMS request costs by about 99%)
• DynamoDB, Valkey, SQS, Firehose
• CloudWatch Logs, Secrets Manager
• AWS Backup vaults

AWS Config rules detect any drift.

Field-level encryption. The AWS Encryption SDK with data-key caching protects especially sensitive fields:

• Calendar OAuth refresh tokens
• Government IDs, if collected
• Any 42 CFR Part 2 (substance-use treatment) or behavioral-health notes, if applicable

In transit:

Trade-off: Service Connect TLS is one-way TLS. It encrypts traffic and authenticates the server, but it does not provide mutual TLS. Caller identity is established by a short-lived, signed internal JWT that carries the service identity and the end-user context. If auditors or pen testers push for workload-identity mTLS, VPC Lattice with IAM auth policies is the managed path, adding roughly $200–300/month.

5.4 Secrets management

• All secrets live in Secrets Manager: CMK-encrypted, replicated to us-west-2, and injected into ECS through the task definition secrets.valueFrom or fetched with the SDK and cached. Nothing is placed in .env files or Terraform state.
  • Terraform uses ephemeral resources and write-only attributes (see the Terraform starter).
• Rotation schedule:
  • Service DB credentials every 30 days, using the alternating-users strategy.
  • Vendor API keys every 90 days where the vendor supports it.
  • The e-Rx client certificate and key are stored as a secret, with an alarm 30 days before certificate expiry.
• Leak prevention: gitleaks pre-commit hooks, GitHub secret scanning with push protection, and Checkov/Trivy in CI. A detected secret fails the build and triggers a rotation runbook.

5.5 Identity, MFA, and session timeout

• How the 15-minute timeout is enforced.
  • The BFF session record stores last_activity, written at most once per 60 seconds to control DynamoDB cost.
  • Any request after 15 minutes of inactivity is rejected, and the session and its refresh token are revoked.
  • The client shows a 2-minute warning dialog.
• Edge cases: during an active video visit or note editing, heartbeat pings and keystroke-driven autosave count as activity. Drafts autosave every 30 seconds so a timeout never loses clinical notes.
• Mobile: after 15 minutes of inactivity or backgrounding, the app locks and requires biometric or PIN re-authentication. Tokens are stored in Keychain/Keystore. FLAG_SECURE and the iOS privacy overlay protect PHI screens.

5.6 RBAC, relationship-based access, and break-glass

• Roles: patient, clinician, clinic_admin, support_tier1, support_tier2, billing_ops, security_admin.
• Policy rules (Cedar/OPA, policy-as-code in git, reviewed via PR):
  • A patient may access only their own records.
  • A clinician may access a patient only with an active care relationship: an upcoming or completed appointment within N days, or clinic-level assignment. RLS enforces the same rule in the database.
  • Support staff see account and scheduling metadata by default and not clinical notes.
• Break-glass for support emergencies:
  1. The support agent requests elevated access for a specific patient with a reason code, free-text justification, and a ticket ID.
  2. A second approver (tier 2 or the on-call security admin) approves. For a declared P1 there is auto-approval with a mandatory review afterward.
  3. A time-boxed grant (60 minutes, one patient) is written to a access_grants table that the policy engine checks.
  4. Every access during the grant carries break_glass=true in the audit event, and a real-time alert goes to #security with IDs only, no PHI.
  5. The Privacy Officer reviews all break-glass sessions weekly. Quarterly reviews are SOC 2 evidence.
• Infrastructure break-glass uses TEAM JIT roles, with sealed root and emergency IAM credentials as a last resort. Use of either triggers a PagerDuty SEV-2 page to security.

5.7 PHI access audit logging (tamper-evident, retained 6+ years)

Event schema, emitted by a shared audit library used by every service:

{ "event_id":"uuid", "ts":"RFC3339", "actor_id":"...", "actor_role":"clinician",
  "actor_org":"clinic-123", "action":"READ|CREATE|UPDATE|DELETE|EXPORT|PRINT",
  "resource_type":"EncounterNote", "resource_id":"...", "patient_id":"...",
  "purpose":"TREATMENT|PAYMENT|OPERATIONS|BREAK_GLASS", "break_glass":false,
  "source_ip":"...", "user_agent":"...", "geo":"US-VA", "session_id":"...",
  "request_id":"...", "outcome":"ALLOW|DENY", "service":"clinical-svc" }

Delivery path.

• Events go to Firehose via PutRecordBatch with retries. If Firehose is unavailable, they fall back to an SQS queue.
• If both fail, the PHI read fails closed. This trade-off favors compliance over availability, and it should be exceedingly rare.
• For writes, the audit event is written through a transactional outbox in the same DB transaction.

Storage and tamper evidence.

• Firehose converts events to Parquet, partitioned by date.
• Records land in the Log Archive account bucket with Object Lock COMPLIANCE, 7-year retention (exceeding the 6-year minimum). The bucket is replicated to us-west-2 and tiers to Glacier IR at 90 days and Deep Archive at 2 years.
• Tamper evidence beyond WORM: an hourly job writes a manifest of SHA-256 object hashes, chains it to the previous manifest's hash, and signs it with an asymmetric KMS key. CloudTrail log-file integrity validation covers the AWS API trail.

Other trails.

• pgAudit: DDL, role changes, and object-level auditing on PHI tables via pgaudit.role, exported to CloudWatch Logs and archived.
• CloudTrail org trail: includes S3 data events on PHI buckets and KMS events.

Detection rules (Athena scheduled queries and CloudWatch metric filters routed through EventBridge to PagerDuty/Slack; alerts contain IDs only):

• A clinician viewing more than N patients with no care relationship.
• Access to VIP or employee records.
• After-hours bulk reads.
• Any break-glass use.

Access to audit data. Queries of the audit data are themselves logged. Only the security_admin and Privacy Officer roles can read it. The system can produce a per-patient access report (accounting of disclosures) on request.

5.8 Data residency

• The org SCP denies every region except us-east-1 and us-west-2.
• Aurora Global, S3 CRR, DynamoDB replicas, Backup copies, KMS replicas, and Secrets replicas all target us-west-2 only.
• Chime SDK MediaRegion is pinned server-side to us-east-1 or us-west-2. Any other value is rejected.
• CloudFront serves only non-PHI static assets and is geo-restricted to the US.
• Third-party residency gaps to close:
  • Push notifications (APNs/FCM): no PHI in payloads. Use "You have a new message" with a deep link. Apple and Google do not sign BAAs.
  • Crash reporting and analytics: no third-party trackers on authenticated screens (see the HHS OCR guidance on online tracking technologies). Use a BAA-covered, US-hosted crash tool, or scrub PHI.
  • Calendar sync: events contain only "VerdantConnect visit" plus a link, with no patient name or visit reason. Personal Google/Outlook calendars are not covered by any BAA.

5.9 Vendor and BAA register (maintain it in the compliance platform)

5.10 Incident response and breach notification

• The plan (owned by the Security Lead, approved annually) covers:
  • Severity matrix (SEV-1 to SEV-4)
  • Roles: IC, Security Lead, Privacy Officer, Legal, Comms
  • Runbooks for: credential compromise, PHI exposure, ransomware, vendor breach, lost device, insider misuse
• Tooling: GuardDuty, Security Hub, and Macie findings flow through EventBridge to PagerDuty.
  • Forensics runbook: isolate tasks by moving them to a quarantine security group and stopping them after capturing logs; snapshot data; preserve CloudTrail and VPC Flow Logs.
  • Legal hold via an Object Lock legal-hold flag.
• HIPAA breach workflow:
  1. Discovery starts the clock.
  2. The four-factor risk assessment (§164.402) is documented.
  3. As a business associate, you notify the covered entity without unreasonable delay and no later than 60 calendar days after discovery. Most BAAs contractually require far sooner, often 5–10 business days; track the shortest obligation per partner.
  4. The covered entity notifies individuals, HHS (immediately if 500+ individuals are affected), and media where more than 500 residents of a state are affected, unless it delegated this to you.
  5. State breach laws can be stricter (some are 30 days). Legal maintains a matrix.
  6. All documentation is retained for 6 years.
• Exercises: tabletop exercises twice a year (one PHI-exposure scenario, one ransomware/DR scenario). After-action reports are SOC 2 CC7.4/CC7.5 evidence.

5.11 Security testing cadence

Regulatory note: HHS proposed Security Rule amendments (NPRM, Jan 2025) that would make encryption, MFA, annual pen testing, asset inventories, and 72-hour restoration capability explicit requirements. I cannot confirm whether they have been finalized. This design already meets those proposals, so it is robust either way.

6. Security Checklist (Requirements → Controls → Evidence)

7. Terraform Starter

This is a representative slice of the data layer, audit archive, service mesh TLS, and org guardrails. VPC, IAM policy documents, and task definitions are referenced as omitted modules and files.

###############################################################################
# VerdantConnect – production starter (representative, not complete)
#
# Suggested layout (separate state per layer, blast-radius isolation):
#   live/org/          -> SCPs, org-level config          (management account)
#   live/prod/network  -> VPCs, endpoints, DNS Firewall
#   live/prod/data     -> KMS, Aurora Global, S3, DynamoDB  (this file)
#   live/prod/compute  -> ECS cluster/services, ALB, WAF
#   live/log-archive/  -> audit archive, CloudTrail bucket
#
# Requires Terraform >= 1.11 (ephemeral resources, write-only arguments,
# S3-native state locking) and AWS provider 6.x. Verify argument names against
# the provider version you pin before applying.
###############################################################################

terraform {
  required_version = ">= 1.11"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.7"
    }
  }

  backend "s3" {
    bucket       = "vc-tfstate-prod-use1" # versioned, SSE-KMS, access-logged, BPA on
    key          = "prod/data/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    kms_key_id   = "alias/vc-tfstate"
    use_lockfile = true # S3-native locking; no DynamoDB lock table needed
  }
}

variable "prod_account_id"        { type = string }
variable "log_archive_account_id" { type = string }
variable "firehose_role_arn"      { type = string } # prod-account Firehose delivery role

variable "aurora_engine_version" {
  type        = string
  description = "Pin to an Aurora PostgreSQL 16.x version that supports Global Database."
  default     = "16.6"
}

locals {
  tags = {
    Application        = "verdantconnect"
    Environment        = "prod"
    DataClassification = "phi" # drives Config rules, Macie scope, and cost allocation
    Owner              = "platform"
    ManagedBy          = "terraform"
  }
  azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
}

# --- Providers -----------------------------------------------------------------
provider "aws" {
  region              = "us-east-1"
  allowed_account_ids = [var.prod_account_id] # prevents applying to the wrong account
  default_tags { tags = local.tags }
}

provider "aws" {
  alias               = "dr"
  region              = "us-west-2" # only permitted DR region (see SCP)
  allowed_account_ids = [var.prod_account_id]
  default_tags { tags = local.tags }
}

provider "aws" {
  alias  = "log_archive"
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::${var.log_archive_account_id}:role/vc-terraform-deployer"
  }
  default_tags { tags = local.tags }
}

###############################################################################
# KMS – one CMK per data domain, multi-Region so ciphertext is portable to DR.
###############################################################################
locals {
  kms_domains = toset(["phi", "secrets", "logs"])
}

resource "aws_kms_key" "this" {
  for_each                = local.kms_domains
  description             = "VerdantConnect ${each.key} CMK (prod)"
  multi_region            = true
  enable_key_rotation     = true
  rotation_period_in_days = 365
  deletion_window_in_days = 30
  # Least-privilege key policy: named task roles only, no kms:* for humans.
  policy = data.aws_iam_policy_document.kms[each.key].json # defined in kms_policies.tf
}

resource "aws_kms_alias" "this" {
  for_each      = local.kms_domains
  name          = "alias/vc-${each.key}"
  target_key_id = aws_kms_key.this[each.key].key_id
}

resource "aws_kms_replica_key" "dr" {
  for_each                = local.kms_domains
  provider                = aws.dr
  primary_key_arn         = aws_kms_key.this[each.key].arn
  deletion_window_in_days = 30
  policy                  = data.aws_iam_policy_document.kms[each.key].json
}

###############################################################################
# Aurora PostgreSQL Global Database
#   RPO: storage-level replication, typically < 1s (alarm on AuroraGlobalDBRPOLag)
#   NOTE: we intentionally do NOT set rds.global_db_rpo – it would block commits
#   in us-east-1 whenever us-west-2 lags, coupling primary availability to DR.
###############################################################################
locals {
  aurora_params = [
    { name = "rds.force_ssl", value = "1", apply = "immediate" },
    { name = "ssl_min_protocol_version", value = "TLSv1.2", apply = "immediate" },
    { name = "shared_preload_libraries", value = "pg_stat_statements,pgaudit", apply = "pending-reboot" },
    { name = "pgaudit.log", value = "ddl,role", apply = "immediate" }, # session-level
    { name = "pgaudit.role", value = "rds_pgaudit", apply = "immediate" }, # object-level on PHI tables
    { name = "log_connections", value = "1", apply = "immediate" },
    { name = "log_disconnections", value = "1", apply = "immediate" },
  ]
}

resource "aws_rds_cluster_parameter_group" "primary" {
  name   = "vc-prod-aurora-pg16"
  family = "aurora-postgresql16"

  dynamic "parameter" {
    for_each = local.aurora_params
    content {
      name         = parameter.value.name
      value        = parameter.value.value
      apply_method = parameter.value.apply
    }
  }
}

resource "aws_rds_cluster_parameter_group" "dr" {
  provider = aws.dr
  name     = "vc-prod-aurora-pg16"
  family   = "aurora-postgresql16"

  dynamic "parameter" {
    for_each = local.aurora_params
    content {
      name         = parameter.value.name
      value        = parameter.value.value
      apply_method = parameter.value.apply
    }
  }
}

resource "aws_rds_global_cluster" "vc" {
  global_cluster_identifier = "vc-prod-global"
  engine                    = "aurora-postgresql"
  engine_version            = var.aurora_engine_version
  database_name             = "verdant"
  storage_encrypted         = true
  deletion_protection       = true
}

# Master password is generated ephemerally and written only to write-only
# arguments, so it NEVER lands in Terraform state or plan files.
# Services do not use this credential: each has its own rotated DB role.
ephemeral "random_password" "db_master" {
  length           = 40
  special          = true
  override_special = "!#$%^&*()-_=+[]{}<>:?" # excludes / @ " and space (RDS-invalid)
}

resource "aws_secretsmanager_secret" "db_master" {
  name        = "prod/aurora/master"
  description = "Break-glass Aurora master credential (rotated by Lambda)"
  kms_key_id  = aws_kms_key.this["secrets"].arn

  replica {
    region     = "us-west-2"
    kms_key_id = aws_kms_replica_key.dr["secrets"].arn
  }
}

resource "aws_secretsmanager_secret_version" "db_master" {
  secret_id = aws_secretsmanager_secret.db_master.id
  secret_string_wo = jsonencode({
    username = "vc_admin"
    password = ephemeral.random_password.db_master.result
  })
  secret_string_wo_version = 1 # bump to push a new value from Terraform
}
# aws_secretsmanager_secret_rotation (30-day, alternating-users Lambda) omitted.

resource "aws_rds_cluster" "primary" {
  cluster_identifier        = "vc-prod-use1"
  global_cluster_identifier = aws_rds_global_cluster.vc.id
  engine                    = aws_rds_global_cluster.vc.engine
  engine_version            = aws_rds_global_cluster.vc.engine_version

  master_username            = "vc_admin"
  master_password_wo         = ephemeral.random_password.db_master.result
  master_password_wo_version = 1

  storage_encrypted = true
  kms_key_id        = aws_kms_key.this["phi"].arn
  # Aurora Standard storage by default. Switch to I/O-Optimized
  # (storage_type = "aurora-iopt1") once I/O exceeds ~25% of Aurora spend.

  db_subnet_group_name            = module.network.aurora_subnet_group_name # data tier, no NAT route
  vpc_security_group_ids          = [module.network.aurora_sg_id]
  db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.primary.name

  iam_database_authentication_enabled = true # break-glass human access via SSM port-forward
  backup_retention_period             = 35   # PITR window (RPO for logical corruption)
  preferred_backup_window             = "07:00-08:00" # UTC = 03:00-04:00 ET, lowest traffic
  preferred_maintenance_window        = "sun:08:00-sun:09:00"
  copy_tags_to_snapshot               = true
  deletion_protection                 = true
  skip_final_snapshot                 = false
  final_snapshot_identifier           = "vc-prod-use1-final"
  enabled_cloudwatch_logs_exports     = ["postgresql"] # pgAudit -> CW Logs (KMS) -> archive
}

resource "aws_rds_cluster_instance" "primary" {
  count              = 2 # writer + same-size reader (failover tier 0) in different AZs
  identifier         = "vc-prod-use1-${count.index}"
  cluster_identifier = aws_rds_cluster.primary.id
  instance_class     = "db.r7g.2xlarge"
  engine             = aws_rds_cluster.primary.engine
  engine_version     = aws_rds_cluster.primary.engine_version
  availability_zone  = local.azs[count.index]
  promotion_tier     = 0

  performance_insights_enabled    = true
  performance_insights_kms_key_id = aws_kms_key.this["phi"].arn
  monitoring_interval             = 15
  monitoring_role_arn             = aws_iam_role.rds_enhanced_monitoring.arn # iam.tf
  auto_minor_version_upgrade      = false # upgrades go through staging first (change mgmt)
}

resource "aws_rds_cluster" "secondary" {
  provider                  = aws.dr
  cluster_identifier        = "vc-prod-usw2"
  global_cluster_identifier = aws_rds_global_cluster.vc.id
  engine                    = aws_rds_global_cluster.vc.engine
  engine_version            = aws_rds_global_cluster.vc.engine_version

  storage_encrypted = true
  kms_key_id        = aws_kms_replica_key.dr["phi"].arn # region-local key required

  db_subnet_group_name            = module.network_dr.aurora_subnet_group_name
  vpc_security_group_ids          = [module.network_dr.aurora_sg_id]
  db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.dr.name
  iam_database_authentication_enabled = true
  deletion_protection             = true
  skip_final_snapshot             = true

  depends_on = [aws_rds_cluster_instance.primary]

  lifecycle {
    # Set by AWS on global-cluster membership; avoids perpetual diffs / failover drift.
    ignore_changes = [replication_source_identifier, global_cluster_identifier]
  }
}

resource "aws_rds_cluster_instance" "secondary" {
  provider           = aws.dr
  identifier         = "vc-prod-usw2-0"
  cluster_identifier = aws_rds_cluster.secondary.id
  instance_class     = "db.r7g.2xlarge" # same size: no post-failover resize inside the RTO
  engine             = aws_rds_cluster.secondary.engine
  engine_version     = aws_rds_cluster.secondary.engine_version
  promotion_tier     = 0
}

###############################################################################
# PHI access audit archive (Log Archive account)
#   WORM via S3 Object Lock COMPLIANCE mode: nobody, including root, can delete
#   or shorten retention. 7 years > HIPAA's 6-year minimum.
###############################################################################
resource "aws_kms_key" "audit" {
  provider                = aws.log_archive
  description             = "PHI audit archive CMK"
  enable_key_rotation     = true
  deletion_window_in_days = 30
  policy                  = data.aws_iam_policy_document.audit_kms.json # grants Firehose role encrypt only
}

resource "aws_s3_bucket" "audit" {
  provider            = aws.log_archive
  bucket              = "vc-phi-access-audit-${var.log_archive_account_id}-use1"
  object_lock_enabled = true
}

resource "aws_s3_bucket_ownership_controls" "audit" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.audit.id
  rule {
    object_ownership = "BucketOwnerEnforced" # cross-account writes owned by Log Archive
  }
}

resource "aws_s3_bucket_versioning" "audit" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.audit.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_object_lock_configuration" "audit" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.audit.id
  rule {
    default_retention {
      mode  = "COMPLIANCE"
      years = 7
    }
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "audit" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.audit.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.audit.arn
    }
    bucket_key_enabled = true # ~99% fewer KMS requests
  }
}

resource "aws_s3_bucket_public_access_block" "audit" {
  provider                = aws.log_archive
  bucket                  = aws_s3_bucket.audit.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_lifecycle_configuration" "audit" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.audit.id

  rule {
    id     = "tier-and-expire"
    status = "Enabled"
    filter {}

    transition {
      days          = 90
      storage_class = "GLACIER_IR" # still Athena-queryable, millisecond retrieval
    }
    transition {
      days          = 730
      storage_class = "DEEP_ARCHIVE" # restore required before querying (12h)
    }
    expiration {
      days = 2585 # 7 years + 30 days; Object Lock blocks any earlier delete
    }
    noncurrent_version_expiration {
      noncurrent_days = 2585
    }
  }
}

data "aws_iam_policy_document" "audit_bucket" {
  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [aws_s3_bucket.audit.arn, "${aws_s3_bucket.audit.arn}/*"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }

  statement {
    sid       = "DenyTLSBelow12"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [aws_s3_bucket.audit.arn, "${aws_s3_bucket.audit.arn}/*"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "NumericLessThan"
      variable = "s3:TlsVersion"
      values   = ["1.2"]
    }
  }

  statement {
    sid       = "AllowFirehoseAppendOnly" # write-only: no Get/Delete for the producer
    effect    = "Allow"
    actions   = ["s3:PutObject", "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:ListBucket"]
    resources = [aws_s3_bucket.audit.arn, "${aws_s3_bucket.audit.arn}/*"]
    principals {
      type        = "AWS"
      identifiers = [var.firehose_role_arn]
    }
  }
}

resource "aws_s3_bucket_policy" "audit" {
  provider = aws.log_archive
  bucket   = aws_s3_bucket.audit.id
  policy   = data.aws_iam_policy_document.audit_bucket.json
}
# CRR of this bucket to us-west-2 (Object Lock retention replicates) omitted for brevity.

###############################################################################
# Service-to-service TLS: ECS Service Connect + AWS Private CA (short-lived mode,
# ~$50/month per CA vs ~$400 general-purpose). Root activation
# (aws_acmpca_certificate + aws_acmpca_certificate_authority_certificate) omitted.
###############################################################################
resource "aws_acmpca_certificate_authority" "svc_mesh" {
  type                            = "ROOT"
  usage_mode                      = "SHORT_LIVED_CERTIFICATE"
  permanent_deletion_time_in_days = 30

  certificate_authority_configuration {
    key_algorithm     = "EC_prime256v1"
    signing_algorithm = "SHA256WITHECDSA"
    subject {
      common_name  = "vc-prod-service-mesh-use1"
      organization = "Verdant Health Technologies"
    }
  }
}

resource "aws_service_discovery_http_namespace" "core" {
  name = "vc-prod.internal"
}

resource "aws_ecs_service" "api_gateway" {
  name                   = "api-gateway"
  cluster                = aws_ecs_cluster.core.id                   # compute.tf
  task_definition        = aws_ecs_task_definition.api_gateway.arn   # ARM64, secrets via valueFrom
  desired_count          = 3
  launch_type            = "FARGATE"
  enable_execute_command = false # no interactive shells into PHI workloads
  propagate_tags         = "SERVICE"

  network_configuration {
    subnets          = module.network.app_subnet_ids
    security_groups  = [aws_security_group.api_gateway.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.api_gateway.arn # HTTPS target group
    container_name   = "api-gateway"
    container_port   = 8443
  }

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  alarms {
    enable      = true
    rollback    = true
    alarm_names = [aws_cloudwatch_metric_alarm.api_5xx_rate.alarm_name]
  }

  service_connect_configuration {
    enabled   = true
    namespace = aws_service_discovery_http_namespace.core.arn

    service {
      port_name      = "https"
      discovery_name = "api-gateway"
      client_alias {
        port     = 8443
        dns_name = "api-gateway.vc-prod.internal"
      }
      tls {
        issuer_cert_authority {
          aws_pca_authority_arn = aws_acmpca_certificate_authority.svc_mesh.arn
        }
        kms_key  = aws_kms_key.this["phi"].arn
        role_arn = aws_iam_role.ecs_service_connect_tls.arn # iam.tf
      }
    }
  }

  lifecycle {
    ignore_changes = [desired_count] # owned by Application Auto Scaling
  }
}

resource "aws_appautoscaling_target" "api_gateway" {
  service_namespace  = "ecs"
  resource_id        = "service/${aws_ecs_cluster.core.name}/${aws_ecs_service.api_gateway.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  min_capacity       = 3 # one per AZ
  max_capacity       = 30
}

resource "aws_appautoscaling_policy" "api_gateway_rps" {
  name               = "alb-requests-per-task"
  policy_type        = "TargetTrackingScaling"
  service_namespace  = aws_appautoscaling_target.api_gateway.service_namespace
  resource_id        = aws_appautoscaling_target.api_gateway.resource_id
  scalable_dimension = aws_appautoscaling_target.api_gateway.scalable_dimension

  target_tracking_scaling_policy_configuration {
    target_value       = 150 # req/s per 1-vCPU Node task; tune after load test
    scale_out_cooldown = 60
    scale_in_cooldown  = 300
    predefined_metric_specification {
      predefined_metric_type = "ALBRequestCountPerTarget"
      resource_label         = "${aws_lb.api.arn_suffix}/${aws_lb_target_group.api_gateway.arn_suffix}"
    }
  }
}

# Pre-warm for the rolling US morning peak (08:00 ET through 10:00 PT).
resource "aws_appautoscaling_scheduled_action" "api_morning_peak" {
  name               = "prewarm-morning"
  service_namespace  = aws_appautoscaling_target.api_gateway.service_namespace
  resource_id        = aws_appautoscaling_target.api_gateway.resource_id
  scalable_dimension = aws_appautoscaling_target.api_gateway.scalable_dimension
  schedule           = "cron(30 7 ? * MON-FRI *)"
  timezone           = "America/New_York"
  scalable_target_action {
    min_capacity = 6
    max_capacity = 30
  }
}

resource "aws_appautoscaling_scheduled_action" "api_midday_reset" {
  name               = "reset-midday"
  service_namespace  = aws_appautoscaling_target.api_gateway.service_namespace
  resource_id        = aws_appautoscaling_target.api_gateway.resource_id
  scalable_dimension = aws_appautoscaling_target.api_gateway.scalable_dimension
  schedule           = "cron(30 13 ? * MON-FRI *)"
  timezone           = "America/New_York"
  scalable_target_action {
    min_capacity = 3
    max_capacity = 30
  }
}
# (Evening 16:30–22:30 ET pair is identical in shape.)

###############################################################################
# Org guardrails (applied from live/org with the management-account provider
# alias "aws.org"). Shown here for reference.
###############################################################################
data "aws_iam_policy_document" "scp_guardrails" {
  statement {
    sid    = "DenyNonUSRegions" # HIPAA data-residency control
    effect = "Deny"
    # Global services are homed in us-east-1 (allowed), but exempt them explicitly
    # so CLI/SDK calls with a different default region don't break.
    not_actions = [
      "iam:*", "sts:*", "organizations:*", "account:*", "route53:*",
      "route53domains:*", "cloudfront:*", "support:*", "budgets:*",
      "ce:*", "health:*", "trustedadvisor:*",
    ]
    resources = ["*"]
    condition {
      test     = "StringNotEquals"
      variable = "aws:RequestedRegion"
      values   = ["us-east-1", "us-west-2"]
    }
    condition {
      test     = "ArnNotLike"
      variable = "aws:PrincipalARN"
      values   = ["arn:aws:iam::*:role/AWSControlTowerExecution"]
    }
  }

  statement {
    sid    = "ProtectAuditAndDetection" # tamper-resistance for SOC 2 / HIPAA audit controls
    effect = "Deny"
    actions = [
      "cloudtrail:StopLogging", "cloudtrail:DeleteTrail", "cloudtrail:UpdateTrail",
      "guardduty:DeleteDetector", "guardduty:DisassociateFromAdministratorAccount",
      "config:StopConfigurationRecorder", "config:DeleteConfigurationRecorder",
      "securityhub:DisableSecurityHub", "organizations:LeaveOrganization",
    ]
    resources = ["*"]
    condition {
      test     = "ArnNotLike"
      variable = "aws:PrincipalARN"
      values   = ["arn:aws:iam::*:role/vc-security-automation"]
    }
  }

  statement {
    sid       = "DenyUnencryptedRDS"
    effect    = "Deny"
    actions   = ["rds:CreateDBCluster", "rds:CreateDBInstance"]
    resources = ["*"]
    condition {
      test     = "Bool"
      variable = "rds:StorageEncrypted"
      values   = ["false"]
    }
  }
}

# resource "aws_organizations_policy" "guardrails" {
#   provider = aws.org
#   name     = "vc-phi-guardrails"
#   type     = "SERVICE_CONTROL_POLICY"
#   content  = data.aws_iam_policy_document.scp_guardrails.json
# }
# resource "aws_organizations_policy_attachment" "root" {
#   provider  = aws.org
#   policy_id = aws_organizations_policy.guardrails.id
#   target_id = var.org_root_id   # residency applies org-wide
# }

CI guardrails for this code:

• terraform plan runs on every PR, and Checkov/Trivy fail the build on high-severity findings.
• A Conftest/OPA policy requires DataClassification=phi resources to set a KMS key.
• Applies to prod happen only from the main branch through the GitHub OIDC deploy role, with a required reviewer.

8. Estimated Monthly Cost

Assumptions: 1-year no-upfront Reserved Instances for Aurora and Valkey, and a 1-year Compute Savings Plan for Fargate. Both are bought after 4–6 weeks of measured steady state. Prices are list, approximate (±15%). Traffic is as stated: 120 req/s average, 900 peak, 9k visits.

Unit economics: about $1.25 of infrastructure per video visit. Track this monthly as the primary FinOps KPI.

Services deliberately not included, and why:

Growth projection (the key finding)

At 12% month-over-month growth, usage roughly doubles every 6 months (1.12⁶ ≈ 1.97) and grows about 3.9× per year. About $2.8k of today's bill scales with usage: Chime, Fargate, WAF requests, egress, logs, audit, Cognito, and Aurora I/O.

Recommendation: convert the fixed $18k ceiling into a unit-cost budget, for example ≤ $1.10 infrastructure per visit, falling as fixed costs amortize. Set AWS Budgets alerts at 80% and 100% of forecast, plus anomaly detection per cost-allocation tag.

Cost-optimization levers

• Graviton everywhere (already applied), plus Fargate Spot for idempotent workers.
• Commitments laddered quarterly rather than bought as one large purchase, so sizing can follow growth.
• Aurora I/O-Optimized once I/O exceeds ~25% of Aurora spend. Cache calendar and availability reads in Valkey. Tune queries with Database Insights.
• Throttle session-activity writes to one per 60s. Aggregate audit events client-side to approach 5 KB per record, since Firehose bills in 5 KB increments.
• Put gateway endpoints (S3, DynamoDB) in front of NAT so NAT data-processing charges stay low.
• CloudWatch Logs Infrequent Access class for verbose application logs. Set retention to 30 days hot, with an S3 archive beyond that.
• At about $250k/year of spend, ask AWS about private pricing or an EDP.

Top 3 cost risks

9. Implementation Roadmap

Critical timing constraint: a first SOC 2 Type II report by Q2 2027 requires an observation period, typically at least 3 months for a first report (confirm with your auditor). That means controls must be operating by about January 4, 2027, roughly 14 weeks from today. This drives the sequencing below.

Rollback strategy by phase

Key dependencies and long-lead items

• Partner allowlisting of new egress IPs and the e-Rx vendor certificate re-issuance: start in week 1.
• Vendor BAAs (SMS/email, APM, support desk): legal lead time of 2–6 weeks.
• Pen-test firm booking: book now for February.
• Auditor selection: decide by November, and agree the observation window length in writing.

10. Explicit Trade-offs (ADR Summary)

Suggested next steps:

1. Answer the three blocking questions, especially the SLO scope for regional disasters.
2. Book the pen test and the auditor this month.
3. Start Phase 0 on Monday.

I can follow up with a detailed failover SSM Automation runbook, the audit-event library interface (TypeScript), or a fuller Terraform module layout.