Hari 43: External Secrets — AWS to K8s Sync

Day 31 deploy SecureBank dengan JWT_SECRET di K8s Secret (base64 di secret.yaml, committed ke git). Checkov CKV_K8S_35 sejak Day 32 flag "secret as env vars". Hari ini fix dengan External Secrets Operator (ESO) — sync secret dari AWS Secrets Manager ke K8s secara otomatis.

Secret Reference Pattern

Prinsip utama: git repo cuma berisi referensi (nama-nama), bukan nilai secret.

File	Git Status	Isi
aws-credentials.yaml	🔒 gitignored	AWS Access Key + Secret Key
secret-store.yaml	✅ committed	Referensi ke aws-credentials (nama saja)
jwt-secret.yaml	✅ committed	Referensi ke AWS secret securebank/jwt-secret (nama saja)
deployment.yaml	✅ committed	envFrom: securebank-jwt (nama K8s Secret)

Kalau repo leak, attacker tahu "ada secret bernama aws-credentials" tapi tidak tahu nilainya. Nilai ada di AWS (encrypted) dan K8s cluster (runtime, tidak di git).

Step 1: AWS Secrets Manager

$ aws secretsmanager create-secret \
  --name securebank/jwt-secret \
  --secret-string "$(openssl rand -hex 32)" \
  --region ap-southeast-1

ARN: arn:aws:secretsmanager:ap-southeast-1:683915449775:secret:securebank/jwt-secret-RTgjr9

$0.40/month per secret. Encrypted, access controlled, audit trail via CloudTrail.

Step 2: Install ESO

$ helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace

3 pods Running, 24 CRDs installed

Step 3: SecretStore + ExternalSecret

SecretStore — koneksi ke AWS Secrets Manager:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: securebank
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials  # Reference only
            key: access-key-id

ExternalSecret — declare: sync ini dari AWS:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: securebank-jwt
  namespace: securebank
spec:
  refreshInterval: 1h  # Auto-sync every hour
  secretStoreRef:
    name: aws-secrets
  target:
    name: securebank-jwt
    creationPolicy: Owner
  data:
    - secretKey: JWT_SECRET
      remoteRef:
        key: securebank/jwt-secret  # AWS secret name

ESO Sync: 20 Seconds

$ kubectl get externalsecret -n securebank
NAME             STORETYPE     STORE         REFRESH INTERVAL   STATUS         READY
securebank-jwt   SecretStore   aws-secrets   1h                 SecretSynced   True

$ kubectl get secretstore -n securebank
NAME          STATUS   CAPABILITIES   READY
aws-secrets   Valid    ReadWrite      True

$ kubectl get secret securebank-jwt -n securebank
NAME             TYPE     DATA   AGE
securebank-jwt   Opaque   1      20s  # Auto-created by ESO!

Deployment Update

# Before (Day 31-42):
envFrom:
  - secretRef:
      name: securebank-secrets  # Manual, base64 in git

# After (Day 43):
envFrom:
  - secretRef:
      name: securebank-jwt  # ESO-managed, synced from AWS

API Verification

$ curl http://localhost:9086/health
{"status":"healthy"}  # HTTP 200 — pods running with ESO secret

$ curl http://localhost:9087/balance
missing authorization header  # HTTP 401 — JWT middleware active

HTTP 401 dari /balance = JWT middleware active = JWT_SECRET berhasil di-load dari ESO-managed secret.

GitHub Push Protection: Real Security Win

Saat commit, GitHub Push Protection blocked the push — Checkov report JSON mengandung AWS keys (scanned dari aws-credentials.yaml yang ada di disk).

remote: - GITHUB PUSH PROTECTION
  Amazon AWS Access Key ID
  locations: checkov-k8s-eso-report.json:1

  Amazon AWS Secret Access Key
  locations: checkov-k8s-eso-report.json:1

Fix: Remove report dari commit, add to .gitignore. GitHub Push Protection = real security layer yang mencegah secret masuk git history. Ini bukan theoretical — ini actually worked!

Checkov: CKV_K8S_35 Trade-off

$ checkov -d k8s/ --check CKV_K8S_35
FAILED: "Prefer using secrets as files over secrets as environment variables"

CKV_K8S_35 masih FAILED — ESO fixes source (AWS vs git), bukan mount method (env vs file). App pakai os.Getenv("JWT_SECRET"), butuh env. Full fix: mount as file + modify app. Trade-off documented.

With skip: 102 passed, 0 failed.

Aspek	Before (Day 31)	After (Day 43)
Secret source	Git (base64 in secret.yaml)	AWS Secrets Manager (encrypted)
Git exposure	❌ Secret in git history	✅ No secret in git
Auto-rotation	❌ Manual	✅ refreshInterval: 1h
Mount method	envFrom	envFrom (unchanged)
CKV_K8S_35	FAILED	FAILED (env mount trade-off)

Pelajaran Hari Ini

Secret Reference Pattern. Git cuma berisi referensi (nama-nama). Nilai secret di AWS (encrypted) + K8s cluster (runtime). Kalau repo leak, attacker tahu nama tapi tidak tahu nilai.

ESO = decouple secrets from git. Source of truth di AWS. K8s Secret cuma cache yang auto-sync. refreshInterval: 1h = kalau AWS secret di-rotate, K8s auto-sync within 1 jam.

GitHub Push Protection actually works. Checkov report JSON mengandung AWS keys (scanned dari gitignored file yang ada di disk). Push Protection block push. Fix: remove report dari commit, gitignore. Real security layer, bukan theoretical.

CKV_K8S_35 = mount method, not source. ESO fixes source (AWS vs git), tapi CKV_K8S_35 flags env mount. Full fix: mount as file + app read file. Trade-off: app butuh os.Getenv().

CRD version evolution. Tutorial pakai v1beta1, chart terbaru serve v1 (GA). Selalu cek kubectl api-resources --api-group=external-secrets.io.

Repo

Semua code dan dokumentasi ada di: https://github.com/stayrelevantid/chalange-devsecops

Kesimpulan

External Secrets Operator berhasil sync JWT_SECRET dari AWS Secrets Manager ke K8s Secret dalam 20 detik. Secret Reference Pattern: git cuma referensi, nilai di AWS. GitHub Push Protection block Checkov report yang mengandung AWS keys — real security win. CKV_K8S_35 trade-off documented (source fixed, mount method unchanged). Besok Day 44 — AI Threat Modeling pada K8s.