Hari 45: Fase 3 Complete — K8s & Runtime Security

Fase 3 selesai! 15 hari (Day 31-45) K8s & Runtime Security. Dari cluster kosong ke 6 defense layers. Dari Kubesec score 0 ke 12. Dari 0 rules ke 29 Falco rules. Dari base64 secret di git ke AWS Secrets Manager sync. Ini retrospective.

Metrics: Before vs After

Metrik	Before (Day 31)	After (Day 45)
Kubesec score	0	12
Checkov K8s passed	85	102
Checkov K8s failed	20	0
Trivy K8s findings	16	0
SecurityContext properties	0	16
NetworkPolicies	0	3
RBAC resources	0	3
Gatekeeper CRDs	0	17
Falco rules	0	29 (25 default + 4 custom)
Falco alerts fired	0	6 (attack simulation)
Defense layers	0	6
Secrets in git	1 (base64)	0 (ESO-managed)

6 Defense Layers

Layer	Tool	Day	What It Does
Admission Control	OPA Gatekeeper	34-36	Block pods without resource limits
Image Security	Distroless + Cosign	18-19	No shell, signed images, 7.97MB
Network Isolation	NetworkPolicy	37	Default deny + whitelist kube-system
Access Control	RBAC	38	Dedicated SA, get configmap only
Runtime Detection	Falco + Falcosidekick	39-42	29 rules, webhook alert routing
Secret Management	External Secrets Operator	43	AWS sync, no secrets in git

Top 5 Lessons

1. Defense in depth works. Day 41 proven: distroless blocks shell (4 attempts fail), Falco detects (6 alerts), NetworkPolicy blocks egress (wget refused). 5/6 layers tested in real attack. No single point of failure.

2. Distroless = prevention > detection. 4 shell exec attempts all fail. Attacker dengan RCE tidak bisa exec shell untuk pivot. Ini security win paling konkret. Falco detection penting, tapi prevention lebih baik.

3. Secret Reference Pattern + GitHub Push Protection. Git cuma berisi referensi (nama), nilai di AWS. GitHub Push Protection actually blocked real AWS keys di Checkov report. Real security layer, bukan theoretical.

4. k3d tracepoint limitation is PARTIAL. Initial assumption "total failure" WRONG. execve + openat work, connect missing. 3/4 custom rules fired. eBPF warm-up ~25 min. Always test, jangan assume.

5. AI sebagai threat modeler. glm-5.2 analyze cluster config, identify 3 attack paths dengan CVSS + MITRE ATT&CK. AI actually read YAML dan produce analysis. Skill threat-modeler dari ai-assistant-brainstorm.md = real capability.

Best Achievement: Day 41 Attack Simulation

Moment paling satisfying di Fase 3. Setelah 11 hari setup hardening, akhirnya test dengan real attack:

Attacker pod (alpine tagged securebank:attacker)
  → sh -c "cat /etc/passwd"
  → Falco: 6 alerts fired (1 WARNING + 3 CRITICAL + 2 NOTICE)
  → NetworkPolicy: both wget attempts "Connection refused"
  → Falcosidekick: 5/6 alerts forwarded to WebUI
  → Distroless proof: 4 shell exec attempts fail on real pods

Semua 5 defense layers proven working together. Bukan theoretical — actually worked.

Biggest Challenges

Challenge	Solution
k3d tracepoint limitation	Document as "total", Day 41 correction: PARTIAL. Test, don't assume
n8n Docker image pull timed out	Pivot ke Python webhook receiver. Pragmatic DevSecOps
GitHub Push Protection blocked push	Checkov report contained AWS keys. Remove, gitignore. Real security win
ESO CRD v1beta1 not found	Chart terbaru serve v1 (GA). Update apiVersion
rakkess repo 404	Now access-matrix krew plugin. Always check repo status

Start / Stop / Continue

Start (Fase 4):

• DefectDojo integration untuk centralized vulnerability tracking
• Slack alerting — CRITICAL Falco alerts route ke team channel
• Chaos security engineering — test defense in depth under stress

Stop:

• Assume k3d limitation without testing
• Secrets in git (base64)
• Force uncooperative tools (n8n timeout → pivot)

Continue:

• Defense in depth pattern (6 layers)
• Default deny + whitelist NetworkPolicy
• Secret Reference Pattern (git = references, AWS = values)
• values.yaml untuk Helm (reproducible)
• AI-assisted security analysis
• Pragmatic pivoting

Connection to Fase 4

• Falco alerts + webhook → Slack #security-alerts (Day 48)
• K8s cluster (hardened) → red team simulation target (Day 52)
• Gatekeeper policies → chaos security engineering (Day 54)
• All scan results → DefectDojo (Day 47)
• AI threat model (3 attack paths) → red team scenarios
• Defense in depth (6 layers) → tested under chaos

Repo

Semua code dan dokumentasi ada di: https://github.com/stayrelevantid/chalange-devsecops

Kesimpulan

Fase 3 complete. 15/15 hari. 6 defense layers. Kubesec 0→12, Checkov 20→0, Falco 29 rules, 6 alerts fired. Defense in depth proven. Distroless = prevention win. GitHub Push Protection = real security. AI threat modeler = unique capability. 45/60 challenge complete, 3/4 fase selesai. Next: Fase 4 — Vulnerability Management & Red Teaming!