Day 4 OTel-Shop: Chaos Injection and Real Unit Tests

Where We Left Off

Day 3 gave us a genuinely distributed checkout: order calls inventory, inventory reads PostgreSQL, payment gets charged. Everything green. But green-everywhere is exactly when you should get suspicious — the flow had never been seen failing, and not a single line of it was covered by a unit test. Day 4 fixed both.

What Got Built Today

Payment learned to fail on purpose. A tiny chaos package now sits inside the payment service. It reads three knobs from the environment: how often to wait before answering (in percent), how long that wait is (in milliseconds), and how often to outright fail. The important design choice: at 0% or 100% the behavior is completely deterministic, so it's testable — randomness only lives in the middle range. It also respects the request context, so a cancelled request doesn't keep it napping.

Then the fun part — flipping the switches in the real cluster:

• PAYMENT_ERROR_PERCENT=100 → checkout honestly reports payment failed (500) ✅
• Revert to the default 10% → checkout is paid again ✅

No code change, no redeploy — just an environment variable. That's the whole point of chaos knobs.

The whole flow got real tests. Thanks to yesterday's interfaces, fakes were cheap to write:

• Order's checkout tested with mock inventory and payment clients: success, bad input, inventory failure, payment failure, and insufficient stock — every branch.
• Order's actual HTTP clients tested against a fake server (httptest), so the real request/response code is exercised too.
• Inventory's handler tested with a fake store, and its database layer tested with sqlmock — including the "row not found" path — without ever touching PostgreSQL.
• Payment's chaos behavior tested deterministically (always fails / never fails / delays actually delay / cancelled context bails out) plus the handler around it.

The Coverage Table

The goal was 70% on every internal package. Actual numbers:

All comfortably above the bar — and every green test.sh run now has backup from 20+ unit tests.

Lessons Learned

• Chaos only works if it's deterministic at the edges. 0% and 100% must be promises, not probabilities — otherwise you can't test the chaos itself.
• Interfaces written "for later" pay off immediately. Every mock in today's tests existed because of interfaces drawn on Day 3, before any test needed them.
• A coverage gate finds real gaps. Watching the numbers forced proper client tests instead of stopping at handler-only coverage — the HTTP layer would have stayed untested otherwise.
• Keep the database out of unit tests. sqlmock simulates the exact scenarios (rows, no-rows, dead connection) in milliseconds, with zero setup and zero flakes.

Conclusion

Day 4 closes the quality-foundation chapter: the system can now be broken on demand, observed failing honestly, and every branch of the checkout flow is pinned down by tests. If tomorrow's changes regress something, the tests will say so before any human notices.

Next is the reason this whole lab exists: the OpenTelemetry SDK goes in, and the first real distributed traces appear in Jaeger. See you there.

Repo is here: https://github.com/stayrelevantid/otel-shop