Day 3 OTel-Shop: Services Start Talking via HTTP

Where We Left Off

The last two days gave us a cluster and three services that knew how to say "ok" on /health — but they didn't really talk. Inventory answered with a hardcoded number, and checkout never phoned anyone. Cute, but not a system. Day 3's job: make them actually communicate.

What Got Wired Today

Two chunks of work, and they finally click together.

Inventory meets the database. Inventory now opens a real connection pool to PostgreSQL (pgx under the hood) and looks up stock by id. I hid the database behind a tiny StockStore interface so the handler doesn't care where the data lives — handy later for tests. Responses are honest now: found → 200 with the number, missing → 404, anything worse → 500.

Checkout learns to orchestrate. Order grew two HTTP clients — one for inventory, one for payment — both behind interfaces so they can be faked in tests. The checkout handler now does a proper little dance:

• Validate the request (no empty item, qty must be positive).
• Ask inventory for stock.
• Complain if there isn't enough.
• Charge payment for qty * 10 (price is a flat 10 per unit for now).
• Only then return a fresh order id with status paid.

When a neighbor fails, checkout stays honest instead of pretending: a downstream error becomes a 500 with a clear message like inventory failed or payment failed.

How it behaved end to end (straight from the host, through the cluster):

• POST /checkout for A123 ×1 → {"order_id":"O-...","status":"paid"} ✅
• Unknown product → 500 inventory failed ✅
• C123 ×999 (only 5 in stock) → 400 insufficient stock ✅
• Garbage request → 400 ✅

And I finally added a proper test.sh that runs all nine of these checks and exits non-zero if anything breaks — so "does it still work?" is one command now, not a memory test.

The Gotcha

First attempt at local testing looked green... but for the wrong reasons. My local services were trying to bind ports 18080–18082, which the cluster already occupies as NodePorts on my laptop. So my curl requests silently hit the old stub pods, not my new code. The fix: run local services on throwaway ports and point the order client at those. Silly, but it's exactly the kind of "it works on my machine because it's actually the cluster" trap worth internalizing.

Lessons Learned

Four things stuck this time:

• Hide infrastructure behind interfaces early. The StockStore and client interfaces cost almost nothing today and are what make the flow testable tomorrow.
• A failed call is data, not a mystery. Returning a clear inventory failed / payment failed turned a confusing 500 into something you can act on.
• Test like a real client, on real ports. My port-clash slip proved that "it returned 200" means nothing if you're not sure which process answered.
• One script beats a checklist in your head. test.sh replaced five manual curls and already caught me thinking I was done when I wasn't.

Conclusion

Day 3 closes with a checkout flow that's genuinely distributed: one request fans out to inventory and payment, talks to a real database, and reports honest results. The architecture finally earns the "microservices" label.

Next: inject some chaos into payment (random delays and errors) and start writing real unit tests behind those interfaces. See you then.

Repo is here: https://github.com/stayrelevantid/otel-shop