Moses Adebayo ·

the bugs that don't throw

the empty-200 bug: status green, payload empty, no stack trace. why the hardest part of building AI systems is the invisible diplomacy between systems that were never introduced to each other.

engineering AI backend debugging distributed systems

the bugs that don’t throw

Schrödinger’s endpoint

there’s a specific flavor of bug i’ve been living in lately. request comes in, response goes out, status 200, everything green, and the payload is just… nothing. empty list where a list should be. no exception, no stack trace, no red text anywhere. the request is simultaneously fine and completely broken until you actually check, which is annoying, because computers are not supposed to make you do philosophy.

crashes are the easy bugs. i mean that with total sincerity, a crash is basically a gift. it hands you a line number and says “here, start here.” you fix it before your coffee gets cold.

the empty-200 bug doesn’t do that. every system involved will swear, under oath, that it did its job correctly. and it did! that’s the annoying part. the bug isn’t inside any single system. it’s in the seam between two systems that each have their own private definition of what an id, or a key, or “this user” actually refers to, and nobody ever forced those definitions to shake hands. every component is locally correct and the whole thing is globally wrong. distributed systems’ answer to “the operation was a success but the patient died.”

you will not find this by reading code. you can read a function for an hour, conclude it’s correct, and be right, given its assumptions. the bug lives one layer up, in the mismatch between assumptions. the only way in is to close the editor and go look at the actual data. not the schema, the schema is a promise, not a fact. the real rows, right now, cross-checked across every store that touches the thing that’s broken. more than once recently the entire fix was just: two systems had quietly drifted apart on what a piece of data meant, so i sat them down and made them agree again. no clever algorithm. glorified marriage counseling for databases.

worth saying because this is the part of “building ai systems” nobody talks about at meetups. the discourse is 90% which model, which prompt, which framework, because those are the parts you can put in a slide. nobody’s giving a talk titled “i spent tuesday afternoon querying three data stores to prove the bug wasn’t where i thought it was.” but that afternoon is most of the job.

getting a model to say something plausible in a demo is, at this point, genuinely not hard. getting that same system wired into something real, with real auth, real legacy data with opinions of its own, real edge cases nobody documented because nobody knew they existed, that’s a different discipline. less “intelligence,” more diplomacy between systems that were never introduced to each other. and the good version of that work is invisible by design. nobody notices the seam that didn’t fail.

which is also, i’d guess, why it looks from a distance like something you could knock together over a weekend with a no-code tool. the demo layer really is that easy now. it’s the part underneath the demo, the part that makes it survive contact with actual users and actual data, that isn’t. that part doesn’t compress into a screenshot. it just quietly decides whether the thing works or not.

so if your week is more data archaeology than prompt-crafting, you’re not doing it wrong. you’re doing the part that counts. the seam that holds is the whole job — it just never makes the screenshot.

Previous one year in.