A spec-compliant OTLP receiver, brought to you by a dead SSD

The dog ate my SSD

One night after school my laptop simply declined to boot. Not a blue screen, not a sad face with a QR code, those I could have worked with / had someone to blame. Instead I get a lovely message saying that there is no boot partition on my SSD. In disbelief, knowing that a mere 90 minutes ago the laptop was running fine and dandy, I tried restarting it about 12 times. That... did nothing. Clearly...

I spent about 6 hours trying to recover my data with any software I could find claiming it was "the best". Recovery tools, a live USB, command-line incantations I half understood, the usual stages of grief.

Denial: it's probably just a loose connection, I opened the laptop to clean it and remove + reinsert the SSD. Outcome = Pointless

Anger: WELL IT WORKED FINE BARELY 2 HOURS AGO SOMEONE MUST HAVE PUT A VIRUS ON MY MACHINE AS A JOKE. Yeah sure thing Mr. High Value Target.

Bargaining: Please Mr Gates I have a conference to attend and maybe 15h of uncommited code I will never complain again!

and eventually acceptance: I gave up and did a clean reinstall. It was almost midnight and I was at a loss for words (and for my data).

Here's the part where the story has a moral. Everything I had pushed survived: the repo was on GitHub, the blog was deployed, the architecture spec was committed. Everything that lived only on that disk: local notes, school projects, a folder containing about 21GB of mods named something like mods-backup-final-final(2) is gone, forever, may it rest. In the first post I wrote "committed early and often" as a productivity principle. Windows reinterpreted it as a survival strategy.

The reckoning

In post #1 there's a nice table. Week 12, October 5–11, is called "buffer: public demo, write-up, breathe." As I write this it is October 12 and the Datadog Summit in Paris is in three days. By the original plan I should be breathing. Oh sweet reader, I am not breathing. I probably look like one of the victims of the "Waterloo Effect".

What happened between July and now is a longer post for a marginally wiser man, but the short version is: life, school, and one (1) freshly formatted SSD. The honest scoreboard is that the code is at week 1 of a 12-week plan. This blog is called Building in Public, not Succeeding on Schedule in Public, so here it is in public: the timeline slipped, badly, and I reserve the right to retcon the milestone table and pretend it never happened. While ridicule and complaints are welcome, just make sure they're correctly addressed to spam@mrandrej.com.

What I refuse to do is show up with nothing. So the scope changed from "mini-Datadog" which was already impossible to "the walking skeleton, but done properly": OTLP logs in, query out, tests, CI. The thing post #1 said weeks 1–2 would exit with. It exits with it now. I will confess that I have tried improving my understanding of Otel in the meantime, partially for work and partially for this project. The main lesson I've learned is that there is quite a bit to know and quite a bigger bit that I do NOT know.

The OTLP receiver, as promised

Junkyard v2 does not embed the OpenTelemetry Collector. It implements the OTLP service interfaces directly using the generated stubs from go.opentelemetry.io/proto/otlp: a gRPC server on :4317 and an HTTP handler on :4318 that speaks both application/x-protobuf and application/json on POST /v1/logs. The reasoning is unchanged from the spec: embedding the Collector would be less code, and it would hide exactly the parts worth understanding. Which brings us to the two things I promised to explain.

What partial_success actually means. Every OTLP export response carries an optional partial_success field with a rejected_log_records count and an error message. It's how a server says "your request was valid, I accepted the batch, and I still dropped 83 of your 1,000 records, here's why." That is a completely different statement from rejecting the whole request with an error status, and conflating the two is how you lose data silently or retry storms are born. Most client SDKs log it and move on, which is exactly why the count has to be real. Junkyard's walking skeleton accepts or rejects whole batches, so on success it honestly reports zero rejected records - but the field is there, populated, and not decorative.

Backpressure is a feature you design, not a bug you fix. The queue is a fixed-capacity Go channel, and that is the entire queue. When it's full, the receiver doesn't block waiting for the disk to catch up: it tells the client to go away and come back later, RESOURCE_EXHAUSTED on gRPC, 503 with a Retry-After header on HTTP, exactly like the spec's retryable-error rules describe, and increments a dropped-batch metric so the loss is visible instead of silent. A slow disk can degrade Junkyard, but it can never stall every connected client behind one full buffer. In post #1 I called this "a lesson I paid for in advance." I have now paid for it twice, the second installment being one SSD.

Every request still travels the same path:

receiver → normalize → [bounded channel] → batcher → single storage writer

The batcher accumulates until 500 records or one second, whichever comes first, and hands one transaction to a single writer goroutine. SQLite's single-writer limitation stops being a problem you fight and becomes a property you design around. The store itself is the evolved v1 design: WAL mode, indexed on timestamp and service, with trace_id/span_id columns ready for the day traces exist. One fun gotcha: FTS5 isn't compiled into the SQLite driver on my fresh Windows install without a build tag, so full-text search quietly falls back to LIKE. Windows has now cost me one SSD and one full-text index. The score is not close.

And because v1's original sin was "no tests, no CI," v2 has table-driven unit tests for the pipeline, the receivers, and the store - including the shutdown-drain path, because a batcher that loses its final partial batch on exit is just a slower way to drop data - plus GitHub Actions running gofmt, go vet, go test, and go build on every push. Here's the whole thing working, unedited, one curl in and one curl out:

$ curl -X POST localhost:4318/v1/logs \
    -H "Content-Type: application/json" -d @smoke.json
{"partialSuccess":{}}

$ curl "localhost:8080/api/v1/logs?service=smoke"
[{"timestamp":"2026-10-12T20:26:40Z","severity_number":9,
  "severity_text":"INFO","body":"hello from smoke test",
  "service":"smoke"}]

Paris, in three days

So what do I bring to the Summit? Not the mini-Datadog, perhaps a hotdog if I'm feeling peckish. A walking skeleton that does one signal end to end and does it honestly, an architecture document I can defend, tests that were there from day one, and a very good story about why the commit graph has a three-month canyon in it. If you're a Datadog SWE and you're reading this, hello again, stranger. The plan changed; the point didn't.

Next up: metrics and the custom TSDB, on a new and significantly more honest timeline. And backups. So many backups.