{"authors":[{"name":"Juno Vale"}],"description":"Notes on the BEAM, backpressure, and the long tail of distributed systems.","feed_url":"https://cherrybomb.dev/t/kage/feed.json","home_page_url":"https://cherrybomb.dev/t/kage/","items":[{"content_html":"

A year ago I put property tests into a codebase that already had good\nexample-based coverage, mostly to find out whether the enthusiasm was\nwarranted. Here is what they actually caught.

\n

The score

\n

Eleven genuine defects. Nine of them were in code that parsed, normalised, or\nmerged data — the boring seams between systems. Two were in domain logic. None\nwere in the places I expected.

\n

The single most productive property was also the dullest:

\n
elixir
property "a document survives a round trip" do\n
  check all document <- document_generator() do\n
    assert document == document |> Codec.encode() |> Codec.decode!()\n
  end\n
end\n
\n

That one found four bugs on its own: an empty tag list that came back as nil,\na timestamp that lost microseconds, a nested map whose key order the encoder\ndepended on, and a Unicode normalisation difference that only appeared for\nstrings containing combining characters.

\n

Every one of those would have been possible to write as an example test. The\npoint is that nobody did, for years, because nobody thought of them.

\n

The properties worth writing

\n

Four shapes covered nearly everything useful:

\n

Round trips. decode(encode(x)) == x. Cheap, and it finds encoder bugs\nexample tests never reach.

\n

Invariants. Something that must be true of every output, regardless of\ninput. Ours: a stock projection never disagrees with a fold over the same\nevents.

\n

Oracles. A second, slower, obviously correct implementation to compare\nagainst. We had a naive O(n²) deduplicator; the fast one had a bug at exactly\nthe block boundary.

\n

Idempotence. f(f(x)) == f(x) for anything that normalises. This found a\nsanitiser that stripped one layer of nesting per call.

\n

Where they were not worth it

\n

I wrote properties for our HTTP handlers and deleted them a month later. The\ngenerators were more code than the handlers, most of the interesting behaviour\nwas in the dependencies, and every failure took twenty minutes to interpret.\nExample tests read better there and caught the same things.

\n

I also stopped trying to shrink my way to understanding. stream_data's\nshrinking is good, but a minimal counterexample for a complex generator is\nstill a puzzle. Now, when a property fails, my first move is to write the\nshrunk case down as an ordinary test with a name. The property found it; the\nexample test explains it.

\n

That split — properties to search, examples to document — is the habit I would\nkeep if I had to give up everything else from this year.

","date_published":"2026-05-06T00:00:00Z","id":"https://cherrybomb.dev/t/kage/a-year-of-property-testing/","summary":"What a year of stream_data actually caught, and the properties that turned out to be worth writing.","tags":["testing","property-testing","elixir"],"title":"A Year of Property Testing","url":"https://cherrybomb.dev/t/kage/a-year-of-property-testing/"},{"content_html":"

When I join a BEAM codebase, the first file I open is application.ex. Not the\nrouter, not the schema — the supervision tree. It is the only place where the\nsystem states, in code, what it thinks should happen when things break.

\n
elixir
def start(_type, _args) do\n
  children = [\n
    Ledgerbeam.Repo,\n
    {Phoenix.PubSub, name: Ledgerbeam.PubSub},\n
    {Finch, name: Ledgerbeam.HTTP},\n
    Ledgerbeam.Inventory.Supervisor,\n
    Ledgerbeam.Ingest.Pipeline,\n
    LedgerbeamWeb.Endpoint\n
  ]\n
\n
  Supervisor.start_link(children, strategy: :one_for_one, name: Ledgerbeam.Supervisor)\n
end\n
\n

Six lines, and each one is a claim.

\n

What the order tells you

\n

Children start in order and stop in reverse. So this tree claims the endpoint\nshould be the last thing up and the first thing down — traffic only arrives\nonce the repo, the pipeline, and the domain supervisors are ready, and traffic\nstops before they go away.

\n

That is almost always what you want, and it is worth checking, because the\nfailure mode is subtle: put the endpoint too early and you accept requests\nagainst a half-built system for a few hundred milliseconds on every deploy.

\n

What the strategy tells you

\n

:one_for_one says these six things are independent — if the ingest pipeline\ndies, the endpoint should carry on. Is that true? For us it was: the site stays\nup and serves reads while ingestion restarts.

\n

But look for a :one_for_all and ask what it is protecting. It says the\nchildren share state that cannot survive one of them restarting, which is a\nreal thing, and also a design smell worth a second look.

\n

:rest_for_one is the interesting one: it says there is a dependency order\nhere, and everything after this child needs it. That is a map of your real\ncoupling, drawn by someone who had to think about it.

\n

Where the tree lies

\n

The tree is a set of beliefs, and outages are where the beliefs were wrong. Two\nthat catch people:

\n

A process that restarts fine but reconnects slowly. The supervisor's job is\ndone in microseconds; the connection pool underneath takes eight seconds. The\ntree looks healthy and the system is not.

\n

restart: :temporary on something that matters. It will not come back, ever,\nand the supervisor will not tell you. This is correct for a one-shot task and\nquietly catastrophic on a consumer.

\n

Neither shows up in the diagram. Both show up in :observer, and in the first\nfive minutes of an incident, which is why I read the tree before I read\nanything else.

","date_published":"2026-01-20T00:00:00Z","id":"https://cherrybomb.dev/t/kage/reading-a-supervision-tree-like-a-map/","summary":"A supervision tree is a diagram of what your system believes about failure. Most outages are places where that belief was wrong.","tags":["elixir","otp","operations"],"title":"Reading a Supervision Tree Like a Map","url":"https://cherrybomb.dev/t/kage/reading-a-supervision-tree-like-a-map/"},{"content_html":"

Every so often I redo the string-date-parsing kata. Not because I have\nforgotten how to parse a date, but because it is the shortest exercise I know\nthat punishes vagueness.

\n

The brief: given a string, return a date. Start with "2025-10-15".

\n
elixir
def parse(input), do: Date.from_iso8601(input)\n
\n

Four minutes in, you are done. Then the examples arrive.

\n

The examples

\n
"2025-10-15"     the easy one\n
"15/10/2025"     day first\n
"10/15/2025"     month first\n
"15 Oct 2025"    names, abbreviated\n
"October 15th"   no year, ordinal suffix\n
"yesterday"      relative to what?\n
"" and nil       the caller has a bug\n
\n

Nothing here is hard. What is hard is that "01/02/2025" is genuinely\nambiguous, and you cannot resolve it inside the parser. It is January 2nd or\nFebruary 1st depending on a fact the string does not contain.

\n

The kata's real lesson is that this is where the design happens. Every attempt\nI make eventually arrives at the same shape: the ambiguity becomes a parameter.

\n
elixir
@type locale_order :: :day_first | :month_first\n
\n
@spec parse(String.t(), keyword()) :: {:ok, Date.t()} | {:error, term()}\n
def parse(input, opts \\\\ []) do\n
  order = Keyword.get(opts, :order, :day_first)\n
  today = Keyword.get(opts, :today, Date.utc_today())\n
\n
  input\n
  |> String.trim()\n
  |> attempt([&iso/1, &numeric(&1, order), &named/1, &relative(&1, today)])\n
end\n
\n

Two details that took me several attempts to arrive at:

\n

today is an argument. A parser that reads the clock cannot be tested for\n"yesterday" without either freezing time or accepting flaky tests. Passing the\ndate in makes the whole thing a pure function, and the tests get boring.

\n

The strategies are a list. Each one returns {:ok, date} or :error, and\nattempt/2 walks them in order. Adding a format is adding a function, and the\nprecedence is visible in one place rather than distributed across a cond.

\n

Why it stays useful

\n

Most katas train you to write code. This one trains you to notice that the\nrequirements are underspecified, and to put the missing decision somewhere the\ncaller can see it. That skill transfers to almost everything, which is more\nthan I can say for FizzBuzz.

","date_published":"2025-10-15T00:00:00Z","id":"https://cherrybomb.dev/t/kage/the-kata-i-keep-coming-back-to/","summary":"Parsing dates from strings looks trivial for about four minutes, which is what makes it a good exercise.","tags":["testing","kata","elixir"],"title":"The Kata I Keep Coming Back To","url":"https://cherrybomb.dev/t/kage/the-kata-i-keep-coming-back-to/"},{"content_html":"

I came to event sourcing expecting an audit log and got something more useful:\na reason to delete about a third of the inventory service.

\n

The code that went away

\n

The old service had a table of stock levels and a great deal of code protecting\nit. Compensating updates when a shipment was cancelled. A nightly reconciliation\njob. A stock_adjustments table nobody could explain, holding rows that existed\nonly to correct earlier rows.

\n

All of that was there because the current level was the only thing we stored.\nWhen it went wrong, we had no way to ask how, so we wrote code to guess.

\n

Storing decisions instead of state removed the guessing:

\n
elixir
defmodule Inventory.Events do\n
  defstruct StockReceived, [:sku, :quantity, :received_at, :purchase_order]\n
  defstruct StockPicked, [:sku, :quantity, :picked_at, :order_id]\n
  defstruct StockWrittenOff, [:sku, :quantity, :reason, :approved_by]\n
end\n
\n

The level is now a fold:

\n
elixir
def level(sku) do\n
  sku\n
  |> Store.stream()\n
  |> Enum.reduce(0, fn\n
    %StockReceived{quantity: n}, level -> level + n\n
    %StockPicked{quantity: n}, level -> level - n\n
    %StockWrittenOff{quantity: n}, level -> level - n\n
  end)\n
end\n
\n

The reconciliation job had nothing left to reconcile. The adjustments table\nbecame a write-off event with a reason and an approved_by, which is what\neveryone had wanted from it in the first place.

\n

What it did not fix

\n

It is worth being clear about what event sourcing cost us, because the posts\nthat skip this part are the ones that get people into trouble.

\n

Versioning is forever. The moment an event is persisted, its shape is a\npublic API. We now write upcasters, and we treat an event struct with the same\ncare as a database migration.

\n

Projections drift. Every read model is eventually consistent with the\nstream, and every part of the UI that pretended otherwise had to learn to say\n"pending".

\n

Streams grow. A fold over five years of movements for a fast-moving SKU is\nnot free. Snapshots solve it, and snapshots are cache invalidation wearing a\nhat.

\n

Those are real costs. I would pay them again for the deletions alone. The\nservice went from a thing we were afraid to touch on a Friday to one where the\nquestion "how did this SKU get to negative four" has a literal answer.

","date_published":"2025-07-09T00:00:00Z","id":"https://cherrybomb.dev/t/kage/event-sourcing-taught-me-to-delete-code/","summary":"The best thing event sourcing did for our codebase was make a whole category of defensive code obviously unnecessary.","tags":["event-sourcing","design","elixir"],"title":"Event Sourcing Taught Me to Delete Code","url":"https://cherrybomb.dev/t/kage/event-sourcing-taught-me-to-delete-code/"},{"content_html":"

The received wisdom about NIFs is that they can bring down the entire VM, which\nis true, and that you should therefore avoid them, which does not follow.

\n

We needed SimHash over a few million documents. In pure Elixir it was about\n40ms per document, which was not going to work. In Rust, via\nRustler, it was under a millisecond.\nThat is the kind of gap worth taking a risk for — provided you know which risks\nyou are taking.

\n

Rule one: never block a scheduler

\n

A NIF that runs longer than about a millisecond starves the scheduler it runs\non. The whole preemption model of the BEAM assumes it can take a process off a\ncore; a native function ignores that assumption entirely.

\n

Rustler makes the fix a single annotation:

\n
rust
#[rustler::nif(schedule = "DirtyCpu")]\n
fn simhash(text: &str) -> u64 {\n
    hash_tokens(tokenize(text))\n
}\n
\n

Dirty schedulers exist precisely for work that cannot be preempted. Use them\nfor anything you have not personally measured under a millisecond, and measure\nbefore you decide you are the exception.

\n

Rule two: do not panic

\n

A Rust panic across the NIF boundary takes the VM with it. Every fallible path\nmust return an error term instead:

\n
rust
#[rustler::nif(schedule = "DirtyCpu")]\n
fn parse(input: &str) -> Result<Document, Error> {\n
    serde_json::from_str(input).map_err(|e| Error::Term(Box::new(e.to_string())))\n
}\n
\n

Then the Elixir side gets an ordinary tagged tuple, and an ordinary supervisor\nhandles it:

\n
elixir
case Native.parse(payload) do\n
  {:ok, document} -> {:ok, document}\n
  {:error, reason} -> Logger.warning("unparseable payload: #{reason}"); :skip\n
end\n
\n

Deny panics at the crate level and the compiler helps you keep the rule:

\n
toml
[lints.clippy]\n
unwrap_used = "deny"\n
expect_used = "deny"\n
\n

Rule three: keep the boundary boring

\n

The NIF should do one computational thing and hold no state. No global\nmutable structures, no threads you spawn yourself, no callbacks back into\nElixir. Everything that needs supervision, retries, or lifecycle belongs on the\nBEAM side where those things already work.

\n
\n\"Two\n
The whole architecture, honestly.
\n
\n
\n

Where the fear lives

\n

Every NIF horror story is a violation of one of the three rules. Check the\nstory against the rules before checking your code against the story.

\n
\n

Three rules, and the fear mostly goes away. What is left is a normal\nengineering tradeoff: a build dependency and a compile step, in exchange for\ntwo orders of magnitude.

","date_published":"2025-04-22T00:00:00Z","id":"https://cherrybomb.dev/t/kage/a-rust-nif-without-the-fear/","summary":"A NIF can take down the whole VM. Here is the small set of habits that keep that from being your problem.","tags":["elixir","rust","nif"],"title":"A Rust NIF Without the Fear","url":"https://cherrybomb.dev/t/kage/a-rust-nif-without-the-fear/"},{"content_html":"

We spent a week tuning a Broadway pipeline that kept falling behind, and the\nfix turned out not to be a number at all.

\n

The pipeline ingested webhook deliveries from a payments provider. Under normal\nload it kept up comfortably. During a merchant's flash sale it did not, and the\nqueue depth climbed until the delivery provider started retrying, which added\nload, which climbed the queue further.

\n

The instinct is to add concurrency:

\n
elixir
def start_link(_opts) do\n
  Broadway.start_link(__MODULE__,\n
    name: __MODULE__,\n
    producer: [module: {SQSProducer, []}, concurrency: 4],\n
    processors: [default: [concurrency: 80]]\n
  )\n
end\n
\n

Eighty processors did make the queue drain. It also pushed the database to the\npoint where unrelated requests started timing out. We had not removed the\nbottleneck, we had moved it somewhere with worse failure behaviour.

\n

The question nobody had asked

\n

What should happen when deliveries arrive faster than we can durably record\nthem?

\n

There are only a few honest answers, and each of them is a product decision:

\n\n

We picked the last one, and the pipeline stopped being a tuning problem:

\n
elixir
@impl true\n
def handle_message(_processor, message, _context) do\n
  message\n
  |> Message.update_data(&Delivery.parse!/1)\n
  |> Message.put_batcher(:record)\n
end\n
\n
@impl true\n
def handle_batch(:record, messages, _batch_info, _context) do\n
  messages\n
  |> Enum.map(& &1.data)\n
  |> Deliveries.insert_all()\n
\n
  messages\n
end\n
\n

Enrichment moved to a separate pipeline reading from the recorded rows, with\nits own concurrency and its own failure domain. The write path got simple\nenough that eight processors outperformed the eighty we had before.

\n

What the numbers were hiding

\n

concurrency: looks like a performance knob and behaves like a policy. Setting\nit high says "I would rather overwhelm my dependencies than fall behind."\nSetting it low says the opposite. Neither is wrong, but you should be able to\nsay which one you meant, and why, without opening the dashboard.

\n

The BEAM makes it unusually cheap to express any of these answers. It does not\ntell you which one your product needs.

","date_published":"2025-02-11T00:00:00Z","id":"https://cherrybomb.dev/t/kage/backpressure-is-a-product-decision/","summary":"Every queue that fills up is answering a question somebody should have answered on purpose.","tags":["elixir","broadway","backpressure"],"title":"Backpressure Is a Product Decision","url":"https://cherrybomb.dev/t/kage/backpressure-is-a-product-decision/"}],"title":"Juno Vale","version":"https://jsonfeed.org/version/1.1"}