The Trampled Basket

GlagalCloud was secure, scalable, and profitable. Glagalbagal had, by any reasonable measure, built something remarkable. He had also, without quite noticing it, built something fragile in a new way — a way that only manifested when the system was under load.

The Double Update

The monthly cycle was the busiest time. Reports arrived from all locations, index updates were triggered, and multiple velociraptors processed records simultaneously. On one particularly busy day, two velociraptors — Plekva and a temporary worker named Grontch — were both assigned tasks that required modifying the same record.

Plekva was updating Bretchka's herd count for Location 1: adding fourteen newborn calves to the existing count of two hundred and eight. Grontch was updating the same record's health status field: changing the health code from "healthy" to "quarantined" after a disease sighting.

The record sat on Bretchka's shelf. Both velociraptors approached it. Plekva read the record: count 208, health "healthy." At approximately the same moment, Grontch read the same record: count 208, health "healthy."

Plekva computed the new count: 208 + 14 = 222. She wrote the updated record back to the shelf: count 222, health "healthy."

Grontch, working from his copy (which still showed count 208), changed the health field. He wrote the updated record back to the shelf: count 208, health "quarantined."

Grontch's write overwrote Plekva's. The final record showed count 208, health "quarantined." The fourteen calves had vanished. Plekva's work was silently lost.

Blortz: Plekva added fourteen calves. The record shows no change in count. The calves exist in the field but not in the records. This is precisely how Glagalbagal's trouble started in Part 1.

Glagalbagal: Two velociraptors modified the same record at the same time. The second one overwrote the first.

Blortz: The result depended on who finished last. If Plekva had been slower, the count would be correct and the health status would be wrong. The outcome is determined by timing, not by logic.

The Race

This was a race condition. Two operations, each correct in isolation, produced an incorrect result because they overlapped in time. The problem was not that either velociraptor did something wrong — both followed their instruction tablets exactly. The problem was that the instruction tablets assumed each velociraptor had exclusive access to the record while working on it. When that assumption was violated, the result was unpredictable.

The Lock

Glagalbagal's solution was borrowed from physical intuition. If a velociraptor was working on a record, no other velociraptor should be allowed to touch it until the first one was done. He introduced the busy stone — a small red pebble placed on top of a record to indicate that it was currently being modified.

The protocol:

Before modifying a record: check for a busy stone. If present, wait. If absent, place a busy stone, then read the record. After modifying: write the updated record, then remove the busy stone.

While the busy stone was in place, any other velociraptor that needed the same record would see the stone and wait. When the stone was removed, the next velociraptor could place its own busy stone and proceed.

This was a lock. The busy stone locked the record for exclusive access. Only the velociraptor holding the lock could read or write. Others were blocked until the lock was released.

Grontch: What if I need to update the health status urgently? The count update might take ten minutes.

Glagalbagal: You wait ten minutes.

Grontch: The disease might spread in ten minutes.

Glagalbagal: Then we need to think about whether locking the entire record is too coarse. Perhaps we should lock individual fields.

Fine-Grained Locks

Blortz proposed locking at the field level rather than the record level. Instead of one busy stone per record, each field (count, health status, animal type, etc.) had its own lock position. A velociraptor updating the count would lock the count field. A velociraptor updating the health status would lock the health field. They could work simultaneously, as long as they were modifying different fields.

This reduced waiting time significantly. Most updates touched only one or two fields. Conflicts — two velociraptors needing the exact same field — were rare.

But fine-grained locks created their own complexity. More lock positions to manage. More opportunities for a velociraptor to forget to remove a lock (the "abandoned busy stone" problem — a velociraptor gets called away for lunch and leaves the stone in place, blocking everyone). And, as the next situation would demonstrate, a subtle new kind of failure.