Distributed execution: observe and experiment

The offline walkthrough exposes success, worker crash before external effects, and unknown external outcome. At each step read the owner, input, durable output, and possible failure before advancing.

Sequence diagram. Acceptance commits before 202; completion is a later event. Redelivery never grants a stale worker authority to overwrite newer state.

At six jobs/minute and 20 seconds/job, arrival rate is 0.1 jobs/s. The required average in-flight work in a stable regime is λW = 0.1 × 20 = 2 jobs. Two workers each able to finish one job in 20 seconds provide exactly six jobs/minute with no nominal headroom. Choose three for the illustrative discussion: capacity becomes nine jobs/minute and utilisation is about 67%, before dependency quotas or runtime variability. These are workload calculations, not measurements of the Python fixture.

If 12 jobs arrive together and three independent workers take exactly 20 seconds/job, four waves finish at 20, 40, 60, and 80 seconds. This ignores claim/commit overhead and external contention. A 90-second job deadline leaves only 10 seconds of margin for the last wave. A 30-second browser timeout should not govern this longer job’s lifetime; return the durable identifier and poll at a bounded interval with backoff instead.

Assuming five state writes per job, six jobs/minute yields about 30 writes/minute, or 0.5 writes/s, before polling and retries. One hundred clients polling every second would add 100 reads/s even when the worker throughput is small. Include polling load in capacity estimates; do not fix a status-query bottleneck by adding more generation workers.

Open the lease experiment. Its two controls are the lease interval (5–30 logical seconds) and crash position. A worker is assumed to stop at time zero; no real time elapses and no provider is called.

An actual worker that lives longer than its lease needs a bounded heartbeat/renewal policy. This slice instead rejects expired completions and uses explicit logical-clock experiments. A shorter lease can improve recovery time but can create false takeovers during pauses. The epoch protects state writes, not duplicate billing.

Run this stage's Python lab in your terminal to see each idea in the stage hold. It needs Python 3.10 or later and nothing else: the standard library and SQLite only.

From the root of your local clone of the repository, run:

python3 10_AI_System_Design/stages/02-distributed-execution/labs/test_jobs.py -v

-v prints one line per test, and each test name states one behaviour. Read a test, change one value in the lab code, predict which test fails, and run it again.

The tests create a temporary database, instantiate/recreate worker stores, race two claimers, expire a lease, attempt a stale completion, record an external start, exhaust safe retries, enforce a deadline, and reject an unapproved tool. They exercise real SQLite commits with a simulated clock. They do not start an HTTP job endpoint or run a network broker. Inspect jobs.py, change a lease/deadline, and explain which test should fail before rerunning it.

A model call started, its worker stopped answering, and its lease expired. May the replacement call the model immediately?

• At six jobs a minute and 20 seconds each, Little's law gives two jobs in flight. Two workers supply exactly six a minute with no headroom, while three give nine a minute at about 67% utilisation.
• Twelve simultaneous jobs on three workers finish in waves at 20, 40, 60 and 80 seconds, leaving 10 seconds of margin under a 90-second deadline. Return the job identifier and poll with backoff rather than tying the job to a 30-second browser timeout.
• Polling is load too. Five state writes per job give about 0.5 writes a second, but 100 clients polling every second add 100 reads a second, which more generation workers cannot fix.
• A shorter lease speeds recovery but risks false takeovers during pauses. A crash before an external effect allows redelivery once the lease expires, while a crash after the external-start marker leaves the job unknown.