Parallelization: sectioning and voting

What you'll learn

• Explain how a graph starts branches together, waits for all of them and merges their writes.
• Calculate a fan-out's wall time from its branch durations, with and without a concurrency cap.
• Diagnose a lost or rejected write when two branches update the same state key.
• Choose between sectioning and voting, and set a vote threshold knowingly.

Chaining runs steps one after another, and routing runs one of several. Parallelization runs all of a fixed set of branches at once and merges what they return.

A Larkspur reply about an outage needs three pieces of evidence: the handbook's outage policy, the status page, and the customer's account. None of the three lookups needs another's result, yet a chain would run them back to back. Running them together cuts the wait to the slowest one, provided the merge is defined.

Fan-out, fan-in and the sync point

In LangGraph the graph and its state schema decide both the scheduling and the merging:

• Fan-out: several edges leave one node, and their targets run in the same step, concurrently.
• Fan-in: several edges enter one node, and it waits until every one of them has finished.
• Sync point: when one path is longer (A to B to X, beside A to C), an edge from the list of both endpoints into the next node makes it wait for the longer path explicitly.

The lesson's sectioning graph writes a story's characters, setting and premise in parallel, then synthesizes an introduction:

Figure 1: The recorded sectioning run. Each branch is labelled with the time it finished.

Two branches, one key

The merge needs one more rule. A reducer says how to combine a node's update with the value already in a state key. Branches that each own a key, as the three story sections do, need none. Branches that write the same key in the same step do.

We ran a model-free fan-out in which a handbook branch and an account branch each append their name to one shared list, in the LangGraph version this course pins (1.2.18):

Table 1: The same two-branch graph with and without a reducer.

We call this failure colliding writes. Here it fails loudly, which is the useful outcome; the fix is to declare the reducer on any key that more than one parallel branch writes.

The critical path

Wall time follows the longest path through the graph, not the sum of the steps. The recorded timing runs measure it first with nodes that sleep 0.5 s each, then with real LLM calls:

Table 2: Recorded wall times.

The synthesis step overlaps nothing, because it needs every section. So a fan-out costs the step before, plus the slowest branch, plus the merge: 7.0 s for the slowest section plus 3.6 s of synthesis gives the 10.6 s total.

Concurrency caps and submission order

Production adds a limit the toy graphs lack. The API behind the branches is rate-limited, so you cap how many branches run at once. In LCEL the cap is a setting on the call: six 0.5 s branches took 0.51 s unbounded and 1.51 s two at a time, three rounds of two.

Under a cap, order starts to matter. The interactive lab schedules the recorded section durations the way a thread pool does: branches are submitted in order and each starts on the first free slot. On the story sections:

Table 3: The lab's schedule of the recorded durations.

A cap of 2 costs nothing as written: the 7.0 s setting branch gets a slot at once, and the premise runs after the characters in the other slot, finishing at 6.3 s. Submit the fast branches first and the setting waits 1.6 s for a slot, which the whole request pays. We call that the queued straggler: the slowest branch held behind faster ones. Start long branches first when you know their durations. The lab adds the printed one-decimal times, so its sequential total shows 13.3 s where the recorded run printed 13.4 s.

The same fan-out mechanics serve two different jobs, and they buy different things:

Table 4: Two uses of one mechanism.

The recorded security vote

The voting graph gives each reviewer one concern (injection, secrets handling, input validation), and no reviewer sees another's vote. Each returns a structured vote, flagged or not, with a reason. The aggregator blocks the code when at least 2 of 3 reviewers flag it:

Table 5: The two recorded reviews.

Setting the threshold

The threshold trades false alarms against misses. On the two recorded snippets every setting gives the same verdict, because the votes were unanimous:

Table 6: Thresholds on the recorded votes. Only split votes would separate them.

The unsafe run also shows a planted weakness. The validation reviewer flagged the SQL injection, which is the injection reviewer's concern, so the votes are not fully independent. We call this correlated reviewers: two flags can be one finding counted twice. Allow for it before lowering the threshold, and test thresholds on snippets where the reviewers disagree.

Both uses appear in one Larkspur reply:

1. Sectioning for evidence. Fan out to the handbook index, the status page and the account. The answer waits for the slowest source, so each branch gets a timeout and a fallback value, and each result keeps its source label so the reply can cite it.
2. Voting before sending. Three narrow reviewers check the draft: does it promise a refund the policy does not allow, does it quote the right refund window, does it commit to a support response time. A threshold over three narrow checks is easier to tune than one judge asked about everything.

Lesson 4 of T7.3 removes the last fixed piece: an orchestrator decides at run time how many branches there are.

• One run per graph. The LLM timings in Tables 2 and 3 come from one recorded run; another run will finish its branches at different times.
• Unanimous votes only. Neither recorded snippet split the reviewers, so the threshold's effect is reasoned, not measured.
• Left out: per-branch timeouts and fallback values. Add them when a branch calls a service that can hang.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A fan-out costs the step before, plus the slowest branch, plus the merge. Declare a reducer on any state key that parallel branches share, or the writes collide.
• Rate limits force a concurrency cap, and then submission order matters: a queued straggler delays the whole request, so start long branches first.
• Sectioning splits one task to cut latency; voting repeats one check to raise confidence. The threshold trades false alarms against misses, and correlated reviewers can count one finding twice.