08. Epochs: follow the sun

Draft 0.2, 2026-11-13. Normative. 01-conventions.md wins over this chapter. Source: DESIGN.md §6. Local names use the prefix EPO_ (constants) and Epo (types).

0. Scope, decisions and constraints#

This chapter specifies how a book moves home every EPOCH_HOURS: schedule, handoff per role, certificate, window and its floor, opening auction, fencing, keys and membership, failures, client migration and chaos tests. Simplex (07), continuous matching (05), sessions (04) and validator keys (09) are specified elsewhere.

How the chapter meets the hard constraints:

1. Words used in this chapter#

2. The schedule#

2.1 Boundaries#

EPOCH_BOUNDARIES_UTC is 00:00 TYO, 08:00 LDN, 16:00 NYC. The boundaries are UTC and fixed. Daylight saving time moves the local opening hour of LDN and NYC by one hour, and never the boundary. The reason: a boundary in UTC is a constant that every client, replica and light client computes with no time-zone table.

ASCII original

UTC     00:00         08:00         16:00         00:00
         │── e ≡ 0 ───│── e ≡ 1 ───│── e ≡ 2 ───│── e+3 ...      (e mod 3)
home     TYO           LDN           NYC           TYO
local    09:00 JST     08:00 GMT     11:00 EST     09:00 JST
                       (09:00 BST)   (12:00 EDT)

T is stored and compared as TAI nanoseconds (Nanos, 01 §3). The conversion from the UTC boundary uses the TAI − UTC offset in force at genesis plus any announced leap second. A leap second changes the TAI value of later boundaries; it is published in EpoConfig like any other change (§2.3).

2.2 Epoch numbers#

Epoch e is the slot that starts nominally at G + 8h·e. The numbering never depends on what happened: a hold makes epoch e longer and e+1 shorter, and a skip leaves an epoch number unused (§2.5). Reasons: 09 derives the sequencer operator (e mod 5), session keys and bonds from e alone, up to 4 epochs ahead; and a client computes the next epoch, home and T from the clock and G alone, so it needs an EpochNotice only for a revision.

Epoch is u32 and the wire carries its low 16 bits (epoch16): 2^16 slots are 59.8 years (computed). epoch16 is only ever tested for equality with the receiver's epoch, so it may wrap.

2.3 EpoConfig: the configuration of the next epoch#

Everything that differs between epochs is fixed in one record in the log of the epoch before.

struct EpoConfig {            // control record in the log of epoch e (07 control path; CHAIN: tx path, 09)
    epoch: Epoch,             // the epoch it configures, > e
    home: HomeCode,           // home(epoch) after any HomeOverride
    t_start: Nanos,           // nominal start (TAI); T of the epoch before it
    roster_hash: Hash,        // R_epoch: member indices, vote keys, seq keys, xchg keys (09 NodeCert)
    seq_order: [u8; 2],       // the home's vyrn-seq boxes: [primary F1, warm standby F2] (01 §14.4)
    lane_map_hash: Hash,      // lane_map(epoch) with each account's class, by replicas (10 §1.3)
    n_lanes: u8,              // lane count of home(epoch)'s boxes: n_ded + n_pool (REF 42 + 7)
    version: u32,             // engine semantics version in force for the epoch (§9.4)
    endpoint_hash: Hash,      // the endpoint table of home(epoch) (§11.1)
    flags: u16,               // bit0 skip (the epoch is empty), bit1 date roll, bit2 leap second
}

Rules:

• The EpoConfig for e+1 must be final at least EPO_CONFIG_LEAD_S (3,600 s) before T. If it is not, the replicas carry the previous epoch's configuration forward with home, t_start and seq_order recomputed from the schedule. Reason: a missing record must never block a handoff.
• 10 publishes the full lane_map(e+1) with the same lead (10 §1.4, ACC_MAP_PUBLISH_LEAD_S).
• The HandoffRecord covers EpoConfig through log_hash, so the certificate certifies the next epoch's configuration as well as the final state.

2.4 How T is announced: EpochNotice#

Every lane sends an EpochNotice record (01 §6.4) to each of its sessions at fixed offsets before T, and once more at the seal.

The notices before the seal are in-stream responses. They are a pure function of logged packets, so replicas reproduce them bit for bit (INV-DET). The trigger is a logged rx_ts and not a timer, because timers fire on order-stream time only (01 §11.3). A session that sends nothing gets the notice in the response to its next heartbeat. 04 requires a heartbeat at least once per second.

EpochNotice has three free bytes. This chapter uses byte 1 for the seq_index of epoch e+1 and bytes 2–3 for flags: bit0 revised, bit1 hold, bit2 skip, bit3 date roll, bit4 endpoint table changed (Requests to 01, item 1).

2.5 Holds, skips and overrides#

There are three ways the schedule changes. Each is a record in the log, so it is certified.

A skip keeps the old home instead of handing to the home after the skipped one. The reason: the old home's staff and replicas are on shift and healthy, and every extra handoff costs one window. A skipped epoch number is never used; EpoGo names the next epoch (§2.6, §5.1).

A hold that would push T past the nominal end of slot e+1 retargets the handoff to e+2 with home(e+2). The skipped slot e+1 is then unused, as with a planned skip. The reason: epoch e+1's keys and sequencer belong to a slot that has passed (09 §3.1).

A hold never shortens anything, and T has no upper bound: if no home is ever ready, the old home keeps the book. That is the only safe default.

2.6 Go/no-go#

At T − EPO_GO_LEAD_S (10 s) the old home decides whether to hand off. The decision is a logged record, so every replica and the new home see the same decision.

1. From T − EPO_PREP_LEAD_S (60 s), every member of R_{e+1} and both sequencer boxes of the new home send an EpoReady every EPO_READY_PERIOD_MS (1,000 ms) to every member of R_e, on both WAN paths (§5.5).
2. At T − EPO_GO_LEAD_S, the old sequencer's control core (BASE) or the Simplex leader (CHAIN) proposes exactly one of:
   • EpoGo{next_epoch, seq_index, next_seq_key, ready: [EpoReady; Q], seq_ready: [EpoReady; 2]}: Q fresh, valid EpoReady from distinct voting members of R_{e+1}, and one from each of the two sequencer boxes in seq_order.
   • EpoHold{epoch: e, t_new: T + EPO_HOLD_S} otherwise.
3. Validity, checked by every voter in R_e before it votes for the block:
   • EpoGo is valid if its Q + 2 EpoReady signatures verify, each is fresher than EPO_READY_FRESH_MS (3,000 ms) at the proposal's rx_ts, and seq_index = seq_order[0], the F1 box. The F2 box is the standby. A home with one ready sequencer box is held: without a standby, one fault right after activation would halt the book (§10.4).
   • EpoHold is valid only if the voter itself does not hold Q + 2 fresh EpoReady as above. A byzantine proposer therefore cannot hold a healthy handoff.
4. If neither record is final by T − EPO_GO_DEADLINE_S (5 s), the replicas treat the proposer as failed and run 07's failover. The next proposer proposes again with a deadline of T − 2 s. If that also fails, the result is a hold by default.

struct EpoReady {
    epoch: Epoch,             // e+1
    member: u8,               // roster index: a voting member of R_{e+1}, or a sequencer box
    follow_height: Height,    // last epoch-e block this node holds, finalized
    lag_ns: u64,              // replay lag: now − local rx of the newest frame not yet replayed
    keys_ready: bool,         // K_seq/K_out derived for all sessions (sequencer boxes only)
    version: u32,             // engine version it runs; must equal EpoConfig.version
    b4_pass: bool,            // replayed the last 3 epochs bit-identically (B4) on this version
    sig: Ed25519Sig,          // by NK(member, home(e+1)), ctx "vyrn      v1 ready"
}

A node sends EpoReady only if lag_ns ≤ EPO_READY_LAG_MS (1 ms), version matches, b4_pass holds, and (for a sequencer box) keys_ready holds. The lag excludes the WAN one-way, which is measured separately (B13). Replay runs at 0.6 × live, so a healthy follower lags by less than one block; 1 ms keeps the lag inside EPO_WINDOW_SLACK_MS (§6.2).

3. The handoff on one page#

ASCII original

 old home (TYO)                        WAN 110 ms                        new home (LDN)
 ─────────────────────────────────────────────────────────────────────────────────────────────
 T−60 s                                                      R_{e+1}: EpoReady every 1 s
 T−10 s  EpoGo logged (seq_index, next_seq_key) ◄─────────── Ready set
 T−1 s   EpochNotice(1 s) to every session
 T       lanes stop at rx_ts ≥ T; Seal step S ──► frames ──────────────┐   new seq: Collect starts
         last block closes (partial)          (tail relayed at once)   │   (rx_ts ≥ T, epoch16 = e+1);
                                                                       │   queued Ack ≤ 1 µs per packet
 T+2     block final (typ; max T+10)                                   │   window blocks final from T+2
 T+2.3   replicas at S: state_root, sign       HandoffSig ──────────┐  │
 T+2.4   any Q sigs = cert(e,S); INV-FENCE                          │  │
                                                                    │  ▼
 T+110.5                                       last tail frames arrive
 T+111.1                                       tail replayed: S′ known → Apply (0.7 ms)
 T+112.4                                       ◄────────────────────┘ cert arrives, verified
 T+112.6                                       Uncross step; first auction fill on the wire
 T+112.6+                                      lanes catch up on post-close packets, then live:
                                               continuous trading of epoch e+1
 T+112.6+  old home: followers of e+1; the old seq answers every stray packet within 1 µs

The numbers are typical budgets for TYO → LDN at normal window load (10% of peak). §6 gives every transition, the maximum and the peak-load case.

4. State machines#

4.1 Messages#

4.2 Old sequencer#

ASCII original

 Active(e) ─T−10 s─► Deciding ─EpoGo final─► Armed(e) ─lane rx_ts ≥ T (or clock ≥ T + slack)─► Sealing
     ▲                  │                                                                     │
     └─ EpoHold final: ─┘                                     all lanes sealed, final cut, rings drained
        T := t_new                                                                            ▼
 Redirect → Follower(e+1), answering strays ◄─cert(e,S) seen─ Sealed(e,S): Seal step, last block signed

The old standby follows the same machine as a follower and takes over the seal if the primary fails before T − EPO_SEAL_GUARD_MS (§10.1).

The seal, step by step:

1. Lane stop. Each lane applies a packet only if rx_ts < T. Its first packet with rx_ts ≥ T seals the lane: the lane publishes its last LaneSeq, sets its sealed flag (one cache line, lane-written) and answers that packet as in §10.7. A lane that receives nothing seals when the control core's PTP time (core 51 on REF) passes T + EPO_SEAL_SLACK_US (5 µs). The slack covers a packet with rx_ts < T that is still in the RX ring. The rule is per packet in NIC time, so all lanes cut at the same instant to within the PHC skew between NICs (B17).
2. Final cut. The cut cores keep publishing ordinary cuts every CUT_REFRESH_NS while lanes seal, so every op with rx_ts < T is acked or filled within LAT_MAX_NS as on any other packet (INV-ACK-AFTER-CUT, 01 §14.1). The seal adds no wait to any response. When every sealed flag is set, the cut cores publish the final cut C_f. It contains every lane's last op.
3. Drain. The BBO core executes every improve and cross in its rings. All of them are at or below C_f, because lanes push nothing after they seal.
4. Seal step. The BBO core executes StepRecord{kind: Seal, cut: C_f} (01 §14.10). Its id is S. The two order-stream cores (stream-A on core 4 for lanes 0–24, stream-B on core 7 for lanes 25–48, 01 §10) drain their halves of the lanes to C_f and write the frames, the Seal record included, to both replication queues.
5. Last block. The sequencer closes the current block at S, though it is shorter than BLOCK_NS, and signs it (seq_sig). This block is the last of epoch e.

Budget: T to the Seal frame on the wire ≤ EPO_SEAL_US (50 µs). The Seal step itself does nothing to the book. The book at S is the book after the last cross.

4.3 Old replicas (R_e)#

ASCII original

 Voting(e) ──Seal frame──► WaitFinal ──block with S final──► Replay to S ──root ready──► Signed
    │                                                                                │
    │ clock ≥ T + EPO_SEAL_GRACE_MS, no Seal frame                                    │ Q sigs seen
    ▼                                                                                ▼
 ForceSeal: 07 failover in seal mode ─► S := last step of the finalized prefix ─► Fenced(e,S)
                                                                                     │ EpoActivated
                                                                                     ▼
                                                                                Follower(e+1)

• Seal by the sequencer. A replica votes for the last block as for any block (07). When the block that contains the Seal step is final, the replica finishes its replay to S, computes state_root and log_hash, builds the HandoffRecord and signs it (§5.3).
• Force seal. If a replica's PTP clock passes T + EPO_SEAL_GRACE_MS (5 ms) with no Seal frame, it sends EpoForceSeal{e, last_height_held} and starts 07's failover in seal mode: the replicas finalize the longest order prefix they hold, as in an ordinary failover, but they promote no standby. S is the last step of that prefix. Q EpoForceSeal signatures are the attestation that 09 slashes as a withheld handoff.
• Failover guard. A failover of the sequencer that starts before T − EPO_SEAL_GUARD_MS promotes the old home's standby, which trades until T and seals. One that starts later runs in seal mode. EPO_SEAL_GUARD_MS = FAILOVER_TARGET_MS (20 ms): a promotion that cannot finish before T cannot trade, so seal mode is then the shorter outage. Either way the outage is at most the published 20 ms, as for any failover.
• Fence. From the first moment a replica holds Q valid HandoffSig on one record, it applies INV-FENCE: it rejects every epoch-e frame with a step greater than S and every epoch-e block above last_height (§8).

4.4 New-home replicas (R_{e+1})#

ASCII original

 Follower(e) ──T−60 s──► Preparing ──T──► Collecting ──cert(e,S) verified, root equal──► Voting(e+1)
     ▲                     sends EpoReady   votes on window     │                          │
     │                                      blocks only         │ root differs             │ opening block
     │                                      (no steps)          ▼                          ▼ final
     │                                                      Faulty: stop voting,       EpoActivated
     └────────────────────────── re-sync from a snapshot ── alarm, re-replay           sent

• Follower(e). The node replays epoch e's finalized log, plus the speculative tail stream, with the same engine as R_e (01 §11.6). Its state lags the old home by one WAN one-way.
• Preparing. It loads EpoConfig(e+1), checks its version, and sends EpoReady.
• Collecting. From T it receives the new sequencer's window PacketRecords and, from its data tier, the window payloads. It votes on window blocks (§7.1) and on nothing else. A window block holds only PacketRecords: it fixes the order of window packets, which does not depend on epoch e. No block with a step may be final before the history it extends is certified (F4). So every queued-acked order is final within FINALITY_MAX_MS, in the window too (01 §14.0).
• Voting(e+1). After it verifies the certificate (§5.4), it persists armed(e+1) in its high-water mark, sends EpoArmed to both sequencer boxes, and votes on all blocks of e+1. The first block with a step is the opening block (§7.1). When it is final, the members send EpoActivated{epoch: e+1, height, finality cert} to the old and third homes.

4.5 New sequencer#

ASCII original

 Follower(e) ─T−60 s─► Preparing ─T (own PTP)─► Collect ─replay reaches Seal step S′─► Apply(S′)
                       derive keys               lanes queue-ack,        lanes apply window ops
                       warm retention            stream PacketRecords,   against the state at S′
                                                 propose window blocks          │
                                   cert(e,S): S == S′, roots equal ◄────────────┘
                                                 │                          │ S ≠ S′ or root differs
                                                 ▼                          ▼
                          Close window → Uncross → catch-up → Live(e+1)     Undo to the last final
                                                                            block, replay to S, Apply(S)

• Preparing (from T − EPO_PREP_LEAD_S): derive K_seq(e+1) and K_out(e+1) for every open session (§9.1); fill the response retention buffers from the replay (§11.3); install the NIC flow filters for lane_map(e+1) (06); allocate the collect buffers.
• Collect (from T by its own PTP clock): each lane accepts a packet only if its rx_ts ≥ T and its epoch16 equals e+1. It runs deadline admission (01 §14.1), the session, bound-address and tag_seq checks, assigns LaneSeqs, pre-encodes the PacketRecord into its order ring (01 §14.5), appends the packet to its collect buffer and answers it with a queued Ack within LAT_MAX_NS (§7.11). It applies no message (§7.1). The control core proposes one window block per BLOCK_NS. A packet with rx_ts < T gets an out-of-stream Reject{EPOCH_MISMATCH} (§11.2), also within LAT_MAX_NS.
• Apply(S′): when the speculative replay executes the Seal step, the box holds the state at S′. Lanes apply their collect buffers against it (§7.3). No response that depends on that state is sent. Lanes keep collecting and queue-acking new packets while they apply.
• Activation: the certificate arrives and verifies, S == S′, and the box's own state_root and log_hash at S equal the record's. The lanes close the window and the BBO core uncrosses (§7.6). The first auction fill waits for EpoArmed from Q members of R_{e+1}; that metro hop runs in parallel with the uncross and fits its budget. Lanes then catch up and go live (§7.10).
• Mismatch: S′ ≠ S happens only when the old sequencer failed or lied in the tail (§10.1, §10.5). The speculative replay and Apply keep an undo log from the last finalized block of e on. The box undoes to that block, replays the certified log to S and re-applies. Budget EPO_RESPEC_MS (15 ms): ≤ 10 ms of unfinalized tail at 0.6 × live, plus the backlog's Apply. If the root still differs, the box is faulty: it stops, and the standby activates (§10.4).

Standby (F2). The new home's standby vyrn-seq runs the same machine without client traffic. It follows the tail, derives K_seq(e+1, b) and K_out(e+1, b) for its box b and every open session, receives every window block and payload, and applies the final window blocks against S′. It receives HandoffSig and EpoArmed like the primary. So at any moment it can take over within FAILOVER_TARGET_MS and activate from the same certificate and window blocks (§10.4). It never votes, so a promotion keeps CLIENT_SPLIT_BASE and CLIENT_SPLIT_CHAIN intact (01 §14.4).

4.6 Data tier#

The new home's data tier must accept e+1 copies from T. Reason: replicas vote on a window block only when they hold every payload it references (07), and window blocks are final within FINALITY_MAX_MS of each packet (§7.1). The opening block then waits for no transfer either.

4.7 Third home#

The third home follows e from the old home until it holds cert(e, S), verified as in §5.4, then follows e+1 from the new home. It relays every HandoffSig and the last block certificates of e to the new home as a second route (§5.5).

4.8 Clients#

Active(e, H) → Scheduled(T) → Switching (resend unacked to H′) → Active(e+1, H′); see §11. A client holds the addresses and keys of both sequencer boxes of H′, so a standby promotion needs no new state (§10.4).

4.9 Timeouts and periods#

5. The HandoffRecord and its certificate#

5.1 Contents#

The record is 01 §6.6's HandoffRecord. Its canonical encoding is fixed here (120 bytes, little-endian):

off size field          value
  0   2  book          BookId
  2   1  next_home     HomeCode of epoch e+1 (may equal home(e): §2.5)
  3   1  reserved      0
  4   4  epoch         e
  8   8  last_step     S: the Seal step, or the last step of the seal-mode prefix
 16   8  last_height   height of the last block of e (the block that contains S)
 24  32  state_root    commitment to the full engine state after S (below)
 56  32  log_hash      BLAKE3 of the BlockHeader at last_height (the chain head)
 88  32  next_seq_key  Ed25519 seq key SK of the new sequencer, from the final EpoGo
120      end

• state_root = BLAKE3 derive_key("vyrn v1 state root", lthash), where lthash is the 2,048-byte lattice hash over every state element at S: accounts and balances per lane, sessions (keys excluded, pkt_seq and resp_seq high-water marks included), resting orders with their priority keys and filled quantities, rate buckets, the last trade tick, and control state. Replicas keep it incrementally, O(1) per change (DESIGN §7). This chapter needs it in BASE mode too (Requests to 01, item 3).
• log_hash covers every earlier block through prev, so it covers EpoConfig(e+1), EpoGo and every packet and step of e.
• The next epoch is EpoGo.next_epoch, which log_hash covers. 01 should make it a field (Requests to 01, item 2), because a light client then needs only the certificate.

struct HandoffCert {
    record: HandoffRecord,
    sigs: [(u8, Ed25519Sig); Q],   // (voting-member index in R_e, signature); distinct, ascending
}
// BASE: 120 + 3 × 65 = 315 bytes.  CHAIN: 120 + 4 × 65 = 380 bytes.

5.2 Signature#

msg = derive_key("vyrn      v1 handoff", genesis_hash ‖ mode ‖ encode(HandoffRecord))
sig = Ed25519_sign(VK(i, home(e)), msg)

genesis_hash and mode stop a record from one deployment or mode from verifying in another. The key is the member's vote key (09 §2.2), because a handoff signature is a consensus statement and 09 slashes it like a vote.

5.3 Who signs, and when#

A member i of R_e signs exactly one HandoffRecord per (book, epoch), and only when all of these hold:

1. It holds a finality certificate (07) for the block at last_height, and that block contains S.
2. Its replay has executed S, and its own state_root and log_hash equal the record's.
3. The finalized log contains EpoGo for e+1 and next_home, next_seq_key equal its fields.
4. It has not signed a HandoffRecord for (book, e) before. It persists (book, e, record hash) and fsyncs before it sends the signature (09 §2.5's high-water rule).

It then sends HandoffSig{record hash, i, sig} to every member of R_e and of R_{e+1}, to both sequencer boxes of the new home, and to the third home, on both WAN paths. Any node that holds Q signatures on one record holds the certificate. Nobody needs to aggregate first; the new home aggregates on receipt.

Only voting replicas sign (01 §14.4). In BASE mode Q = Q_BASE = 3 of the N_BASE = 4 voting replicas R1–R4. In CHAIN mode Q = Q_CHAIN = 4 of the 5 validator boxes. Neither sequencer box of the old home signs, in either mode, and a promotion of the standby changes no signer. A book group (01 §12.3) seals all its books at the same T and produces one record per book.

Uniqueness. Two certificates for one (book, e) with different contents need two quorums. Any two quorums intersect in 2Q − n members: 2 in BASE (3 + 3 − 4) and 3 in CHAIN (4 + 4 − 5). At least one of them is honest, because F = 1, and an honest member signs once (rule 4). So there is at most one certificate per (book, e).

5.4 Verification at the new home#

A node of R_{e+1} (and the third home) accepts cert(e, S) when every step passes:

Checks 1–2 cost Q Ed25519 verifications: ≤ 0.2 ms (budget, at ≤ 50 µs each). Checks 3–5 compare hashes the node already holds. A failure of check 5 at a node does not change the certificate: Q members of R_e signed it, so the node, not the certificate, is wrong.

The roster keys of R_e come from EpoConfig(e), which cert(e − 1) covered, back to the genesis record. A node that starts from a snapshot verifies that chain of certificates (07).

5.5 The log tail over the WAN#

At T the new home lacks only the tail, the last WAN one-way of log and data. The tail moves by streaming, so the end of an epoch needs no bulk transfer.

• Each WAN path carries at least EPO_WAN_GBPS_MIN = 250 Gbps from the active home to each follower home (budget). Each path carries the whole stream at all times, so one lost packet or one failed path never leaves a gap that a pull must fill, and no backlog grows the window.
• Followers replay the unfinalized frames speculatively. They treat state as final only up to the last finalized block. The speculative stream lets the new sequencer reach S′ one finality time before the certificate arrives (§6.1).
• Pull. After the certificate arrives, a node that misses a block or a payload sends EpoTailReq{heights | pkt_hashes} to the old home and the third home at the same time. The first EpoTailResp wins. For TYO → LDN the third home (NYC) answers first: a request at T + 110 ms gets its answer at ≈ T + 180 ms from NYC (35 ms each way, NYC got the data at T + 80) and at ≈ T + 330 ms from TYO.
• Follow lag at T adds to the window one for one; EpoReady caps it at EPO_READY_LAG_MS (1 ms, §2.6), which EPO_WINDOW_SLACK_MS covers. A pull is needed only when both paths lost the same data, which is two faults (§10.6).

6. Timeline per transition#

6.1 Components#

The certificate and the tail travel in parallel:

ASCII original

cert path:  T ─seal─► S ─finality─► final ─root+sign─► cert ─WAN─► new home ─verify─► ready
tail path:  T ─seal─► S ─relay─► WAN ─► new home ─replay tail─► S′ ─apply backlog─► applied
activation = max(cert path, tail path);   W = activation + uncross + catch-up − (last old fill)

The backlog per lane at S′ is the window's message rate per lane times t_wan. At 1B msgs/s into the new home (peak) it is 20M/s × 110 ms = 2.2M messages per lane for TYO → LDN, which applies in 6.6 ms at 3 ns. At normal window load (10% of peak, budget assumption) it is 0.66 ms. The catch-up backlog is the per-lane rate times t_unx, and it drains at LANE_NS_PER_MSG against the lane's spare capacity: at peak (load 0.6) and t_unx = 0.5 ms, 10k messages × 30 ns / 0.4 = 0.75 ms.

6.2 Windows#

All values in ms, budgets. "Peak" = the window receives 1B msgs/s.

The bound. W ≤ t_wan + FINALITY_MAX_MS + EPO_WINDOW_SLACK_MS (3 ms) for every transition, at every load up to 1B msgs/s, with no fault. The cert path is at most t_wan + FINALITY_MAX_MS + 1.3 ms; the tail path at peak is at most t_wan + 7.7 ms, never above the cert path's maximum; uncross and catch-up add at most 1.25 ms. Open question 1 guards the 3 ns Apply this relies on. This tightens B13's BLD_HANDOFF_SLACK_MS (5 ms) to 3 ms. DESIGN §6 gives ~120, ~45 and ~90; DESIGN rounds away the signing, verification and uncross. Faults add only the outages of §10, each bounded by FAILOVER_TARGET_MS unless two faults coincide.

6.3 The floor: no window shorter than one WAN one-way latency#

Claim. Let d be the smallest one-way delay of any channel from the old home to the new home. Any handoff protocol that keeps a single price-time book (no order fills beyond its quantity, and no fill trades through a better price that the book holds) has a window W ≥ d.

Proof. Let t₀ be the last moment at which the old home may still fill (its seal), and suppose the new home fills an order at t₁ < t₀ + d. Nothing sent from the old home at or after t₀ can have reached the new home by t₁ (causality; d is the fastest channel). So the new home's state at t₁ is the same in any two executions that differ only in what the old home received at or after t₀. Two such pairs give a contradiction:

1. Double fill. A sell order X of 1 lot at 100 rests in the book. Execution A: at t₀ a buy of 1 lot at 100 arrives at the old home and fills against X. Execution B: no buy arrives at the old home. At t₁ a buy of 1 lot at 100 arrives at the new home. In B the book must fill it against X, or it fails to match a marketable order against a resting one. The new home cannot tell A from B, so it fills it in A too. In A, whose last old fill is at t₀, X sells 2 lots out of 1.
2. Trade-through. The best ask is 100. Execution A: at t₀ the old home receives a sell at 99, which rests. Execution B: it does not. At t₁ a buy at 100 arrives at the new home. In B it fills at 100. In A the book holds an ask at 99, so a fill at 100 trades through it. The new home cannot tell A from B.

So the first fill at the new home is at t₁ ≥ t₀ + d, and W ≥ d. ∎

What the proof does not allow. Stopping the old home earlier only moves t₀. Letting both homes trade at the same time needs a cross-home agreement per fill, which costs ≥ d per fill and breaks the 1 µs fill. Moving the book in pieces lets an aggressor at one home miss a better price that is still at the other home, which is case 2.

Why finality adds to d. With F = 1, the old sequencer may be the faulty node. The new home must start from a last step that a quorum fixed, or a faulty old sequencer could later show the new home a different end of epoch e. A quorum decision takes at least one round of votes inside some metro after the last step. Our design runs that round at the old home before the WAN crossing. Running it at the new home instead (new-home nodes voting on epoch-e frames) costs the same round after the crossing. So W ≥ d + one metro round, and our W ≤ d + t_fin + EPO_WINDOW_SLACK_MS. The window costs fills only. Orders still get a 1 µs answer and a final place in the log in it (§7.11), because neither needs the state at S: the proof above is about fills.

The light numbers. Fiber carries light at ~4.9 µs/km (refractive index ~1.47, vendor). Great circles: TYO–LDN ~9,560 km, LDN–NYC ~5,570 km, NYC–TYO ~10,850 km (computed). The straight-fiber floors are 47, 27 and 53 ms. Today's cable routes are longer, which gives the planning figures 110, 35 and 80 ms. A straighter route or a radio path lowers d. Nothing lowers it to zero, and no protocol goes below it.

7. The opening auction#

7.1 Phases and the opening block#

ASCII original

 T ───────── Collect ──────────┬──── Apply ─────┬─ Close ─┬─ Uncross ─┬─ catch-up ─┬─ Live(e+1)
   lanes queue window packets   S′ known:        cert ok:   step U     post-close   snapshot v_U;
   queued Ack ≤ 1 µs each       lanes apply ops  lanes      = (e+1, 1)  packets as   every packet
   window blocks final ≤ 10 ms  against S′       close                  continuous   fills on arrival

• Collect needs no state. Each lane checks epoch16, rx_ts ≥ T, deadline admission and tag_seq, assigns LaneSeqs, copies the packet into its collect buffer, emits its PacketRecord with fills_applied = 0, and answers with a queued Ack (§7.11). Cost EPO_COLLECT_NS (10 ns) per message (budget). While the box still replays the epoch-e tail, a lane at peak carries ~60% replay plus ~20% collect. Replay runs in slices of at most EPO_REPLAY_SLICE_NS (40 ns) between RX polls, so replay never delays an answer by more.
• Pooled and dedicated lanes behave the same in the window (01 §14.15). From T until the lane is live (§7.10), a pool core serves only its own RX queue: it steals nothing, and every pooled account's token stays parked on its pool-queue core (10 §1.2), where from_state puts it. So Collect, Apply and catch-up are lane-local on every lane, with the same costs and the same LaneSeq order, and no AcctTake is logged in the window. Collect at 10 ns per message leaves no queue for stealing to drain. Stealing resumes at the first packet after the lane goes live.
• Window blocks. The control core proposes one window block per BLOCK_NS: WinBlock{book, epoch: e+1, wheight: u32, prev: Hash, anchor: Hash, order_root, data_root, seq_sig}, with anchor = BLAKE3 of the final EpoGo record and wheight from 1. It holds only PacketRecords (and control-session records such as EpoHalt, §10.3). R_{e+1} finalizes window blocks with 07's Simplex in its own instance (book, e+1, window), at metro finality (FINALITY_MAX_MS). A window block fixes order, not outcome: the outcome is a pure function of the final window blocks and cert(e, S), so finalizing it before the certificate is safe (F4).
• Apply starts at S′. The box first builds the epoch-(e+1) engine from the state at S′ (EpochEngine::from_state, §7.9): it places each account in its lane_map(e+1) class and lane (a pooled account on its pool queue, with its token parked there), folds each carried order's filled quantity into its remaining quantity, and starts the filled-quantity array at zero. Then each lane applies its buffer in LaneSeq order (§7.3), in the same slices.
• Close. At activation the control core publishes t_close = its PTP time + EPO_SEAL_SLACK_US. A lane includes in the auction exactly the packets with rx_ts < t_close, exactly as the old home sealed at T (§4.2). Later packets are still queue-acked on arrival and wait in the collect buffer for continuous mode (§7.10).
• Uncross is step U = (e+1, 1), StepKind::Uncross (01 §14.10). Its cut contains every op with rx_ts < t_close. No step runs in epoch e+1 before U.
• The opening block has height last_height + 1 and prev = log_hash. Its order_root covers the head of the window chain and the Uncross step, so it spans the whole window. Its window packets are final already, so its vote waits for nothing after U.

7.2 What each message does in the window#

A window New gets a window reference wref in its queued Ack: the lane's collect index of the message, which Apply maps to the handle it allocates. A Cancel, Reduce or Replace in the window names a window order by setting byte 1 bit 0 and putting its wref in the handle field (Requests to 01, item 10). After U every window order has a handle, which its final record carries.

If EpoConfig(e+1).flags.date_roll is set (the boundary into a TYO epoch, 00:00 UTC), Apply first expires every carried Day order: Out cause 4, expired (01 §14.10). Day means the UTC trading date. In v1 every resting order is Day, so the TYO open starts with the window orders only.

7.3 Apply#

fn lane_apply(l: &mut Lane, buf: &CollectBuf, cfg: &AuctionCfg) {
    for pkt in buf.iter_in_lane_seq_order() {
        if pkt.flags.shed { continue; }                  // answered BUSY in Collect (01 §14.1)
        let s = l.session_mut(pkt.session);
        if pkt.pkt_seq <= s.hw_pkt_seq { l.cnt.dup_across_epoch += 1; continue; }   // applied in e
        if pkt.pkt_seq != s.hw_pkt_seq + 1 { l.outcome(pkt, BAD_SEQ); continue; }    // 04 P5
        s.hw_pkt_seq = pkt.pkt_seq;
        if let Err(r) = l.rate_check(s, pkt.rx_ts) { l.outcome(pkt, r); continue; } // 04 §9
        for (i, m) in pkt.msgs().enumerate() {
            let out = match m.kind {
                New => l.auction_new(m, cfg),                // risk (10 §3.1), band, TIF table §7.2
                Cancel | Reduce => l.apply_cancel(m),        // carried, or window by wref; hist −
                Replace => l.apply_replace(m, cfg),
                CancelAll => l.apply_cancel_all(s),
                Heartbeat => Outcome::None,
                ResendReq => l.queue_resend(s, m),
            };
            l.record_outcome(pkt, i, out);                   // final records are built after U
        }
    }
}

auction_new checks the TIF table (§7.2), the band (§7.5) and risk (10 §3.1–3.2), allocates a handle in LaneSeq order, appends the order to its lane FIFO with its PrioKey (§7.4) and adds its quantity to the lane's histogram. Cost EPO_APPLY_NS = 3 ns per message, amortized (budget). It is 10× below LANE_NS_PER_MSG because RX, MAC, parse, classification and the response were done in Collect or come after U. Carried-order cancels touch the order arena, which the lane prefetches 16 messages ahead. Packets that arrive during Apply are queue-acked, join the end of the buffer and are applied in turn.

Speculation is invisible. Apply against S′ changes only the lanes' internal state. The queued Ack depends only on the packet and the earlier window packets, never on S′. No response that depends on S′ leaves the box before activation, and activation requires S = S′ and equal roots (§4.5). Replicas of e+1 later replay the same ops against the certified state at S, in the same LaneSeq order, and reach the same state (INV-DET).

7.4 Priority key#

PrioKey = rx_ts (64 bits) ‖ entry_rank (8 bits) ‖ entry_lane_seq (56 bits)      as one u128

entry_rank is the lane's rank in the tie-break order of the epoch in which the order entered (DESIGN §10: lane-id tie-breaks rotate each epoch; 05 owns the rotation). An order keeps its key for life, also when its account moves lanes or changes class (10 §1.3). Every carried order has rx_ts < T by the old home's seal rule, and every window order has rx_ts ≥ T by the new home's collect rule. So every carried order has priority over every window order at the same price, whatever the clock skew between the homes. entry_lane_seq < 2^56 holds for 69 years of one lane at full rate.

7.5 Reference price and band#

ref = last_trade_tick(e)                    // the last fill price of epoch e
      or bid(S) if no trade ever, or ask(S), or BookSpec.ref_tick (10) if the book is empty
if bid(S) exists: ref = max(ref, bid(S))
if ask(S) exists: ref = min(ref, ask(S))   // bid(S) < ask(S) by INV-UNCROSSED, so this is defined
band = [max(min_tick, ref − EPO_HIST_HALF), min(max_tick, ref + EPO_HIST_HALF)]   // 8,193 ticks

The clamp into the carried spread makes every carried order that can execute lie inside the band: a carried buy is at most bid(S) ≤ ref ≤ band top, and a carried sell is at least ask(S) ≥ ref ≥ band bottom. Carried buys below the band and carried sells above it cannot execute, because the clearing tick is inside the band. They stay out of the histogram and rest. A window New outside the band is rejected BAD_TICK. So no histogram bucket ever aggregates prices outside the band.

7.6 The uncross rule#

Let b[t] and s[t] be the total remaining buy and sell quantity at tick t over all lanes, carried and window orders together, for t in the band.

D(t) = Σ_{u ≥ t} b[u]       demand at t: buys willing to pay t or more
S(t) = Σ_{u ≤ t} s[u]       supply at t: sells willing to take t or less
V(t) = min(D(t), S(t))      executable volume
I(t) = D(t) − S(t)          imbalance (signed)
p*   = the tick that maximizes, in order:
       1. V(t)                     maximum executable volume
       2. −|I(t)|                  minimum imbalance
       3. −|t − ref|               nearest the reference
       4. −t                       lowest tick

If max V = 0, nothing crosses. The book opens with no uncross fills and U only changes the phase.

Lemma 1 (the candidates are intervals). D is non-increasing and S non-decreasing in t. For any v, {t : V(t) ≥ v} = {t : D(t) ≥ v} ∩ {t : S(t) ≥ v} is an intersection of a down-set and an up-set, an interval. So C1 = argmax V is an interval. I is non-increasing, so {t : |I(t)| ≤ m} is an interval for every m, and C2 = C1 ∩ {|I| ≤ min_{C1} |I|} is an interval [a, b]. ∎

Lemma 2 (rule 4 never decides). C3 = argmin_{t ∈ [a,b]} |t − ref| is the single tick clamp(ref, a, b), because ref is a tick. ∎ Rule 4 stays in the code so that the Rust engine, the Vx client and the oracle cannot disagree on a malformed input.

Allocation. Let V = V(p*). Uniform price: every execution is at p*.

• If I(p*) = 0: every buy at ≥ p* and every sell at ≤ p* fills in full.
• If I(p*) > 0 (buys are rationed): every sell at ≤ p* fills in full. Buys fill in price-time priority: higher limit first, then lower PrioKey, until V is used. The marginal tick p_m is the highest t with D(t) ≥ V; buys above p_m fill in full, buys below it get nothing, and at p_m the remaining q_rem = V − D(p_m + 1) goes to the orders in key order. The marginal order is the first order at p_m at which the cumulative quantity reaches q_rem; it fills partly or exactly.
• If I(p*) < 0: the mirror image, with sells rationed, p_m the lowest t with S(t) ≥ V.

Lemma 3 (the residual book is uncrossed). Suppose after the uncross a buy at β and a sell at α ≤ β both have quantity left. Take I(p*) > 0. Every sell at ≤ p* filled in full, so α > p*, and β ≥ α > p*, so the buy is rationed. Price priority means every fill went to buys at ≥ β ≥ α, so D(α) ≥ V + (rest of the buy) > V. The sell at α was not eligible, so `S(α) ≥ S(p*) + (its qty)

V. Then V(α) > V, which contradicts rule 1. I(p*) < 0is the mirror, and forI(p*) = 0every residual buy is below p* and every residual sell above it. ∎ So the snapshot after U satisfiesINV-UNCROSSED`.

7.7 Pseudocode#

struct UncrossRecord {          // written by the BBO core only; read by every lane
    p: Tick, v: u64, imb: i128, // clearing tick, volume, I(p*)
    p_m: Tick,                  // marginal tick on the rationed side (= p if imb == 0)
    marg: PrioKey, marg_qty: Qty, // marginal order and its fill; marg_qty = 0 if imb == 0
    ref_tick: Tick, band_lo: Tick,
}

fn uncross(lanes: &[LaneView; N_LANES], cfg: &AuctionCfg) -> Option<UncrossRecord> {
    // 1. Sum the lanes' histograms over the union of their touched ranges.  O(N_LANES × B)
    let (lo, hi) = lanes.touched_union();
    let (mut b, mut s) = ([0u64; EPO_HIST_TICKS], [0u64; EPO_HIST_TICKS]);
    for l in lanes { for t in lo..=hi { b[t] += l.hist.buy[t]; s[t] += l.hist.sell[t]; } }
    // 2. One pass over the band, rules 1–4 as one lexicographic key.  O(B)
    let d = suffix_sums(&b); let sp = prefix_sums(&s);
    let mut best: Option<(u64, Reverse<u128>, Reverse<u32>, Reverse<Tick>, usize)> = None;
    for t in lo..=hi {
        let v = min(d[t], sp[t]);
        let imb = d[t] as i128 - sp[t] as i128;
        let k = (v, Reverse(imb.unsigned_abs()), Reverse(cfg.tick(t).abs_diff(cfg.ref_tick)),
                 Reverse(cfg.tick(t)), t);
        if best.map_or(true, |bk| k > bk) { best = Some(k); }
    }
    let (v, _, _, _, t) = best?;
    if v == 0 { return None; }
    let imb = d[t] as i128 - sp[t] as i128;
    // 3. Marginal tick and remaining quantity on the rationed side.  O(B)
    let (side, p_m, q_rem) = match imb.signum() {
        0  => return Some(UncrossRecord::balanced(cfg.tick(t), v, cfg)),
        1  => { let m = (t..=hi).rev().find(|&u| d[u] >= v).unwrap();
                (Buy, m, v - if m < hi { d[m + 1] } else { 0 }) }
        _  => { let m = (lo..=t).find(|&u| sp[u] >= v).unwrap();
                (Sell, m, v - if m > lo { sp[m - 1] } else { 0 }) }
    };
    // 4. Marginal order by weighted selection over the lanes' FIFOs at (side, p_m).
    let (marg, marg_qty) = select_marginal(lanes, side, p_m, q_rem);
    Some(UncrossRecord { p: cfg.tick(t), v, imb, p_m: cfg.tick(p_m), marg, marg_qty,
                         ref_tick: cfg.ref_tick, band_lo: cfg.band.lo })
}

/// Finds the order with the smallest key such that the quantity at or below it reaches q_rem.
/// Each lane's FIFO at (side, p_m) is sorted by PrioKey; the lane builds its prefix quantities.
fn select_marginal(lanes: &[LaneView], side: Side, p_m: usize, q_rem: u64) -> (PrioKey, Qty) {
    let (mut lo, mut hi) = ([0u32; N_LANES], lanes.fifo_lens(side, p_m));
    let mut below = 0u64;                                   // qty known to be below every live key
    loop {
        // Pivot: weighted median of each live range's middle key, weight = range length.
        let piv = weighted_median(lanes.iter().map(|l| (l.key_at((lo[l] + hi[l]) / 2), hi[l] - lo[l])));
        // One parallel round: lane l returns r[l] = #keys < piv in its live range, q[l] = their qty.
        let (r, q) = lanes.rank_round(piv, &lo, &hi);
        let q_less = below + q.iter().sum::<u64>();
        let q_piv = piv.qty;
        if q_less < q_rem && q_rem <= q_less + q_piv { return (piv.key, (q_rem - q_less) as Qty); }
        if q_less >= q_rem { for l in 0..N_LANES { hi[l] = lo[l] + r[l]; } }
        else { below = q_less + q_piv;
               for l in 0..N_LANES { lo[l] += r[l] + (l == piv.lane) as u32; } }
    }
}

/// Lane side: the fill of one own order. Pure function of the record; no BBO-core write.
fn auction_fill(o: &Order, u: &UncrossRecord) -> Qty {
    let better_or_at = |t| if o.side == Buy { t >= u.p } else { t <= u.p };
    if !better_or_at(o.tick) { return 0; }
    let rationed = (u.imb > 0 && o.side == Buy) || (u.imb < 0 && o.side == Sell);
    if !rationed { return o.remaining; }
    let beyond = |t| if o.side == Buy { t > u.p_m } else { t < u.p_m };
    if beyond(o.tick) { o.remaining }
    else if o.tick != u.p_m { 0 }
    else if o.key < u.marg { o.remaining } else if o.key == u.marg { u.marg_qty } else { 0 }
}

Each round of select_marginal discards at least a quarter of the live keys (the weighted median of medians). So it takes at most log_{4/3} N + 1 rounds: 49 rounds for N = 10^6 orders at p_m. A round costs ~2 µs (budget): the BBO core writes the pivot line, each lane binary-searches its own L2-resident array and writes one reply line, and the BBO core reads 49 lines (42 dedicated lanes and 7 pool cores, 01 §10) and takes a weighted median of 49 pairs.

Writers. The BBO core writes only UncrossRecord. Each lane computes the fills of its own orders with auction_fill, rewrites their remaining quantities and settles its own balances (01 §12.2). The filled-quantity array is not written by the uncross. Every field still has one writer (INV-WRITER), and no lane decides a fill by itself: auction_fill only reads the BBO core's decision (INV-LANE-NOMATCH).

Fill records. Each executed order gets one Fill at tick p with liq = 2 (auction, 01 §14.10). A window order's fill answers its packet (index = its message index); a carried order's fill is unsolicited (index = 0xFF). The execution ID is (U, ordinal) with ordinal = lane_id × 2^32 + k, where k counts the lane's executed orders in price-then-key order. Each lane computes its ordinals alone.

STP does not apply to the uncross. In a uniform-price auction no buy is paired with a particular sell, so there is no self-match to prevent. The drop copy flags an account that is on both sides of one uncross (10).

7.8 Scale and time budget#

At peak, 1B msgs/s for a 110 ms window is ~110M messages, 2.2M per lane. Sizing:

Uncross budget, from activation to the first auction fill on the wire (EPO_UNCROSS_FIRST_US):

To the last uncross fill at 10^6 executed orders: each lane emits 20k fills at ~15 ns (0.3 ms), and 62.5k response packets of 16 records (17.5 MB) leave 4 × 100GbE in 0.35 ms. Total ≤ 1,000 µs, which is B14's BLD_UNCROSS_US_MAX. This chapter fixes B14's pass at that value.

7.9 Interface with 05's auction mode#

05 owns the engine. 08 drives it through this interface, which 05 implements in vyrn-engine:

trait EpochEngine {
    /// Build the e+1 engine from the state at S: lane placement by lane_map(e+1), carried orders
    /// with remaining = qty − filled, empty filled-quantity array. Runs before seal().
    fn from_state(state: &StateAtS, map: &LaneMap, n_lanes: u8) -> Self;
    /// Lanes take window ops (§7.3); no classification, no BBO ring traffic.
    fn enter_auction(&mut self, cfg: AuctionCfg);            // band, ref, date_roll
    fn apply_window(&mut self, lane: LaneId, buf: &CollectBuf);
    /// Lanes close at rx_ts ≥ t_close; returns the final cut.
    fn close_window(&mut self, t_close: Nanos) -> Cut;
    /// The Uncross step U: BBO core runs uncross(), publishes UncrossRecord, logs StepRecord.
    fn uncross(&mut self, cut: Cut) -> Option<UncrossRecord>;
    /// Lanes apply auction_fill to their orders, settle, catch up on post-close packets, go live.
    fn resume_continuous(&mut self) -> Snapshot;              // snapshot v_U, INV-UNCROSSED
}

The same calls serve any auction inside an epoch (a reopening after BOOK_HALTED, 05 and 10), with the last trade before the halt as the reference.

7.10 From auction to continuous trading#

1. After U, the BBO core publishes snapshot v_U with the residual best bid and ask (Lemma 3).
2. Each lane applies auction_fill to its orders, moves residual window orders into its lane FIFOs at their ticks (they are already there in auction mode), and sets its phase to continuous.
3. Catch-up. The lane then processes its post-close packets (rx_ts ≥ t_close, already queue-acked) as continuous ops in LaneSeq order, classified against v_U (05), ordered after U. A packet that arrives while this queue is not empty is queue-acked within LAT_MAX_NS and joins its end: a fill sooner would overtake an earlier packet. When the queue is empty the lane is live and fills each packet on arrival. Catch-up takes at most EPO_CATCHUP_US (0.75 ms at peak, §6.1). W ends when the last lane is live.
4. Final records. The final records of window messages drain at low priority. Each lane builds, per session in (pkt_seq, index) order: Fill, then Ack for a resting rest, Out for an IOC or auction-only rest, Reject and CancelAck as recorded. Uncross fills go first, and leave within BLD_UNCROSS_US_MAX (§7.8). Continuous responses always go before the drain. A lane posts a drain packet only when its port's TX load is below EPO_DRAIN_LOAD_CAP (0.80). At peak (~110M records, ~1.9 GB) the drain takes ≤ EPO_DRAIN_MS (150 ms) in the ~50% of port capacity that continuous traffic leaves. Final records take their resp_seq at U, before any continuous response of the session; the client accepts continuous responses ahead of them and sends no ResendReq for a gap younger than EPO_DRAIN_MS.
5. LAT_MAX_NS (REQ-LAT-1) applies to every message at every moment, the window included: each window message got its queued Ack within it (§7.11). What waits for U is the fill and the final record, never the first answer.
6. vyrn-md publishes one AuctionResult{p, v, imb, ref} right after U, then the book (10). It publishes no indicative price during the window, because the new home has no state at S until the last few milliseconds of the window.

7.11 The 1 µs answer in the window#

The window costs fills only (§6.3). Every packet that reaches a new-home lane gets its first response within LAT_MAX_NS, like any other packet (01 §14.0, E10).

The queued Ack (Requests to 01, item 8) is Ack (0x81) with bytes 2–3 = EPO_ACK_QUEUED, handle = the wref (New, Replace) or the target handle (Cancel, Reduce), tick = the message's tick or 0, and the client_tag. It states: this message is in the log of e+1 at (lane, LaneSeq), and it is resolved at U. It does not state that a New passed risk: balances are state at S, which the box does not hold yet. Lanes send it under the same gates as any ack: after a published cut includes the op (INV-ACK-AFTER-CUT) and after its PacketRecord left on both replication paths (INV-FRAME-FIRST, 01 §14.5). So a queued-acked message is held by R_{e+1}, survives a sequencer crash (§10.4), and is final in a window block within FINALITY_MAX_MS (§7.1).

The window stream. The box does not know a session's resp_seq at S until the tail arrives, so window answers use a separate stream: response header flags bit 1 = window, and resp_seq counts the session's window answers from 1 (Requests to 01, item 9). Every window answer is a pure function of the logged window packets, so replicas reproduce it and the recovery service keeps it (01 §14.6). A ResendReq with byte 1 bit 0 set asks for window answers. Final records after U use the ordinary stream, continuing from the state at S (F6).

Budget at message index 9 (REF, as 01 §14.2 with §14.17 item 1): 320 RX + 45 session and MAC + 10 × 10 collect + 40 replay slice + 170 logged cut (LaneCut, its ~20 ns poll included) + 10 frame gate + 20 response + 240 TX = 945 ns ≤ 980 ns (LAT_MAX_NS − DEADLINE_GUARD_NS). Deadline admission (01 §14.1) runs unchanged with the collect path as predicted_path_ns. A late packet gets Reject BUSY for each message in the window stream, and is logged with PacketRecord.flags.SHED in its window block; Apply skips it. Collect has no BBO stage, so a window packet can be shed only at the lane stage. B3's per-class bounds (01 §14.17 item 2: zero lane-stage BUSY on dedicated paced lanes, ≤ 1e-6 at ≤ 20% load on pooled lanes) hold in the window as everywhere, and pooled and dedicated lanes run the same collect path (§7.1).

Messages that cannot be queued. Each gets a definite answer within LAT_MAX_NS:

A cancel of a carried order is queued, not refused. The lane cannot confirm it, because the order's state at S has not arrived. But the cancel's place in the log is fixed, nothing trades between S and U, and Apply runs every queued op before the uncross. So a queued cancel of an order that rests at S always succeeds (CancelAck); it is TOO_LATE if the order filled in the tail, and UNKNOWN_HANDLE if no such order existed. A ResendReq for an epoch-e response that the retention buffer does not hold yet is queued the same way and served at activation.

8. Fencing (INV-FENCE)#

8.1 Epoch tags#

8.2 Rules#

• F1 (INV-FENCE). After a replica holds cert(e, S), it rejects every epoch-e step above S and every epoch-e block above last_height. It counts and alarms on each one; a signed one is evidence (09).
• F2. A member signs one HandoffRecord per (book, epoch), persisted before it is sent (§5.3).
• F3. A node signs consensus messages of epoch e only if home(e) is its home and it is in R_e (09 §2.6). After it signs the HandoffRecord of e, it signs nothing else of epoch e.
• F4. A node of R_{e+1} votes on no epoch-(e+1) block that holds a step before it has verified cert(e, S). Before that it votes only on window blocks, which hold PacketRecords and control-session records and no step, and whose anchor is the final EpoGo (§7.1). It accepts epoch-(e+1) frames only from the sequencer box whose key is next_seq_key, or from the standby after a failover in e+1 (§10.4).
• F5. A lane checks epoch16 first. A packet whose epoch16 is not the lane's active or collecting epoch is not MAC-checked with the lane's key and never closes a session. It gets at most one out-of-stream answer (§10.7). Reason: a client on the wrong side of a boundary holds the other epoch's key, and closing its session for that would punish a race the client cannot avoid. This is 01 §14.8: a bad MAC closes a session only from the bound address and with epoch16 equal to the receiver's epoch. After its seal the old home closes no session at all, because the sessions' state at S already belongs to e+1 (§10.7).
• F6. A response to a packet that does not enter the log is out of stream (resp_seq = 0, 04 P6). Only logged packets consume resp_seq, so the new home can continue every session's resp_seq from the state at S. Window answers use the window stream (§7.11) and consume no resp_seq.

8.3 Why clock skew cannot fork the book#

The history of epoch e ends at the certified S, and the history of e+1 starts from it. Clocks only decide when the seal happens and when collection starts. Take an old-home clock that is off by δ against the new home's:

• Old clock late (δ > 0). The old home seals at true time T + δ. For δ the old home still trades and the new home already collects. The collected orders do not execute until cert(e, S), and that certificate covers everything the old home did until its seal. The window grows by δ.
• Old clock early (δ < 0). The old home seals at T − |δ|. The new home starts at T. For |δ| nobody trades. The window grows by |δ|.
• A packet on both sides. A client may send one pkt_seq to both homes (a resend). If the old home applied it before S, the state at S holds hw_pkt_seq ≥ pkt_seq and the new home drops its copy (§7.3). If not, only the new home applies it. Either way it is applied once.

Two histories never exist: one certificate per (book, e) at most (§5.3), and every e+1 block extends it (F4). The alarm EPO_SKEW_ALARM_NS (1 µs; GPS-disciplined PTP keeps ~100 ns, vendor) protects fairness and the window length, not safety. vyrn-sim tests skew up to ±10 s (§12).

9. Keys and membership at the boundary#

9.1 Sequencer keys#

• Block signing. next_seq_key is SK(seq_index, home(e+1)) of the F1 sequencer box (09 §2.2). It is fixed in EpoGo and then in the HandoffRecord. The replicas of e+1 accept seq_sig only under this key until a failover in e+1 moves the role to the F2 standby, whose SK the roster also holds (§10.4).
• Session keys. K_seq(e+1, i) and K_out(e+1, i) come from X25519 between the session's key and XK(i, home(e+1)), then BLAKE3 derive_key with the epoch and i in the input (01 §4; 09 §5.3; 04 §4). Clients derive them ahead for both sequencer boxes i of the new home. Each box derives its side for every open session in Preparing: ~2 × 10^5 sessions at ≤ 25 µs each is 5 s of core time, 0.1 s on 49 lanes (budget). A session opened in the last 60 s is derived when the replay reaches its open record.
• Open cutoff. The old home accepts no OpenSession after T − EPO_OPEN_CUTOFF_MS (500 ms); it answers "retry at home(e+1)", and the new home serves the open after activation. Reason: 500 ms > t_wan + FINALITY_MAX_MS + EPO_READY_LAG_MS on every pair, so every session that can send in the window has its open record, and its keys, at the new home before T. No window packet then waits unverified. An open is not a 1 µs message (04: ≤ 50 ms); only opens in the last 500 ms wait for the window.
• K_com (CHAIN) is per (e, sequencer) and comes from 09's committee keys. It changes at the boundary like K_seq. 08 adds nothing.

9.2 Lane map#

lane_map(e+1) is computed by replicas (10 §1.3), fixed in EpoConfig(e+1) and published 1 h ahead. It gives each account a class, dedicated or pooled (01 §14.15), and a firm is promoted or demoted only here, at a boundary. At most ACC_MAX_MOVES_PER_EPOCH (1,024) accounts move per boundary, class changes included. An account that stays pooled keeps its pool queue and does not move. 04 binds a session to its lane for life, so for a moved account:

1. The client pre-opens a session on the new lane during epoch e, valid from e+1 (04), before the open cutoff (§9.1).
2. It trades on the old session until its own T, like every client, and on the new session at H′ from T. It does not quiesce. A packet still unacked on the old session at T has an answer on its way from H: an in-stream response if H applied it before S, or the out-of-stream EPOCH_MISMATCH if it arrived after the seal. The client resends only the rejected contents, as new packets on the new session. So nothing is lost and nothing applies twice, and the account loses no trading time beyond the window.
3. The old session closes at S (SESSION_CLOSED, detail EPO_D_LANE_MOVED). The new home knows lane_map(e+1), so it answers a packet on the old session within LAT_MAX_NS with that reject (§7.11).
4. from_state moves the account's balances and resting orders to the new lane. Each order keeps its PrioKey (§7.4), gets a new handle, and is re-acked on the new session after U with Ack{index: 0xFE} (Requests to 01, item 5). The client maps client_tag to the new handle.

9.3 Replica sets#

R_{e+1} is fixed in EpoConfig(e+1) and changes only at a boundary: in CHAIN by 09's governance (NodeCert.valid_from), in BASE by an admin record. A new box follows for at least one epoch before it is listed, so it passes EpoReady's lag and B4 checks. A box leaves after its home's last epoch.

9.4 Software upgrades#

10. Failure cases#

10.1 The old sequencer dies before the certificate#

If the dead sequencer's frames reached the new home but not the old replicas, the new home's S′ is beyond S. The new sequencer undoes to the last finalized block and re-applies (§4.5): +EPO_RESPEC_MS (15 ms) at worst. That costs time and never correctness, because nothing that depends on the speculative state was sent. Queued Acks do not depend on it (§7.11).

10.2 The new home has fewer than Q healthy replicas, or no standby#

Decision: do not hand off. The go/no-go logs EpoHold, the old home keeps epoch e, and every session gets a revised EpochNotice (flags bit1). The same holds when either sequencer box of the new home is not ready (§2.6). The attempt repeats every EPO_HOLD_S (900 s), and moves to the next slot's home when slot e+1 ends (§2.5). A hold costs no window and no fill.

If the new home loses quorum after EpoGo is final, the handoff proceeds: the certificate does not depend on the new home. The book is then certified at S and nobody is active. It stays halted until R_{e+1} has Q healthy members again (restart and follower catch-up, minutes) or until a reassignment (§10.3). This needs two voter faults after a healthy go/no-go 10 s before, which is beyond F. One voter fault, or one sequencer-box fault (§10.4), leaves the handoff on budget.

10.3 A WAN partition during the handoff#

The new home keeps queue-acking and finalizing window blocks until T + EPO_HALT_COLLECT_MS (250 ms; the certificate arrives by t_wan + 11.3 ms on the direct path, and by 201 ms on the slowest third-home route, LDN → TYO → NYC). If no certificate has arrived by then, the control core logs EpoHalt in a window block. From the cut after it, lanes answer New and Replace with Reject BOOK_HALTED within LAT_MAX_NS, and keep queueing Cancel, Reduce and CancelAll, which only reduce risk. When a path heals, activation proceeds with a longer window and the same uncross rule.

Decision: no automatic resumption at the old home. A node of R_e cannot tell a dead new home from a partitioned live one, and a live one activates as soon as it holds the certificate. After EPO_REASSIGN_MIN_S (60 s), operators may reassign epoch e+1 with EpoReassign{e+1, home, win_head}:

1. Each signing operator first fences its own node in the new home through its out-of-band management path, so that node can no longer sign for e+1. It then reads the node's persisted high-water mark and stops here if the mark shows armed(e+1). It also reads the node's highest final window block. If it cannot read the mark, it does not sign.
2. It then signs EpoReassign with its operator key (CHAIN: OIK, 09 §2.10 tightening; BASE: Q admin keys). win_head is the highest final window block that any of the Q fenced nodes holds.
3. Q signatures make it valid. The target home fetches the window chain to win_head and its payloads from the fenced nodes' log stores, then activates e+1 from cert(e, S), the reassignment and those window blocks.

A fill of e+1 at the new home needs Q armed members, and a reassignment needs Q operators whose members are not armed. Two quorums share an honest operator (§5.3), so exactly one of them can happen, and no fill sent by the abandoned home can be reverted. Every queued-acked order is in a final window block, which an honest fenced node of any Q holds (quorum intersection), so win_head covers it: no acked order is lost.

10.4 The new sequencer fails#

The new home's standby vyrn-seq in F2 takes over through 07's failover (SF-25), at any moment from T on. No voting replica is promoted, so the 2 + 2 (BASE) and 3 + 2 (CHAIN) client splits stay intact (01 §14.4).

• During the window, before activation. R_{e+1} detect silence (FAILOVER_SILENCE_MS) and finalize the longest window prefix they hold. Every queued-acked packet is in it, because its record left on both paths before its Ack (§7.11). The standby already follows the tail and has applied the final window blocks against S′ (§4.5), so it continues collection under its own key and activates when the certificate arrives. Window effect: none if the promotion ends before the certificate arrives; at most FAILOVER_TARGET_MS otherwise.
• After activation, the Uncross frame left on both paths. Every auction fill sent is in the log (INV-FRAME-FIRST). The standby replays the opening block and continues.
• After activation, it did not. No auction fill was sent. The standby uncrosses from the final window blocks and cert(e, S), with the same rule and the same result (INV-DET). Packets that were never queue-acked are not in the log; their clients resend them.

Clients switch tag_seq to K_seq(e+1, standby), which they derived ahead (§9.1), on the failover notice (04, 07 §9.5). After a promotion the home has no standby until operators restore one (07 §13); 03 places the next standby in F2.

10.5 A byzantine old sequencer signs two different final steps#

Two signed last blocks with different S cannot both be final: Simplex finalizes one block per height (07). Members sign a HandoffRecord only for the finalized block (§5.3 rule 1), so at most one certificate exists. The two seq_sigs on conflicting headers are equivocation evidence, slashed automatically in CHAIN mode (09, SF-19). A different tail sent only to the new home yields S′ ≠ S there and costs the re-replay of §10.1. A sequencer that never seals is force-sealed (§4.3).

10.6 Data-tier gaps at the new home#

10.7 Clients that miss the EpochNotice#

A client does not need the notice for the default schedule: it computes T from G and EPOCH_BOUNDARIES_UTC. The notice matters only for revisions, which vyrn-md and the admin channel (BASE) or chain state (CHAIN) also publish. A client that still sends to the old home after the seal gets, from the old sequencer in Redirect state, an out-of-stream response: EpochNotice (seconds 0) and Reject EPOCH_MISMATCH for each message (detail = the active epoch). The packet is MAC-checked with K_seq(e), is never applied, and is answered within LAT_MAX_NS, every packet, to the bound address only. The answer is at most 8 bytes longer than the request, so it cannot amplify an attack. A bad MAC is dropped and counted and closes nothing (F5). The old box keeps answering after it rejoins as a follower, between replay slices (§7.1), for as long as it runs: no packet that reaches a sequencer box goes unanswered.

11. Client migration#

11.1 The endpoint table#

At session open the client receives the endpoint table of all three homes in its key bundle (04 §4.4): per home and per roster index, the order address for each lane, the recovery address, the data nodes, XK and SK; plus the port base. Its hash is EpoConfig.endpoint_hash. When an EpochNotice has flags bit4 set, the client fetches the new table (admin channel or chain state) before T.

11.2 What the client does#

ASCII original

 Active(e, H) ── first EpochNotice(e+1, H′, s, seq_index) ──► Scheduled(T)
   prepare: T = now + s (or the schedule), K_seq/K_out(e+1, i) for both sequencer boxes i,
            own lane in lane_map(e+1); if moved: pre-open before the cutoff (§9.1, §9.2)
 Scheduled ── own clock ≥ T, or EpochNotice(0), or Reject EPOCH_MISMATCH from H ──► Switching
   1. stop sending to H
   2. resend every unacked packet to H′ in pkt_seq order: flags bit0 = 1, epoch16 = e+1,
      tag_seq under K_seq(e+1, seq_index), both copies (04 D2), logical content unchanged (04 D3)
   3. then send new packets to H′; each gets a queued Ack within 1 µs of reaching H′ (§7.11)
 Switching ── first response with epoch16 = e+1 (window or ordinary stream) ──► Active(e+1, H′)
   4. after U: ResendReq for any resp_seq gap older than EPO_DRAIN_MS, including late
      responses of epoch e (§11.3); window-stream gaps at any time (byte 1 bit 0)

A revised notice (flags bit0) restarts Scheduled with the new T. A packet that reaches H′ with rx_ts < T gets an out-of-stream Reject EPOCH_MISMATCH, so switching early gains nothing.

11.3 Exactly once across the boundary#

• A packet with pkt_seq ≤ hw_pkt_seq(S) was applied in epoch e. The new home drops every copy. Its responses are in the new sequencer's retention buffer, which the replay filled from T − EPO_PREP_LEAD_S on (INV-DET makes them identical to what the old home sent), and in the recovery service (04 P8).
• A packet with pkt_seq > hw_pkt_seq(S) was not applied in e (gapless pkt_seq). The new home applies it once, in the window or after U.
• resp_seq continues from the state at S, because only logged packets consume it (F6). Window answers have their own stream (§7.11), so the new home never needs resp_seq(S) before U.
• A queued-acked packet is in a final window block. No failure loses it (§10.3, §10.4).

So a client never needs the old home after T, and no order is lost or doubled.

11.4 Latency the client sees (TYO → LDN, typical, W = 112.6 ms)#

Every packet gets its first answer one round trip to H′ plus at most 1 µs after it is sent, in the window too; the gap is in fills only. The new home's local client sees almost no fill gap: it saw the old book 110 ms late all along. The old home's local client pays the window plus the return trip. Every client that reaches the new home before t_close competes in one uncross.

12. Testing#

12.1 Chaos plan in vyrn-sim#

vyrn-sim (11 §6) runs three homes with N_BASE or N_CHAIN voters and 2 sequencer boxes each, data tiers, WAN links with the §6 delays and a heavy-tailed jitter, and 10^4 simulated clients with distances to all homes. Epochs are compressed to 60 s of virtual time. Each scenario runs over all three transitions.

12.2 Pass criteria#

Every run must pass all of these. One failure fails the build.

Volume: 10^4 seeds per scenario per night on DEV and CI machines. On hardware, 11 §6.7 runs EPO-T2, T3, T5, T7 and T14 with netem WAN delay, and B13 and B14 measure §6.2 and §7.8.

Interfaces to other chapters#

Local definitions#

Constants (all budgets). The timeouts and periods are in §4.9.

Types: EpoConfig, EpoReady, EpoGo, EpoHold, EpoHalt, EpoForceSeal, EpoReassign, HandoffSig, HandoffCert, EpoArmed, EpoTailReq/EpoTailResp, EpoActivated, WinBlock, UncrossRecord, PrioKey, AuctionCfg, CollectBuf, trait EpochEngine. Invariants:

Requests to 01#

01 §14 resolved the earlier items on Seal/Uncross (§14.10), Fill.liq = 2 (§14.10), the Out causes with expired (§14.10), the MAC rule with epoch16 (§14.8) and sessions that span epochs (§14.8). These remain:

1. §6.4 EpochNotice: byte 1 = seq_index of the next epoch; bytes 2–3 = flags (bit0 revised, bit1 hold, bit2 skip, bit3 date roll, bit4 endpoint table changed).
2. §6.6 HandoffRecord: add next_epoch: Epoch at offset 120, and fix the 120-byte encoding and the signing context of §5.1–5.2.
3. §4: state that state_root in HandoffRecord is the lattice hash of the full state in both modes, and that log_hash is BLAKE3 of the BlockHeader at last_height.
4. §6.6: every order-stream frame header carries epoch: Epoch (INV-FENCE needs it per frame).
5. §6.4 Ack: index = 0xFE marks a re-ack of a carried order with a new handle after a lane move.
6. §3: add PubKey = [u8; 32] (Ed25519), Ed25519Sig = [u8; 64] and HomeCode = u8.
7. Promote EpoConfig (with seq_order: [u8; 2]) to §6.6 (10's Requests to 01, item 5, asks for the same record), and EPO_WAN_OW_MS, EPO_HIST_HALF and EPO_WINDOW_SLACK_MS to §5, since 02, 03 and 11 use them.
8. §6.4 Ack: bytes 2–3 = flags; bit 0 = queued (EPO_ACK_QUEUED), with handle = wref for a window New or Replace (§7.11).
9. §6.3: response flags bit 1 = window stream; resp_seq then counts window answers (§7.11).
10. §6.2: byte 1 bit 0 of Cancel, Reduce and Replace = the handle field holds a wref; of ResendReq = the range is in the window stream.
11. §6.6: add WinBlock (§7.1), and state in §14.0 that FINALITY_MAX_MS holds for window orders through it.
12. §14.5 and §14.15 item 3: with two order-stream cores, name the one that writes StepRecords (the Seal and Uncross records included, §4.2, §7.8), or state that each writes the step's record for its half. §14.5 still names a single core 4.
13. §14.15: state that a pool core steals nothing and logs no AcctTake from T until its lane is live (§7.1), so the window path is the same for both lane classes.
14. §5: N_LANES_REF = 50 against the §10 map's 49 lanes; this chapter's lane sums use 49.

Open questions#