07. Replication: log, order frames, data tier, Simplex, storage, failover

Draft 0.2, 2026-11-13. Normative for vyrn-seq stream cores 4 and 7, the frame gate of TX cores 3, 5 and 0, the order hasher (core 50) and the finality listener (control, core 51), vyrn-data, vyrn-replica, vyrn-validator (replication shell), vyrn-log, vyrn-consensus. It follows 01-conventions.md (§6.6 records, INV-FRAME-FIRST, INV-DET, INV-QUORUM) and the binding decisions of 01 §14. Rationale: DESIGN.md §5 and §7.

0. Decisions in one table#

1. Scope and hard constraints#

This chapter owns the following:

• the order-stream wire format
• the frame transport between vyrn-seq and the replicas
• the data tier
• block cutting and signing
• the use of Simplex, the vote rule and certificates
• replay inputs, snapshots and state roots
• storage, sequencer failover, Byzantine handling, the client-diversity alarm, WAN followers, recovery and membership

It does not own the following. 05-engine.md owns lane, cut and BBO semantics and the replay schedule. 09-chain.md owns the K_com tag rules, exclusion semantics and slashing. 08-epochs.md owns handoff and fencing across epochs. 04-protocol.md owns client-visible messages.

How this chapter meets the constraints:

• C-UNIX: every process here is a Linux userspace process. vyrn-sim runs the whole protocol on macOS with a simulated network.
• C-NOCHIP: frames are built, hashed and signed on CPU cores. The NICs only do what an ASIC NIC does as shipped: DMA, CTPIO, hardware timestamps, flow steering. No GPU hashes anything on the live path. No switch runs our logic. Switches forward, replicate multicast and apply static ACLs.
• C-COTS: the parts are Solarflare X2541 and X2542 and ConnectX-6 Dx/7 NICs, merchant 100GbE switches with PTP, and U.2/E3.S NVMe drives.
• C-ONEBIN: the warm standby box runs vyrn-seq in standby beside a non-voting vyrn-replica. Both link the same vyrn-engine, and promotion hands over the replayed engine arenas in place (§9.5). Voters run vyrn-replica only. AWS uses the same frames over ENA, using DMA in place of CTPIO (§4.4).

Boxes of one home (01 §14.4, §14.7).

The fault budget F = 1 counts every box of the home, so a faulty sequencer uses it up.

2. The log#

2.1 Two parts#

ASCII original

   client ──copy 1──► vyrn-seq─order stream (ORDER + HASH + HEADER frames, 2 paths)──┐
     │                                                                                ▼
     └────copy 2──► vyrn-data (session's data home) ──data frames──────────────► vyrn-replica
                                                                                     │
                     join key (session, pkt_seq); checks: pkt_hash, data_root ◄──────┘

The log of a home is the sequence of blocks. Block h is the triple (header h, the order bytes of h, the payloads referenced by h). Block h is final when it carries a finalization certificate (§6.6) or is an ancestor of a final block.

2.2 Identities#

3. Order-stream wire encoding#

3.1 Transport#

• UDP over IPv4 multicast runs on two dedicated VLANs, one per path: REP_VLAN_A and REP_VLAN_B. Path A uses switch A and path B uses switch B. The two paths have separate NIC ports, separate switches and diverse metro fibre routes. Nothing else shares those switches.
• Each box b has its own group pair, G_ord(b, A) and G_ord(b, B). A static ingress ACL on each switch admits traffic to G_ord(b, ·) only from box b's replication ports. A replica subscribes to the groups of the current sequencer. At failover it changes its subscription. The ACLs never change.
• UDP ports: REP_UDP_ORDER = 7400 (frames), REP_UDP_CTRL = 7401 (NACK, acks, retransmits, unicast), REP_UDP_DATA = 7402 (data frames), REP_UDP_REPAIR = 7403 (peer repair). Consensus, TailReports and state-root votes use commonware-p2p (authenticated, encrypted, TCP) on the same VLANs.
• MTU 9216 on both VLANs. ORDER frames are ≤ 250 B on the wire (REP_ORDER_FRAME_MAX, §4.5), because CTPIO write time grows with size and bounds the gate (§4.7). HASH frames are ≤ 9000 B and go by DMA.
• Frames carry no per-frame MAC or CRC. Ethernet FCS checks each hop. order_root checks every block within ≤ 1 ms (§6.1). A frame that fails the root is repaired (§4.10). Section 10.2 gives the authenticity argument.

3.2 Frame header (24 bytes, little-endian)#

off size field
  0   2  magic       u16   0x4250 ("PB")
  2   1  version     u8    1
  3   1  kind        u8    1 ORDER, 2 HASH, 3 HEADER, 4 HEARTBEAT
  4   2  book        BookId
  6   2  epoch16     u16   low 16 bits of the epoch
  8   8  frame_seq   u64
 16   1  flags       u8    bit0 path (0 A, 1 B); bit1 retransmit; bit2 peer-repaired; others 0
 17   1  tenure      u8
 18   2  body_len    u16   bytes of body after this header
 20   1  sender      u8    sequencer index (§2.2)
 21   1  stream      u8    0 A, 1 B (§2.2)
 22   2  reserved    0
 24   …  body        sections (§3.4) or a header (§3.10)

flags is the only field that differs between the two path copies and retransmissions of one frame. INV-REP-3: for a fixed (epoch, tenure, stream, frame_seq), the bytes at offsets 0–15 and 17–end are identical on both paths and in every retransmission.

3.3 Frame kinds#

3.4 Sections#

Every section starts with one tag byte. Sections never straddle frames.

N_LANES ≤ 64 is required by the 6-bit lane tag. REF has 49 lanes (0–48, 01 §10; 01 §5's N_LANES_REF = 50 is a request below) and N_LANES_CONSUMER = 11. A LaneRun appears only on the substream of its lane's half.

3.5 Decoder state#

The encoding is a delta against state that both sides hold. The encoder state lives in each lane for its own records (lanes pre-encode, §4.2) and in each stream core for the step records of its substream. The decoder state lives in the replica, one StepDec per substream. Both are part of the snapshot (§7.4).

struct LaneDec {                       // one per lane
    next_lane_seq: LaneSeq,            // first_lane_seq expected for the next PacketRecord
    prev_rx_ts: Nanos,
    prev_fills_applied: StepId,
    slot_session: [SessionId; MAX_SESSIONS_PER_LANE],   // set by SessionBind
    slot_next_pkt_seq: [u32; MAX_SESSIONS_PER_LANE],
    awaiting_hash: Fifo<RecordRef>,    // PacketRecords whose pkt_hash has not arrived
}
struct StepDec {                       // one per substream H
    prev_step: StepId,                 // the last step this substream carried a record of
    prev_cut: [LaneSeq; N_HALF_MAX],   // this half's lanes only
}
const N_HALF: usize = (N_LANES + 1) / 2;   // 25 on REF: half A = lanes 0–24 (cut core 1, stream
                                           // core 4), half B = lanes 25–48 (cut core 6, stream core 7)

TenureStart sets every base (§3.9). There are no keyframes. Decoding is strictly sequential in frame_seq order, and a gap stalls decoding until it is repaired (§4.9). The stall is bounded by repair time (≤ 100 µs in a facility, ≤ 600 µs across the metro, budget). Keyframes would cost bandwidth on every block and would still need the slot map, which is epoch-scoped.

3.6 PacketRecord (O-part) encoding inside LaneRun(l)#

byte h:   bits 0–3  count              (0..10; 11..15 → REP_E_COUNT); numbered messages (05 §3.1)
          bit 4     SEQ_EXC            pkt_seq ≠ slot_next_pkt_seq[slot]
          bit 5     LSEQ_EXC           first_lane_seq ≠ next_lane_seq
          bit 6     FA_NZ              fills_applied ≠ prev_fills_applied
          bit 7     SHED               PacketRecord.flags.SHED (01 §14.1): the lane shed the packet
slot      LEB128 u16                   lane-local session slot, < MAX_SESSIONS_PER_LANE (pool-wide
                                       on pooled lanes); REP_SLOT_ADVANCE = MAX_SESSIONS_PER_LANE marks
                                       an Advance record, REP_SLOT_TAKE = MAX_SESSIONS_PER_LANE + 1 an
                                       AcctTake (below)
rx_ts     ZigZag LEB128 of (rx_ts − prev_rx_ts) as i64
[FA_NZ]   LEB128 (fills_applied − prev_fills_applied)          // non-decreasing per lane
[LSEQ_EXC] LEB128 (first_lane_seq − next_lane_seq)             // > 0; lane ops that are not packets
[SEQ_EXC] LEB128 pkt_seq                                       // absolute; never in an Advance record

Decoding rebuilds the full logical record of 01 §6.6:

fn decode_pkt(l: LaneId, d: &mut LaneDec, r: &mut Reader) -> Rec {   // Rec::Pkt or Rec::Take
    let h = r.u8();
    let count = h & 0x0F;
    let shed = h & 0x80 != 0;
    if count as usize > MAX_MSGS_PER_PKT { fail(REP_E_COUNT) }
    let slot = r.leb_u16();
    if slot as usize == REP_SLOT_TAKE {                          // 01 §14.17 item 4: pooled lanes only
        if h != 0 || !is_pool_lane(l) { fail(REP_E_COUNT) }
        return Rec::Take(AcctTake { account: AccountId(r.leb_u32()), ver: r.leb_u64(), lane: l });
    }
    let advance = slot as usize == REP_SLOT_ADVANCE;             // 05 §6.2: a boundary, no packet
    if advance && (count != 0 || shed || h & 0x30 != 0) { fail(REP_E_COUNT) }
    let rx_ts = d.prev_rx_ts.checked_add_signed(r.zigzag_i64()).unwrap_or_else(|| fail(REP_E_TS));
    let fills_applied = if h & 0x40 != 0 { d.prev_fills_applied + r.leb_u64() } else { d.prev_fills_applied };
    let first_lane_seq = if h & 0x20 != 0 { d.next_lane_seq + r.leb_u64() } else { d.next_lane_seq };
    d.prev_rx_ts = rx_ts; d.prev_fills_applied = fills_applied;
    d.next_lane_seq = first_lane_seq + count as u64;             // a shed packet keeps its LaneSeqs
    if advance {
        return Rec::Pkt(PacketRecord { lane: l, count: 0, flags: ADVANCE, session: 0, pkt_seq: 0,
                              rx_ts, pkt_hash: PktHash(0), first_lane_seq, fills_applied });
    }
    let pkt_seq = if h & 0x10 != 0 { r.leb_u32() } else { d.slot_next_pkt_seq[slot] };
    d.slot_next_pkt_seq[slot] = pkt_seq + 1;
    let rec = PacketRecord { lane: l, count, flags: if shed { SHED } else { 0 }, session: d.slot_session[slot],
                             pkt_seq, rx_ts, pkt_hash: PktHash::PENDING, first_lane_seq, fills_applied };
    d.awaiting_hash.push(rec.r#ref());                           // shed packets are hashed and joined too
    Rec::Pkt(rec)
}

rx_ts uses ZigZag because a lane polls up to 4 NIC RX queues, so its processing order is not timestamp order. SEQ_EXC covers a resend and the first packet after a SessionBind whose next_pkt_seq is unknown. LSEQ_EXC covers lane ops that are not packets, if 05-engine.md defines any.

• SHED (01 §14.1). Deadline admission answered the whole packet Reject BUSY. Its messages keep their LaneSeqs, so LaneCut gates the answers (§4.11). Replay reads the bit and never recomputes the decision (INV-DET). The payload is still joined, so data_root covers it.
• flags is 01 §6.6's byte: bit 0 SHED, bit 1 ADVANCE. Neither bit costs a byte on the wire: SHED is bit 7 of h, and ADVANCE is the slot value REP_SLOT_ADVANCE. Every other bit is 0.
• Advance record (05 §6.2, 01 §14.11): flags = ADVANCE, count = 0, session = 0, pkt_seq = 0, pkt_hash = 0. It carries only rx_ts (05: the lane's previous prio_ts) and fills_applied. It has no payload, no HashRun entry and no data_root entry. A real packet whose messages take no number also has count = 0, but it has a real slot and a payload.
• AcctTake { account, ver, lane } (01 §14.17 item 4, 05 R15). A pooled lane logs one each time it takes an account's token, in its LaneRun just before the first record that uses the account: h = 0, slot = REP_SLOT_TAKE, then account LEB128 and ver LEB128 (the token's version after the take). It takes no LaneSeq, no rx_ts, no hash and no payload. Replay lane l applies its next record only after the account's previous holder has applied every record before its own take of ver − 1, so the pool's work order is replayed exactly. A take on a dedicated lane is REP_E_COUNT. Size ~7 B. A take happens only when an account changes pool core, ≤ 3M/s at peak, ~0.2 B per record over the whole book (budget, §3.13).
• Pooled slots. On a pooled lane slot indexes the pool's shared session table, and slot_next_pkt_seq is kept per pool, so consecutive packets of a session may sit on different pool lanes. A packet framed out of pkt_seq order across pool lanes takes SEQ_EXC.

Size (budget): h 1 + slot 1.6 + rx_ts 2.2 + exceptions 0.2 ≈ 5.0 B. Budget 6 B. An Advance record is ~5 B; at full load its rate is near zero (05 §6.2). The LaneRun header adds 2 B per lane per frame.

3.7 HashRun (H-part)#

HashRun(l, k) gives pkt_hash for the next k records in LaneDec[l].awaiting_hash, in order. Lanes compute the full BLAKE3-256 of each packet in batches of REP_HASH_BATCH = 16 with the hash_many SIMD path. They push the 8-byte prefix to their half's stream core and the 32-byte hash to the order hasher on core 50 (§4.6). INV-REP-8: the HashRun for a record is in a frame sent at most REP_HASH_LAG_NS = 4000 ns (budget) after the record's ORDER frame, on the same substream. A replica raises REP_E_HASHLAG if a record waits more than 1 ms. That is a sequencer fault.

The H-part is not part of order_root (§6.1). It is implied by data_root, which covers the full hashes. The 8-byte pkt_hash is used for three things: the lookup key into the data store, an early per-packet check, and the only check available for an unsigned failover tail (§9.3).

3.8 StepRecord encoding inside StepRun#

A step is logged as one or two half records. Its home record (kind, source, payload, and the home half's cut delta) goes on the substream of the home half: the half of src_lane, or A when src_lane = 0xFF. When the other half's cut also moved at that step, the other substream carries a cut-only record with that half's delta. So each stream core frames only its own lanes' records and its own half-cut, and a step that spans both halves is the one with two records (§4.2 gives its gate).

byte 0:  bits 0–3  kind        0 Cross, 1 Improve, 2 Depletion, 3 Rescan, 4 Idle, 5 Shed,
                               6 Control, 7 Seal, 8 Uncross; 9–15 → REP_E_KIND (01 §14.10);
                               a cut-only record repeats the home record's kind
         bit 4     HOME        1 home record, 0 cut-only record
         bits 5–6  mode        this half's cut delta: 0 SAME, 1 DENSE, 2 SPARSE, 3 → REP_E_CUT
         bit 7     OTHER       the other substream carries a record of this step too
gap      LEB128 (step − prev_step − 1)                      // steps this substream does not carry
[HOME]   src_lane u8 (0xFF = none); [src≠0xFF] ZigZag LEB128 (src_lane_seq − cut[src_lane]);
         [payload] kinds Rescan, Control, Seal, Uncross only: plen LEB128, then plen bytes (05 owns
         them: the Rescan band, the Control code and argument, the uncross cut reference)
[DENSE]  |H| × LEB128 (cut[l] − prev_cut[l])                 // ≥ 0 by INV-CUT-MONO per half
[SPARSE] bitmap ceil(|H|/8) bytes (bit i = lane H.first + i changed),
         then LEB128 (cut[l] − prev_cut[l]) for each set bit, in lane order; each value ≥ 1

step is prev_step + 1 + gap on that substream; StepId stays gapless per epoch across the two, and TenureStart sets each substream's base. The step's cut is the union of two half-cuts (01 §14.3): half A from cut core A (core 1) and half B from cut core B (core 6). A half with no record at step s keeps its previous half-cut. The replica merges the two StepRuns by StepId: step s is complete when its home record and, if OTHER is set, its cut-only record are decoded. INV-CUT-MONO holds per half, so it holds for the union. For each record the encoder picks SAME when the half is unchanged, and otherwise the shorter of DENSE and SPARSE. Both clients must decode every form. The choice does not affect replay, because replay consumes the decoded cut. A decoder rejects a SPARSE delta of 0, any decrease, a cut-only record with mode SAME, and a home record whose src_lane is not in its substream's half (REP_E_CUT). The clamped step-cut rule of 05 (01 §14.11) is applied before encoding, so the logged cut is the clamped one.

Size (budget): each cut core publishes every ≤ CUT_REFRESH_NS = 100 ns, so each half changes ≤ 10M times/s and the two halves ≤ 20M times/s together. A changed half moves ~2 ops per lane, so its DENSE form is ~25 B + 1 B. A home record is ~4 B with its gap byte, and ≤ 10M cut-only records/s add ~2 B each. Per step at 30M steps/s: 4 B + 20M × 26 B / 30M + 10M × 2 B / 30M ≈ 22 B, summed over both substreams. Budget 23 B.

3.9 BlockMark and TenureStart#

BlockMark:   height LEB128, last_step LEB128 (delta from the previous BlockMark's last_step),
             lane_bounds: bitmap ceil(|H|/8) + LEB128 delta per set bit, this half's lanes only
             // lane_bounds[l] = last lane_seq of lane l whose PacketRecord precedes this mark
TenureStart: epoch u32, tenure u8, sender u8, first_step u64, first_height u64,
             prev_header_hash [32], then for each lane of the half: next_lane_seq LEB128,
             rx_ts_base u64, fills_applied_base LEB128; then prev_cut: |H| × LEB128 (absolute)

Each substream ends block h's ORDER bytes with its own BlockMark(h), with the same height and last_step. Both cut the block at the same step: when core 4's PTP-disciplined clock crosses k × BLOCK_NS, it posts a close request, the BBO core puts a mark entry into R_step after its current step, and each stream core appends its mark when it reaches that entry (§6.1). The mark makes the boundary part of the stream. Replay therefore never reads a clock (INV-DET, 01 §11.3). TenureStart is the first section of frame 1 of each substream in every tenure.

3.10 HEADER frame: BlockHeader bytes and seq_sig#

off size field                         (01 §6.6, field order kept)
  0   8  height        Height
  8   4  epoch         Epoch
 12   2  book          BookId
 14   8  first_step    StepId          = previous header's last_step + 1
 22   8  last_step     StepId          = first_step − 1 for a block with no step
 30  32  prev          Hash            header_hash(height − 1)
 62  32  order_root    Hash            §6.1
 94  32  data_root     Hash            §6.1
126  64  seq_sig       Ed25519Sig      Sign(seq_key, "vyrn      v1 block" ‖ bytes 0..126)
190      end

seq_key is the sequencer key that REP_Membership (§13) registers for the sender. A seal block (§9.4) has seq_sig = 64 zero bytes and is valid only with its seal certificate.

3.11 Worked example#

Lane 7 has a packet with 10 messages, slot 300, the expected pkt_seq, rx_ts +512 ns after its previous packet, and fills_applied unchanged. Then the BBO core takes a Cross from lane 7 whose op is inside the new cut. Half-cut A moves lanes 3, 7 and 12 by +2, +10 and +1. Half-cut B is unchanged.

LaneRun   07 01                    tag lane 7, k = 1
  record  0A AC 02 80 08           h = count = 10; slot 300 = AC 02; zigzag(512) = 1024 = 80 08
StepRun   40 01                    tag, m = 1
  step    50 00 07 00              Cross 0 | HOME (1<<4) | SPARSE (2<<5), OTHER 0; gap 0; src 7; zigzag(0)
          88 10 00 00              half A bitmap: bits 3, 7 (0x88), bit 12 (0x10); 25 lanes = 4 bytes
          02 0A 01                 deltas for lanes 3, 7, 12

All of it is on substream A, because lane 7 is in half A and half B did not move. That is 20 bytes for one packet of 10 messages and one cross. 01's logical records hold 43 B and 410 B respectively (the cut alone is 49 × 8 B).

3.12 Decoder loop (replica core 3)#

fn on_frame(f: &Frame, st: &mut DecState) -> Result<(), RepErr> {
    match f.kind {
        ORDER => for sec in f.sections() { match sec.tag {
            0x00..=0x3F => { check_half(sec.tag, f.stream)?;           // a lane only on its half's substream
                             for _ in 0..sec.k { let r = decode_pkt(sec.tag, &mut st.lane[sec.tag], sec);
                                                 st.emit(r); } },
            0x40 => for _ in 0..sec.m { let h = decode_step_half(&mut st.step[f.stream], sec)?;
                                        st.merge_step(h)?; },        // emits Rec::Step when complete
            0x42 => { let m = decode_mark(f.stream, sec)?; check_mark(&m, st)?; st.close_block_half(m); },
            0x43 => st.bind(sec.lane, sec.slot, sec.session, sec.next_pkt_seq),
            0x44 => return Err(REP_E_TENURE),        // only valid as the first section of frame 1
            _    => return Err(REP_E_TAG),
        }},
        HASH => for run in f.hash_runs() { st.attach_hashes(run)?; },   // checks against data copies
        HEADER => st.on_header(BlockHeader::parse(f.body)?)?,           // §6.4
        HEARTBEAT => st.on_heartbeat(f),
    }
    st.order_root[f.stream].update_bytes(f.order_body());   // ORDER section bytes only, per substream
    Ok(())
}

check_step enforces the structural rules that a vote needs (§6.4): INV-CUT-MONO per half, no cut[l] above the lane's decoded next_lane_seq − 1, a src_lane_seq that names a decoded op of src_lane, the src_lane and payload rules of the kind table (§3.8), and no Shed step for an op whose packet carries SHED (it was never forwarded). Every error is a sequencer fault. The replica stops voting for the block, records the error with (epoch, tenure, stream, frame_seq) and raises the sequencer-fault alarm (§10.1).

3.13 Bandwidth budget per link (peak, budget)#

The inputs are 1B msgs/s, 100M pkts/s (10 msgs/pkt), 30M steps/s (10M crosses plus 20M improves or depletions), 20M half-cut changes/s and ≤ 12M ORDER frames/s over both substreams (one flush per gated step and substream at worst, a step that spans both halves flushing both; the 160 B body cap needs ≥ 8.9M/s). Advance, Shed and Control records are near zero at peak. A path is the pair of replication ports on one leaf, one per substream (06 §2.3); the table sums both.

Record payload (O + H + steps) is 2.24 GB/s ≈ 17.9 Gbps. Frame overhead is the rest, and it falls when flushes coalesce under load (§4.5). On the sequencer each path is two 100GbE ports at ≤ 14% each; at a replica each path is one port at ≤ 27%, carrying both substreams. Replicas send the sequencer ≤ 1 Mbps of acks and NACKs.

Other links at peak (budget):

4. Order frames from the sequencer#

4.1 Cores and rings#

The order stream has two substreams (01 §14.15 item 3). Stream-A (core 4) owns lanes 0–24 and stream-B (core 7) owns lanes 25–48. Each owns one port on each path (06 §2.3: port a and port b of the two X2542 cards), its own frame_seq, frame ring, heartbeats, HASH frames, NACK service and released word, and the LaneCut of its lanes. Stream-A also closes blocks, signs headers and runs the finality gate. The three fill TX cores (A = 3, B = 5, C = 0, 01 §14.17) each own client ports (06 §2.3) and check both release words.

ASCII original

 half A: lanes 0–24 (cores 8–32)                  CCD 0
 ┌──────────────┐ R_pkt[l], R_hash8[l]           ┌──────────────────────────────────────────┐
 │ lane l       ├───────────────────────────────►│ core 4  stream-A                         │
 │  parse, MAC, │                                │  CTPIO → port A.a, port B.a; sfence      │
 │  risk,       │                                │  released[A]; LaneCut[l ∈ A]             │
 │  classify,   │                                │  block close, signer, HEADER, fence, gate│
 │  encode O,   │ half B: lanes 25–48            ├──────────────────────────────────────────┤
 │  AcctTake,   │ (cores 33–49, pool 56–62)      │ core 7  stream-B                         │
 │  hash batch  ├───────────────────────────────►│  CTPIO → port A.b, port B.b; sfence      │
 └──────┬───────┘                                │  released[B]; LaneCut[l ∈ B]; fence      │
        │ R_hash32[l]                            └──────▲───────────────▲───────────────────┘
        └────────────────► core 50 order hasher ◄───────┼─ frame rings  │ R_step (+ need, marks)
 ┌──────────────┐ R_step: one writer, three cursors ────┴───────────────┘
 │ core 2  BBO  │ R_fill_X: fills tagged (s, need_A, need_B), X = the TX core of the recipient's port
 │  step s      ├──────► core 3 TX-A, core 5 TX-B, core 0 TX-C
 └──────────────┘        wait: released[A] ≥ need_A(s) and released[B] ≥ need_B(s), then CTPIO fill
 lanes: every response of op q waits for LaneCut[l].cut ≥ q (written by l's stream core after sfence)

Every ring is SPSC except R_step (one writer, three cursors, INV-WRITER). Lanes encode their own records, because lane l's encoder state is lane-local: ~5 ns per packet, 1.7% of its budget.

need_H(s). It is the newest step ≤ s whose records on substream H a response of s depends on: s itself when H carries a record of s (H is the home half, or H's half-cut moved at s), else need_H(s − 1). The BBO core keeps both values as it pushes each step, at ~1 ns.

4.2 The gate protocol#

Lane l, per packet (05 owns the surrounding loop):

let enc = lane_enc.encode(&pkt_rec);            // §3.6, against the lane's LaneDec mirror
R_pkt[l].push(enc);                             // (1) plain store into the ring slot, then ring tail Release
lane_counter[l].store(new_lane_seq, Release);   // (2) publish; the cut core may now include the op
// ... every response of the packet waits for LaneCut[l].cut ≥ its last LaneSeq (§4.11)
hash_batch.push(pkt);                           // full BLAKE3 done 16 at a time, off the response path

Lanes encode in pass 1 (01 §14.2), so (1) precedes the BBO push of any forwarded op of the packet. A pooled lane pushes its AcctTake into R_pkt[l] before the first record that uses the token.

BBO core (core 2), per step s:

// as soon as kind, src and the (clamped) cut are fixed, before any fill (05 states the point per kind)
let need = need.advance(s, home_half(src_lane), moved_halves(&cut, &prev_cut));   // §4.1
R_step.push(StepEntry { rec: StepRecord { step: s, kind, src_lane, src_lane_seq, cut }, need });
for fill in step_fills { R_fill[tx_of(fill.lane)].push(fill.tagged(s, need)); }   // Release on tail
step_done.store((s, fill_lanes), Release);                                       // step end
if let Some(h) = close_req.take() { R_step.push(StepEntry::mark(h, s)); }       // §3.9

Stream core H (4 or 7), main loop:

loop {
    // A. steps: drain this half's lanes up to each step's half-cut, then append this half's record.
    while let Some(e) = R_step.peek_acquire(H) {
        if let Some(h) = e.mark() { frame.append_mark(h, e.last_step, lane_bounds(H)); R_step.pop(H); continue; }
        if e.need[H] == e.step {                             // H carries a record of this step
            for l in lanes_changed(H, &e.cut, &framed_to) {  // only this half's lanes whose cut moved
                let upto = max(e.cut[l], if e.src_lane == l { e.src_lane_seq } else { 0 });
                while framed_to[l] < upto { frame.append_lane(l, R_pkt[l].pop_acquire()); } // INV-REP-2
            }
            if frame.len() + e.enc_len(H) > REP_ORDER_FRAME_MAX { flush_order(); }
            frame.append_step_half(H, &e);                   // home or cut-only record (§3.8)
            pending_gate = e.step;
        }
        seen = e.step;
        R_step.pop(H);
    }
    // B. records ahead of any cut, so that the next step's drain is short; never a flush on its own
    //    while the BBO core is mid-step (step_busy), so a gated step never queues behind it
    for l in round_robin_slice(H) { frame.append_lane_all_upto(l, &mut R_pkt[l], REP_ORDER_FRAME_MAX); }
    // C. flush rule (§4.5)
    if pending_gate > released[H] || (frame.age_ns() >= REP_FLUSH_MAX_NS && !frame.is_empty() && !step_busy()) {
        flush_order();
    }
    // D. LaneCut for this half's lanes that received fills in released steps (step_done.fill_lanes)
    publish_fill_lane_cuts(H);
    // E. background, each item ≤ REP_SIGN_SLICE_NS: HASH frames, NACKs, heartbeats, acks RX,
    //    self-fence; on core 4 also block close, one signer slice and the finality gate (§4.8, §6.8)
    background_slice(H);
}

fn flush_order() {
    if seq_fenced.load(Acquire) || !self_fence_ok() {  // INV-REP-6: REP_FENCE_ACKS fresh voter acks
        seq_fenced.store(true, Release); go_silent(); return;   // silences both stream cores
    }
    let bytes = frame.seal(H, next_frame_seq(H));       // header + sections, into frame ring H
    ctpio_write(PORT_A[H], bytes);                      // WC stores into this core's port on path A
    ctpio_write(PORT_B[H], bytes.with_path_bit());      // and on path B
    sfence();                                           // both WC buffers drained to the uncore
    released[H].store(seen, Release);                   // every record H carries for steps ≤ seen is out
    for (l, c) in frame.cut_lanes() { lane_cut[l].store16((pending_gate, c)); }  // the ack gate (§4.11)
}

Fill TX core X (3, 5 or 0), per fill:

let (s, na, nb) = (fill.step, fill.need_a, fill.need_b);
while released[A].load(Acquire) < na || released[B].load(Acquire) < nb { core::hint::spin_loop(); } // INV-REP-1
ctpio_write(client_port(fill), fill_frame);

A step that spans both halves has need_A(s) = need_B(s) = s: it has a record on each substream, so its fills wait for the later of the two releases. The two stream cores frame it in parallel, each from its own R_step cursor, so the wait is the slower core's, not the sum (§4.7). A step that touches one half waits for that half's core, plus the other half's need, which is an older step that is normally released long before.

4.3 Why INV-FRAME-FIRST holds#

INV-REP-2 (drain before step, per half): stream core H appends its record of step s only after it has appended every PacketRecord of its half with lane_seq ≤ cut_s[l], and the src op when src_lane is in H. The records are present by happens-before. The lane writes R_pkt[l] (1) before its release store of the counter (2). The cut core acquires (2) and then releases the cut. The BBO core acquires the cut and then releases R_step. Core H acquires R_step. So when core H reads step s, every R_pkt entry of its half at or below cut_s is visible. The src op of a cross may be above the cut. Its entry was pushed before the lane pushed the order to the BBO ring, so it is visible too.

INV-REP-1 (order before response): a TX core writes a fill of step s only after it reads released[A] ≥ need_A(s) and released[B] ≥ need_B(s) with acquire. Core H stores released[H] with release after the sfence that follows the CTPIO writes of its frame on both paths, and its frames are flushed in frame_seq order. Every record that a response of s depends on is on the substream of its lane's half, and the newest step whose records on H it needs is need_H(s) by definition. So the step's records, every record in its cut, and every earlier frame of both substreams that holds them, are written to both replication TX queues before any fill of s is written. That is INV-FRAME-FIRST (01 §7, §14.16). Responses that a step produces other than fills (05 decides whether an improve's Ack comes from a TX core) pass the same check.

INV-REP-11 (every lane response is gated): a lane of half H writes a response of op q only after it reads LaneCut[l].cut ≥ q with acquire. Core H stores LaneCut[l] = (s, C_s[l]) only after the sfence of its frame that holds its record of step s, which logs half H's cut, and that frame follows every PacketRecord of lane l up to C_s[l] (INV-REP-2). So the op's record, and the record that logs the cut covering it, are on both TX paths before the response. An ack needs only its own half: if a crash loses the home record of s on the other substream, the seal drops the orphan cut-only record and q stays in the prefix as a trailing lane op (§9.4), so an acked order still survives. This covers joins, cancels, rejects, BUSY answers and settled maker fills: every response (01 §14.5). INV-ACK-AFTER-CUT becomes "after a released cut includes the op".

Test: vyrn-sim runs a model checker over the seven threads (a lane of each half, cut, BBO, stream-A, stream-B, one TX core) with loom-style exhaustive interleavings on a 2-lane, 3-step workload that includes a spanning step (T-REP-1). On hardware, B-REP-2 checks INV-FRAME-FIRST for every response, acks included, and asserts zero violations (§4.4).

4.4 Why CTPIO, and the frame lead#

The decision is that ORDER and HEARTBEAT frames go by CTPIO on the REF box, and HASH and HEADER frames go by DMA.

• With DMA, the doorbell would be written first, but the NIC fetches the descriptor and the data over PCIe (~0.6–1.0 µs, vendor). The fill goes out by CTPIO in ~250 ns. The fill would lead its frame on the wire by ~0.5 µs, and a power loss in that window loses the frame.
• With CTPIO, the frame's bytes are in the replication NIC before the fill's first byte is written to the client NIC. Both start transmitting ~250 ns after the write (vendor). So the frame leaves the box no later than the fill, except for PCIe and NIC jitter.

Two different checks. INV-FRAME-FIRST is a correctness invariant the code enforces: the TX cores and the lanes wait on released[A], released[B] and LaneCut. B-REP-2 logs the frame's and the response's TX-queue writes for every response, acks included, and asserts zero violations. It is pass/fail, never a percentile.

The wire lead is a separate, physical quantity: lead(r) = t_wire(response r) − t_wire(ORDER frame that r depends on), from the hardware TX timestamps of both NICs, which share one PHC source on one box (06). The written order cannot make the NICs transmit in order, so lead < 0 can occur through PCIe or NIC jitter. It matters only if the box loses power in that window. B-REP-2W measures its distribution, and §9.6 (c) uses the measured fraction.

On AWS (ENA via DPDK), frames go by DMA. The gate still holds as written, at the doorbell. The in-box exposure of §9.6 is then ~1 µs per sequencer death instead of ~0. This is published for the AWS target (01 §14.0 fact 1). On CONSUMER, path B runs on its own X2522 port (01 §14.7) with CTPIO, about 6 Gbps at ~250M msgs/s, so CONSUMER has two independent paths and the same revert analysis as REF.

4.5 Flush policy#

A frame is flushed when any one of these holds:

1. A step that can release a response has been appended (pending_gate > released): it has fills, or it moved a half-cut (any mode ≠ SAME), since a moved cut releases lane responses. This is the latency-critical trigger.
2. The next section would take the frame body past REP_ORDER_FRAME_MAX = 160 B.
3. The oldest record in the frame is REP_FLUSH_MAX_NS = 1000 ns old, and the BBO core is not mid-step.

The rules apply per substream. There is no minimum interval. Under load, coalescing comes for free: while a stream core writes one frame (≤ 32 ns busy), further steps queue in R_step, and they go into its next frame. The ORDER frame rate is therefore ≤ min(gated steps/s, 1 / t_flush) ≈ 6M/s per substream and ≤ 12M/s together at peak. A step with a SAME cut and no fills does not trigger a flush.

4.6 Core 50: order hasher#

Core 50 (CCD 6, the order hasher of 01 §10) does three jobs:

• It follows both frame rings behind the stream cores and keeps order_root_A(h) and order_root_B(h), BLAKE3 incremental hashes over each substream's ORDER section bytes of block h (§6.1).
• It follows R_step (its own cursor) and the decoded O-order of each substream. From R_hash32[l] it appends each packet's full 32-byte hash to that substream's data_root hasher, in O-order.
• It keeps the full-hash list of each block for REP_SEQ_HASHLIST_MS = 50 ms (~3.2 MB per block at peak) and serves HashListReq (§5.6).

When both substreams' BlockMark(h) have passed, it hands (order_root, data_root) to core 4 through an SPSC word pair. It reads the rings across CCDs, prefetched; it is on no response path, so the distance costs only seal latency (§6.2). Load (budget): 2.24 GB/s + 3.2 GB/s of BLAKE3 at ~0.5 cycles/byte ≈ 2.7 G cycles/s, about 54% of a 5 GHz core (B-REP-6).

4.7 Load and latency budget of a stream core#

Per stream core, each with half the lanes (budget):

This is above the ≤ 0.25 of 01 §14.15 item 3: the CTPIO writes per frame do not halve when the lanes do (Requests to 01). The gate bound below does not depend on load.

The gate as a maximum. 01 §14.2 charges a cross 100 (step) + 60 (frame gate) + 40 (TX core), so the gate must be open ≤ 160 ns after the step starts, always. Three rules make that a bound and not a percentile:

1. Early push. The BBO core pushes StepRecord s when kind, source and cut are fixed, ≤ REP_KIND_FIX_NS = 10 ns into the step (05 states the point per kind). The stream cores frame it while the BBO core is still matching.
2. Small frames. REP_ORDER_FRAME_MAX = 160 B of body (≤ 250 B on the wire), so one flush keeps a stream core busy ≤ REP_FLUSH_BUSY_NS = 32 ns (seal 6 + 2 × ≤ 8 of CTPIO + sfence 10).
3. No blind flush. Part B never flushes on its own while step_busy is set, so a gated step waits behind at most one flush per stream core that started before the step did.

Worst case from step start, max (budget), for a step that spans both halves: kind fixed 10 + R_step hop on CCD 0 30 + one in-progress flush 32 + drain of the src lane 3 (pre-encoded in pass 1, normally drained in part B) + own flush 32 = 107 on each stream core, in parallel; then the TX core sees both release lines, 30 for the first and 5 more for the second, whose load it issues at the same time = 142 ns ≤ 160 ns. A step that touches one half is ≤ 137 ns. REP_FRAME_GATE_NS = 60 is therefore a maximum past the 100 ns step budget; a burst coalesces into the next capped frame and adds no second wait. With 01 §14.15's 15 ns step the gate still opens ≤ 142 ns after the step starts, so the cross fill at index 9 is 320 + 45 + 50 + 10 + 100 + 142 + 40 + 240 = 947 ns, not 01 §14.2's 880 (Requests to 01). The BBO core's deadline test (01 §14.1) uses these terms, so a step that would still miss the deadline is shed before it fills (StepKind::Shed).

4.8 Heartbeats, acks and self-fence#

• Heartbeat. When a stream core has sent no frame for REP_HB_US = 50 µs, it sends a HEARTBEAT on its substream on both paths. The highest frame_seq of each substream is therefore never older than 50 µs at any live replica, and a lost final frame is detected (§4.9).
• Ack. Every voter and the standby send RepAck { epoch, tenure, contiguous_frame_seq: [u64; 2], path_mask } (one value per substream) every REP_ACK_US = 50 µs, unicast to the REP_UDP_CTRL port of each stream core, on both paths. A box sends an ack only if it received a frame of the tenure in the last REP_ACK_LIVE_US = 100 µs, so a box that can send but not receive stops acking. Each stream core polls the RX rings of its own replication VIs in its background slice and keeps its own last_ack_ns[v] per voter.
• Self-fence (INV-REP-6, 01 §14.16). The sequencer fences itself unless it holds fresh acks from REP_FENCE_ACKS = n − Q + 1 + f = 3 distinct voters (3 in BASE and in CHAIN). If a stream core finds fewer than 3 voters with now − last_ack_ns[v] ≤ REP_SELF_FENCE_US = 250 µs, it sets seq_fenced. Both stream cores read seq_fenced before every flush, so both stop advancing released[·] and LaneCut and stop all frames, heartbeats included, within one flush. No response passes the gate after that. The silence makes the voters start a failover (§9.1). It never resumes in the same tenure. Acks from the co-located R1 alone never keep it alive: R1 is one voter of the three.
• Why 3. A deposition needs Q Silence messages from voters (§9.2), at most f = 1 of them Byzantine. With ≥ 3 fresh voters, ≥ 2 honest voters receive frames and stay silent, so at most n − 2 can send Silence: 2 < 3 (BASE), 3 < 4 (CHAIN). REP_FENCE_ACKS = n − Q + 1 + f = 3. The old rule (the newest ack from any replica) is unsafe once the sequencer does not vote: an F1 cut off from F2–F4 would keep the co-located R1 acking while R2–R4 depose.
• A sequencer cut off from enough voters stops releasing within REP_ACK_LIVE_US + one one-way delay + 250 µs (≤ 1.15 ms, max budget), then falls silent and is deposed. A stalled sequencer (an SMI, a page fault, a hung core) finds the stale acks at its next flush and fences before it sends anything more.

4.9 Loss detection at the replica#

Replica core 1 runs one receive loop over both replication ports. Each substream has its own next_expected and holes; the loop below runs per substream (f.stream):

fn on_rx(f: Frame) {
    if f.epoch16 != cur.epoch16 || f.tenure != cur.tenure || f.sender != cur.seq { drop_count(); return }
    if f.frame_seq < next_expected || have(f.frame_seq) { dup_count(f.path()); return }   // other path's copy
    store(f);                                          // frame store, §4.10
    while have(next_expected) { deliver(next_expected); next_expected += 1; }
    if f.frame_seq > next_expected { holes.open(next_expected .. f.frame_seq, now) }
}
fn tick(now) {
    for hole in holes.older_than(now, REP_HOLE_WAIT_US) { nack_seq(hole) }           // 5 µs
    for hole in holes.nacked_older_than(now, REP_NACK_RETRY_US) { nack_peer(hole); nack_seq(hole) } // 50 µs
    if now − last_rx_any > FAILOVER_SILENCE_MS { failover::suspect(now) }            // §9.1
}

REP_HOLE_WAIT_US = 5 µs covers the skew between the two paths (budget: path skew ≤ 2 µs, B-REP-3). A replica counts per-path losses. If a path loses more than REP_PATH_LOSS_ALARM = 1e-6 of frames over 10 s, or is silent for REP_PATH_DOWN_MS = 100 ms while the other path is live, the replica raises the alarm REP_A_PATH. Trading continues on one path (§14).

4.10 Repair#

Sequencer retransmit (first choice). Nack { epoch, tenure, stream, from_seq, to_seq } goes unicast to that substream's stream core at REP_UDP_CTRL. It looks up each frame in its frame ring. Each ring is REP_RTX_RING_BYTES = 64 MiB, about 40 ms of its substream at peak. The core re-posts the frame by DMA, unicast to the requesting replica, on the path the NACK came from, with flags.retransmit set. It does not copy the frame and does not use CTPIO, so a retransmit costs ~15 ns of the stream core per frame. Each stream core serves at most REP_RTX_MAX_FPS = 200k frames/s. More than that is dropped and counted, and the replica falls back to its peers. A frame that has left the ring gets NackTooOld { oldest_seq }.

Peer repair (fallback, and the only source for older frames). RepairReq { epoch, tenure, stream, from_seq, to_seq } goes to one peer (a voter or the standby), chosen in round-robin order by index. Peers serve from their in-memory frame store (REP_PEER_RING_MS = 2000 ms, ~6 GB at peak) and then from their NVMe order journal (§8). A peer-repaired frame is stored with flags.peer_repaired set. Two rules follow from that flag:

• A peer-repaired frame is trusted only once the order_root of its block checks (§6.4).
• It does not count as a direct holding in a failover TailReport (§9.3).

A peer serves at most REP_PEER_RTX_MAX_FPS = 1M frames/s. Peer repair runs on replica core 6.

Repair time (budget): hole wait 5 µs + NACK one-way ~5–250 µs (metro) + ring lookup 1 µs + return. That is ≤ 100 µs within one facility and ≤ 600 µs across the metro, max under a timely network. The finality budget charges one cross-metro repair (§6.6).

4.11 Lane responses are gated too#

Draft 0.2 gates every response (01 §14.5). A lane sends an Ack, CancelAck, Reject, BUSY, Out or settled maker Fill of op q only when LaneCut[l].cut ≥ q (INV-REP-11). The lane reads LaneCut anyway to settle (05 §6.2; ~20 ns per poll, inside the 170 ns cut term of 01 §14.17 item 1), so the gate adds no cross-CCD miss. Its only cost is that the lane's stream core writes LaneCut after the sfence of its frame and not right after it reads the step: REP_ACK_GATE_NS = 10 ns (max, budget, 01 §14.2). The gate is normally already open, for two reasons. Lanes pre-encode their PacketRecords in pass 1, so the record of q is in R_pkt[l] before the cut core can publish q. And part B of the stream core's loop frames those records ahead of any step, so the step that logs the cut finds them drained and its frame holds only the step.

A cut that covers q is logged by the next step, an Idle step if nothing else runs (05 §8.1), and that frame triggers a flush (§4.5 rule 1).

No exposure remains. Acks, rejects and BUSY answers are in the log before the client sees them, so §9.6 applies to them exactly as to fills: an acked order is never lost outside the cases that also revert a fill. 04 needs no ack-revert note.

5. The data tier (vyrn-data)#

5.1 Placement and sizing#

The decision is one vyrn-data box per facility: n_data = 4 in BASE and 5 in CHAIN, where each validator operator runs its own. The data tier is not a consensus participant. Its failures affect liveness only (§5.7). Data boxes are separate from replica boxes, so a replica crash (which uses up f) does not also cost a data home.

Each session has one data home, assigned at session open (04 owns the message, 10 owns the policy, and the default is the client's nearest facility, balanced by expected rate). The client sends copy 2 of every request packet to its data home's REP_UDP_DATA port.

A data home is never in the active sequencer's facility. If F1 held both the sequencer and a session's data home, losing F1 would lose both copies of every packet the data box had not yet forwarded (≤ REP_DATA_BATCH_US + one hop). Those packets become unavailable, and their fills revert (§9.6 d): ~0.1 F1 losses/yr × 5 µs × 20M fills/s ≈ 10 fills/yr. So the default policy homes each session in the nearest facility other than the sequencer's, and the F1 data box only serves as a backup home. The rule holds in both modes (01 §14.16): in CHAIN each operator's primary data box is in the operator's F2–F5 facility, never in F1 (09 §4.8). The cost is one more metro hop on the data path than on the order path, ≤ 0.71 ms of data lag, which §6.6 and 09 §4.8 budget.

Egress means the box sends its share of packets to every voter and to the warm standby. The active sequencer is not a destination: it has its own copy. When the standby is promoted, it leaves the destination set, and the next standby joins it once it starts warming (§9.5).

5.2 Ingest pipeline#

ASCII original

 RX queue (RSS by session) ─► parse header ─► MAC check ─► dedupe ─► full BLAKE3 ─► store ─► fan-out
   ~12 cores at peak          16 B            K_seq (BASE)  (session,   32 B kept    RAM ring   data frames
                                              K_com (CHAIN) pkt_seq)    pkt_hash=8   + index    to replicas

1. Parse. A packet is valid if count ∈ 1..=10, len = 32 + 16·count, reserved = 0, and book is a book this home serves. Anything else is dropped and counted.
2. MAC. In BASE, the data box holds K_seq (the operator holds all keys) and checks tag_seq. In CHAIN, it holds the operator's K_com and checks tag_com (09 owns key schedules). A failed MAC is dropped and reported to the sequencer's control core as DataBadMac{session, pkt_seq}. The data tier never closes a session. 01 §12.5 rules, and the sequencer applies them.
3. Dedupe by (session, pkt_seq), with hi (the highest stored pkt_seq) and a 256-bit window below it per session. A new pkt_seq is stored. A repeat with an equal full hash is dropped (flags.resend packets land here). A repeat with a different full hash is client equivocation: the box stores both, emits CopyConflict{session, pkt_seq, h1, h2} with both packets as evidence, and keeps forwarding the first. A pkt_seq below the window is late.
4. Hash. The full BLAKE3-256 is taken over bytes 0..16+16n with the tags excluded (01 §3). pkt_hash is its first 8 bytes.
5. Store. The bytes go into a RAM ring, indexed by a fixed open-addressed table keyed by pkt_hash that holds (session, pkt_seq, ring offset). 8-byte collisions do occur at 100M/s (~3e-4 per 10 s window), so a lookup always matches (session, pkt_seq) too.
6. Fan-out. The packet is appended to one data frame per destination replica (§5.3). A frame is sent when it reaches 9000 B or is REP_DATA_BATCH_US = 5 µs old.

5.3 Data frame#

off size field
  0   2  magic 0x4450 ("PD")      2 1 version 1     3 1 kind (1 DATA, 2 DATA_HB, 3 FETCH_RESP)
  4   2  book                     6 2 epoch16       8 8 data_seq (per data box, per destination)
 16   1  src data box index      17 1 n_pkts       18 2 body_len     20 4 reserved
 24   …  n_pkts × { pkt_len u16, pkt bytes (pkt_len), pad to 2 } 

data_seq lets a replica detect loss from one data box. A gap is fetched by DataFetch{from, to} from that box's RAM ring. A data box sends DATA_HB every 100 µs when idle. A replica that hears nothing from a data box for REP_DATA_SILENCE_MS = 1 ms marks it down (§5.7).

5.4 The replica's data store#

A replica keeps the same table as §5.2 step 5, filled from data frames. It also computes the full hash of every received packet itself, on replica cores 50, 52–55 and 63 (§7.2). It does not trust a data box's hash. If it did, a faulty data box could feed the same wrong bytes to every replica of both clients, and the state roots would agree on a wrong state.

Join. For each decoded PacketRecord r, the replica looks up r.pkt_hash and matches (r.session, r.pkt_seq). The record is resolved when a payload with that key is present, its 8-byte hash equals r.pkt_hash, and its number of numbered messages (05 §3.1) equals r.count. Advance records have no payload and are never joined. Until the HashRun arrives, the record is matched on (session, pkt_seq) only and stays unhashed.

5.5 Fetching a missing packet#

A record that stays unresolved for REP_DATA_WAIT_US = 300 µs after its ORDER frame triggers a fetch, with these sources in order:

1. the session's data home: DataFetch{session, pkt_seq}
2. one peer replica in round-robin order: PeerDataFetch
3. the sequencer's control core (core 51): SeqDataReq, served from lane packet retention (REP_SEQ_PKT_RETAIN_MS = 50 ms, owned by 05) at ≤ REP_SEQ_DATA_MAX_PPS = 50k pkts/s

Every fetched packet is checked like a pushed one. Source 3 is the only source of the bytes the sequencer actually matched, so it settles every conflict (§5.6).

5.6 Mismatch between the client's two copies#

The two copies of one packet can differ in three ways.

The vote for a block waits until each mismatch is settled (§6.4). This is the only data-tier event that can make the vote wait on the sequencer. It happens once per misbehaving session, because the session is closed. If the sequencer is dead and the canonical copy cannot be fetched, the packet is unavailable. The seal proposal lists it, and replay excludes it (§9.4). The exclusion semantics are the same as CHAIN's excl rule (09 owns them).

5.7 Data box failure#

1. Replicas mark data box d down after REP_DATA_SILENCE_MS = 1 ms of silence, and gossip DataDown{d}. Q such messages make it effective.
2. Each session homed on d is reassigned to the backup data home d' = next live box in facility order. This is deterministic, so no message is needed to choose it.
3. Every live data box sends DataRedirect{d → d', resend_from} to the affected sessions' bound addresses. 04 owns the record. The data boxes know those addresses from the copies they have received. resend_from is per session: the lowest pkt_seq that no replica has resolved. The replicas compute it and send it to the data boxes.
4. Clients resend copy 2 from resend_from to d'. Clients keep sent request packets for REP_CLIENT_RETAIN_MS = 100 ms (04 must require this). The resends carry flags.resend, and dedupe drops anything already stored.

Effect: fills and acks are unaffected. Blocks that reference the affected sessions cannot be voted until their data arrives. Finality stalls for those blocks for ≤ 1 + 0.5 + 1 + 0.5 = 3 ms (budget) and then catches up (§6.7).

6. Blocks and consensus#

6.1 Block cut and roots#

• Cut. Core 4 asks for block h's close when its PTP-disciplined clock passes t_h = epoch_start + h_local × BLOCK_NS; the BBO core places the mark entry after its current step, and each stream core appends its BlockMark(h) there (§3.9). The gate does not hold a mark back. Block h's ORDER bytes on substream H are all its ORDER section bytes after BlockMark(h−1), up to and including BlockMark(h), in frame_seq order. The first block of a tenure starts after TenureStart.
• order_root(h) = BLAKE3(order_root_A(h) ‖ order_root_B(h)), each the BLAKE3 of that substream's ORDER section bytes of h, with the frame headers, HASH frames and HEADER frames excluded. A replica computes it from what it decoded. Byte identity (INV-REP-3) makes it equal at every replica. With one stream core (CONSUMER, AWS), order_root_B is BLAKE3 of nothing.
• data_root(h) = BLAKE3(data_root_A(h) ‖ data_root_B(h)), each the BLAKE3 of the concatenated full 32-byte hashes of the packets of every PacketRecord of that substream in block h, in O-order. O-order is the order of records in a substream's ORDER bytes.
• first_step, last_step are the first and last StepIds in the block. An empty block has last_step = first_step − 1.
• Blocks with no traffic. Core 4 still asks for their close, so heights advance at 1 kHz and BlockMark doubles as a liveness beat. An empty block costs one ~200 B HEADER frame.

6.2 Signing seq_sig on core 4#

Core 4 signs, so that one core owns the key and the order of headers. Ed25519 signing (RFC 8032, deterministic nonce) costs ~10 µs of CPU (budget at 5 GHz, B-REP-5). Run in one piece, it would stall the gate for 10 µs. The decision is to make the signer a resumable state machine:

enum Signer { Idle, HashNonce(Sha512State), ScalarMul(CombState, u8 /*window*/), HashChallenge(Sha512State), Done(Sig) }
// background_slice() advances it by one step of ≤ REP_SIGN_SLICE_NS = 50 ns:
// one SHA-512 block, or one fixed-base comb addition (~40 ns), then returns to the main loop.

About 200 slices finish one signature. Core 4 spends ≥ 50% of its time outside the hot loop (1 − REP_STREAM_MAX_LOAD), so a signature completes ≤ 35 µs after the roots arrive. The roots come ≤ 10 µs after BlockMark, because core 50 lags by one frame batch on each substream. The HEADER frame therefore leaves ≤ REP_SEAL_US = 100 µs (max, budget) after the block closes. The signing key stays in core 4's memory only. It is loaded from the secret store at tenure start and is never written to disk (06 owns the secret store).

6.3 Simplex mapping#

Payload. A Simplex payload is a digest of a REP_Proposal:

REP_Proposal (encoded LE, 1 + 4 + 8 + 8 + 32 + 32 = 85 bytes)
  kind         u8      0 = Headers, 1 = Seal (§9.4)
  epoch        u32
  first_height u64
  last_height  u64     last_height − first_height < REP_MAX_HDRS_PER_PROPOSAL (16)
  last_hash    Hash    header_hash(last_height); prev links commit to the whole range
  excl_root    Hash    BASE: [0; 32]. CHAIN: BLAKE3 over sorted (session, pkt_seq) excluded in the range (09)
digest = BLAKE3("vyrn      v1 proposal" ‖ REP_Proposal bytes)

The parent rule is that first_height equals the parent proposal's last_height + 1. Genesis of an epoch is the HandoffRecord's last_height (08).

Leader propose. The leader takes every contiguous header after the parent's last_height that it holds with a valid seq_sig, up to 16 of them. It waits for at least one, and for its own verify of that range to pass. It then returns the digest. Usually the range has 1 header, because views and blocks both run at ~1 kHz. After a nullified view the range catches up.

6.4 The vote rule (when a replica votes)#

INV-REP-4. A voter returns verify(ctx, digest) = true only when all of these hold for the proposal P behind digest:

1. Headers. It holds headers P.first_height ..= P.last_height. Each has a valid seq_sig under the current tenure's seq_key, the prev links chain, header_hash(last) = P.last_hash, and the steps are contiguous: first_step(h+1) = last_step(h) + 1.
2. Order bytes. It holds every ORDER frame through BlockMark(P.last_height) with no gap. Its decoder accepted them with no error (§3.12), and its order_root(h) equals each header's.
3. Payloads. Every PacketRecord in the range is resolved (§5.4). Every mismatch is settled (§5.6). Its own data_root(h) equals each header's.
4. CHAIN tags. Every packet in the range has a tag_com verdict, and the set that fails gives excl_root = P.excl_root. 09 owns the tag rules, the key schedule and the exclusion semantics. 07 owns this timing: the check is done by the replay lane cores as data arrives (§7.2), before any replay of the block.
5. Fences. No header in the range is above an INV-FENCE boundary (08) or after a seal of the tenure (§9.4). No REP_A_DIVERGE is active, and the state-root gate holds (§7.4).

verify stays pending while items 1–4 are incomplete. It returns false at once on a contradiction, such as a root mismatch after repair or a bad signature. It also returns false after REP_VERIFY_WAIT_MS = 4 ms pending. A false result leads to nullify. A vote never waits for replay or for the state root of the block it votes on. Replay only gates the state-root certificates, and those trail by up to 5000 blocks.

Cost per block at peak, on top of continuous work: finishing two BLAKE3 hashes and up to 16 Ed25519 verifies (~40 µs each budget, batched). That is ≤ 0.1 ms (max, budget).

6.5 Notarization, finalization, certificates#

Simplex's own messages carry notarize(c, v), nullify(v) and finalize(c, v). A notarization is Q notarize votes, which lets the next view start. A finalization is Q finalize votes (INV-QUORUM). It makes every header in the proposal final, and every ancestor with it.

VYRN finality certificate (exported by vyrn-consensus, consumed by 08, 09, 10 and clients):

REP_FinalityCert
  off size
    0   4  epoch       u32
    4   8  view        u64
   12   8  parent_view u64
   20  85  proposal    REP_Proposal
  105   1  count       u8          ≥ Q_BASE or Q_CHAIN
  106  65·count entries  { index u8, sig [u8; 64] }   ascending index, distinct (01 §4)

Each sig is the participant's Ed25519 finalize signature as produced by simplex::scheme::ed25519 over Proposal{ round: (epoch, view), parent: parent_view, payload: digest } under namespace b"vyrn-v1". The exact signed bytes are those of the pinned commonware-consensus version. vyrn-consensus exports verify_finality_cert(&cert, &membership) -> Result<(), CertErr>. No other crate reimplements the check. A Vx verifier links that function through FFI. A certificate is about 0.37 KB (Q = 4).

A fill is final when the block holding its step is final. The finality event is the moment the local voter has Q finalize votes. 02 owns where that is measured. Finality is a different fact from durability, and §6.9 keeps them apart.

6.6 Finality latency#

The fill lands uniformly inside a block. The hop count is four: the header goes from the sequencer to every participant, the leader proposes, everyone votes notarize, and everyone votes finalize. Simplex's "3 network hops to finality" counts from the proposal. The header hop comes before it.

FINALITY_MAX_MS = 10 is a maximum for every accepted order while the metro network meets its delay bound (01 §14.0 fact 3). The bound is REP_HOP_MAX_MS = 0.8 ms one-way. A box whose links break it counts as the one fault f.

A fill at the end of its block sees ~3.1 ms typical, which is DESIGN §5's "~3 ms". In a single facility (hops ~10 µs) the floor is ~1.1 ms. Under f = 1 two failed views in a row cannot happen: the elector is round-robin over the voters and the sequencer is not one of them, so a faulty voter leads one view in n. A Byzantine sequencer that stalls is deposed, and the orders it delayed are final ≤ FAILOVER_TARGET_MS + FINALITY_MAX_MS (01 §14.0 fact 3). Beyond the fault model, the finality gate halts the book (§6.8). Benchmark B-REP-1 measures every fill's finality with the voters in 4 real facilities.

6.7 Pipelining and backlog#

• Views advance at notarization (~2 hops ≈ 1 ms), and blocks arrive at 1 kHz. A view whose leader has no new header waits up to t_l, then nullifies. Normally the header arrives first.
• When verification is late (a data box failure, a repair), headers queue. The next proposal covers up to 16 of them. A backlog of B blocks clears in about B/16 views. A 3 ms stall clears within ~2 views.
• A voter that has notarized view v may vote in v+1 before v is final. Simplex finalizes v and v+1 independently, so heights become final in order: a final proposal makes its ancestors final.

6.8 The finality gate (both modes)#

FIN_GATE_BLOCKS = 8 (01 §14.9) applies in BASE and in CHAIN. INV-REP-10: the sequencer never releases a response while more than 8 closed blocks are not final at the sequencer: block h + 9 opens only when block h is final. 09 §4.6 (INV-CHN-TXGATE) uses the same count.

• Voters → sequencer. Every voter sends its finalize vote to the sequencer's and the standby's REP_UDP_CTRL as well as to the other voters (09 §4.6), and later the REP_FinalityCert. The finality listener, a thread of control on core 51 (01 §10), batch-verifies the votes (≤ 4 × ~40 µs per block at ≤ 1 kHz, ≤ 16% of core 51) and stores final_height once it holds Q valid votes for a proposal. So the sequencer learns finality at hop 4, like the voters, not one hop later. No single voter can forge it.
• Core 4 checks at every flush: lag = closed_height − final_height. If lag > FIN_GATE_BLOCKS, it puts HaltReq on the BBO control ring. The BBO core emits Control(Halt) as its next step (≤ 1 µs). From that step's cut on, every lane answers new ops BOOK_HALTED. Those rejects are logged and gated like any response, so replay reproduces them (INV-DET). Frames, blocks and heartbeats keep flowing on both substreams, so finality can catch up.
• Resume. Control(Resume) at lag ≤ REP_GATE_RESUME_BLOCKS = 4; after > 1000 ms, via a reopening auction (08).
• It never trips within f = 1. Block h is final at every voter and at the sequencer ≤ 7.65 ms after it closes (§6.6, signature checks included). h + 8 closes 8 ms after h closes, so ≤ 8 closed blocks are open (0.35 ms spare). In CHAIN the bound is 7.11 ms after close (09 §4.8).
• What it bounds. Outside the fault model (no Q, a slow metro), at most ~9 ms of fills are released unfinal before the halt, none reverted; they finalize when Q returns. Draft 0.1's 100 ms halt (a 10 ms escape) is removed.

6.9 Durability and finality: two separate attestations#

01 §14.13 item 3 fixes the order: volatile execution (the 1 µs Fill or Ack) → durability → finality → settlement (CHAIN, 09). 07 defines the middle two. They are never merged into one "confirmed" flag in any API, the drop copy or market data.

• Clients. DUR_PATHS is the response itself. DUR_QUORUM and FINAL arrive as two separate unsolicited watermark records from control on core 51 (Durable per ≤ 1 ms, Final per finalization, each with StepId and height; codes requested from 01).
• Drop copy and market data (10). Every fill and ack carries its StepId and height. Two separate streams follow it: durable_step (from REP_DurableMark) and final_height with the certificate. A fill record shows three separate fields: executed, durable, final. A replay of the drop copy never infers one from another.
• Times. DUR_PATHS with the response (≤ 1 µs); DUR_QUORUM ~0.6 ms; FINAL ≤ 10 ms (§6.6).

7. Determinism and replay#

7.1 Replay mode#

vyrn-engine in replay mode is the live engine with two substitutions (01 §11.6). Decoded PacketRecords and their resolved payloads replace the NIC, and StepRecord.cut replaces the cut core. 07 delivers the following to the engine:

The schedule is owned by 05. It must respect two dependencies:

• Lane l processes record p only after the replay BBO core has emitted every fill event of steps ≤ p.fills_applied.
• The BBO core executes step s only after every lane l has applied its ops ≤ cut_s[l].

The live engine classifies an op against a snapshot version. Replay does not need that version: the snapshot is only a routing hint (INV-ENG-1, 01 §14.11), so draft 0.1's SNAP bit is gone.

Replay outputs are every response record (with resp_seq), every fill event, the per-lane and BBO state, and the root at root heights (§7.4). INV-DET makes them identical at every replica of either client. vyrn-sim checks this on every CI run with the Rust engine, vyrn-vx-client and vyrn-vx-oracle (benchmark B4).

7.2 Replica core map (REF box, Rust client)#

Ports. Ports 0–1 are data ports on the facility fabric: data frames (78% each at peak), commonware-p2p, the archive feed and recovery-service resends. Ports 2–3 are replication paths A and B (27% each, both substreams): ORDER, HASH and HEADER frames, acks, NACKs, peer repair and catch-up.

A Vx replica runs the same Rust shell (network, data, consensus, storage, §11). Its replay engine is vyrn-vx-client on cores 2, 8–49 and 56–62. The warm standby runs this map without core 4's voting and without core 7's journal; core 4 keeps its TailReport and certificate listener.

7.3 Replay speed (budget)#

Replay must run faster than live at every load, or a replica that falls behind never catches up.

REP_REPLAY_SPEEDUP_PEAK = 1.5× (budget, the BBO core bounds it, B-REP-7). At 10% average load, the BBO core and lanes run at ~6% load, and replay is bounded by the network and NVMe: REP_REPLAY_SPEEDUP_AVG = 10×. A replica L seconds behind closes the gap at speedup − 1: in 2L seconds at peak and in L/9 at average load. vyrn-vx-client must reach ≥ 1.2× at peak (B-REP-VX). This is a release gate, not a target: a Vx build that misses it does not ship (11), because a slower Vx client would stop root certificates under sustained 1B msgs/s, and 1B msgs/s is a hard requirement (01 §14.0) with no limit on its duration.

Steady-state replay lag (receipt to applied), budget: ≤ 2 ms. Nothing waits on it. It sets the age of state roots and of market data (10).

7.4 State roots and the gate#

• Root heights. A root height is a height h with (h + 1) % REP_ROOT_BLOCKS == 0, where REP_ROOT_BLOCKS = 1000 (≈ 1 s). The rule uses heights, so it is deterministic. The state root R(h) is the commitment over state after block h.

• Function. CHAIN uses 09's lattice commitment over the interval's net diff. BASE calls the same function. 09 owns it, and 07 schedules it in both modes. One function means one cross-client test vector set.

• Vote. Each participant signs digest = BLAKE3("vyrn v1 root" ‖ epoch ‖ h ‖ R(h)) and submits it to a commonware-consensus::aggregation engine. Item index = (h + 1) / 1000 − 1, Ed25519 scheme, namespace b"vyrn-v1-root". Q signatures form REP_RootCert, which has the layout of REP_FinalityCert with proposal replaced by (h u64, R(h) [32]).

• Gate (INV-REP-7). verify(P) requires a REP_RootCert for every root height ≤ P.last_height − REP_ROOT_GATE_BLOCKS, where REP_ROOT_GATE_BLOCKS = 5000. Normal operation never waits: roots certify ~3–10 ms after their block is replayed. If replay stalls at so many participants that no root reaches Q, finality stops 5000 blocks later. This is the backstop against silent divergence.

• Divergence (INV-REP-9). REP_A_DIVERGE fires at a participant in either of two cases:

  • Two different digests for one root index each have ≥ f + 1 = 2 valid signatures.
  • No digest has Q signatures REP_ROOT_TIMEOUT_MS = 2000 ms after the root height was finalized.

  On REP_A_DIVERGE, the participant votes for nothing more (verify returns false, and it nullifies every view), and it sends Diverge{index, digests, sigs} to the sequencer's control core. The sequencer halts the book with a Control(Halt) step (§6.8) within 1 ms of receiving it. §11 gives the procedure.

One Byzantine participant cannot fire the first case, because its wrong root has 1 < f + 1 signatures. Its root is recorded as a fault (§10.2). It also cannot block a certificate: the remaining n − 1 ≥ Q participants certify without it.

7.5 Snapshot format#

Snapshots are taken at root heights where (h + 1) % 60,000 == idx × (60,000 / n), with REP_SNAP_INTERVAL_S = 60 and idx the participant index (BASE offsets of 15 s, CHAIN offsets of 12 s; both are multiples of 1000 blocks). Each participant snapshots every 60 s, and at most one is paused at a time.

REP_Snapshot file                         (little-endian, fixed-width, no maps dumped)
  off size field
    0   8  magic       "PBSNAP01"
    8   4  version     1
   12   2  book        14 1 n_lanes      15 1 client (0 Rust, 1 Vx; informative only)
   16   4  epoch       20 4 reserved
   24   8  height      state after this block
   32   8  last_step
   40  32  state_root  must equal the certified R(height)
   72  32  header_hash header_hash(height)
  104   4  n_sections  108 4 reserved   112 8 table_off   120 8 file_len
  table:   n_sections × { id u16, encoding u16, reserved u32, off u64, len u64, blake3 [32] }
  file hash = BLAKE3(header ‖ table)       (the table already commits to every section)

Taking a snapshot. At the boundary, replay pauses and the live state is copied into a staging arena (budget ≤ 4 GB at ~20 GB/s across 49 lane threads ≈ 200 ms). Replay then resumes and catches up at ≥ 1.5×. Core 7 serializes from the staging arena at ~2 GB/s and writes it to the NVMe snapshot area. Votes do not pause, because they never wait for replay. Response rings are not in the snapshot. A standby loaded from a snapshot can resend only responses produced after it, so it may be promoted only after REP_PROMOTE_HISTORY_S = 10 s of replay since loading, which fills its 8 s hot response ring (01 §14.6). Older resends go to the recovery service (§8.5).

Loading. The loader checks every section hash and the file hash. It then recomputes the state commitment from sections 1–4 and 6 and compares it with the certified R(height). A mismatch rejects the snapshot, and the loader tries the next source. A snapshot is therefore trusted through the certificate, not through its source.

7.6 Catch-up from snapshot plus log#

ASCII original

 t0 ── subscribe to live frames + data; verify and vote at once (votes need no history) ──────────►
  │
  ├─ fetch the newest snapshot ≤ the newest RootCert (local NVMe, else peers via REPAIR, else archive)
  ├─ verify it (§7.5)
  ├─ fetch finalized blocks snapshot.height+1 .. live, striped by height across all peers
  │     (block record + cert, §8.2), verify certs and roots, write them to the local journal
  ├─ replay at max speed; buffer live input; live input already counts toward votes
  └─ replay lag ≤ REP_LIVE_LAG_MS (10 ms): the replica joins root votes and is "caught up"

Time to catch up (budget). The fetch runs on the two replication ports, which have ~150 Gbps free at peak and ~190 Gbps at average load:

8. Storage#

8.1 Hardware and layout (replica, REF)#

The decision is 12 × 61.44 TB Gen5 NVMe (U.2 or E3.S, TLC), ~737 TB, with no RAID (06 §2.2; 01 §14.7's 30.72 TB cannot hold §14.16's 6 h at peak, Requests to 01). Redundancy comes from the n voters and the archive nodes.

Writes go to drive h mod 12 for block h, at about 2.2 GB/s per drive. Reads at peak on a voter are at most ~12.5 GB/s, which is one peer catch-up plus the archive pull, about 1.05 GB/s per drive. R3 is last in every peer's catch-up order and serves ≤ 0.1 GB/s of resend reads, so its mix (27.4 W + 0.1 R) is lighter than a voter's (26 W + 12.5 R). Gen5 TLC sustains both mixes at ≤ 50% of its vendor sequential rates (vendor, 61.44 TB class: ~5 GB/s write, ~12 GB/s read; 2.2 W is 44%). That is benchmark B-REP-9. PCIe lanes: 4 × X2541 (x16) = 64 and 12 NVMe (x4) = 48, total 112 of the EPYC 9575F's 128.

8.2 Block journal#

A block is written once, when its verify passes. It is complete then. Packets the sequencer never sequenced (bad MAC, rate limit, epoch mismatch) and duplicates are never written. They age out of the RAM ring. That is the compaction: there is no rewrite pass.

ASCII original

REP_BlockRecord   (4 KiB aligned, on drive h mod 12)
  off size field
    0   8  magic "PBBLK001"         8 8 height        16 4 epoch     20 1 tenure   21 1 kind (0 normal, 1 seal)
   22   2  reserved                24 190 BlockHeader (§3.10)
  214   4  order_len              218 4 hash_len       222 4 data_len 226 4 n_unavail  230 26 reserved
  256   …  order bytes │ pkt_hash list (8 B × records) │ data bytes (O-order; self-delimiting by count)
           │ unavailable (session, pkt_seq) list │ pad to 4 KiB

• Cert journal. Each finality cert, root cert and seal ext is appended to a small append-only journal mirrored on drives 0 and 1, about 1 MB/s.
• Index. height → (drive, offset, len) and (session, pkt_seq) → height, the second at 1-second granularity. Both are built on commonware-storage journal and archive primitives.
• Durability point. A replica acks nothing to anyone based on NVMe. Durability before finality is Q replicas' RAM, and after finality it is Q replicas' NVMe within ≤ 5 ms (one io_uring batch). A replica crash loses at most its own RAM tail, which its peers hold.

8.3 Retention per mode#

01 §14.16 fixes the on-box window: at least 6 h at sustained peak and 12 h at ≤ 20% average load, with the archive node streaming continuously. Retention never blocks replication.

At 10% average load the log grows by 22.3 × 0.1 × 86,400 ≈ 193 TB/day of raw blocks (DESIGN §7's ~230 TB/day uses the 26 GB/s budget). Local snapshots: the last 10, plus hourly ones for 12 h. The on-box log is a ring: the oldest block goes first. A replica deletes blocks the archive node has acknowledged before others; when the ring is full it deletes the oldest block even if the archive has not taken it, raises REP_A_ARCHIVE, and keeps receiving, storing and voting. So a slow archive never stops replication or votes. REP_A_ARCHIVE also fires early, when the archive lag reaches 5 h. At 90% disk use the replica stops serving catch-up reads.

8.4 Archive node#

One archive node per facility (06 §2.2: a single-socket storage chassis, 24 × 122.88 TB QLC, ~2.9 PB) keeps 7 days at ≤ ~18% average load (1.35 PB raw at 10%). It streams continuously: it pulls each block from its voter's data ports as soon as the block is final, never in batches, at ≤ 25 Gbps at sustained peak (ports ≤ 90%) and ≤ 50 Gbps otherwise. At sustained peak the feed is below the ~180 Gbps ingest, so the archive falls behind and the ≥ 6 h on-box window covers the gap; a sustained peak longer than that window loses archive coverage of its oldest hours (Open question 10). Hourly objects (header + cert + zstd -3 --long of order bytes and data, plus the hour's last snapshot) go to the long-term erasure-coded store across ≥ 3 facilities (RS 6+3); compression ≥ 2.5× (B-REP-10). At ~8 cores of compression it is a separate box. It is also the WAN feeder (§12) and the catch-up source beyond the on-box window.

8.5 Recovery service (01 §14.6)#

The sequencer keeps RESP_HOT_RING_BYTES = 4 GiB per lane (~8 s at peak). The recovery service keeps the rest: the last RESP_RETAIN_PKTS = 65,536 response packets of every session. It is a role on one voter per home, R3 (Rust, F3). F3 holds neither the sequencer (F1) nor the standby (F2), so one facility loss never takes both the hot ring and the service.

• Source. R3's replay produces every response packet with its resp_seq (§7.1, INV-DET).
• Encoding. REP_RspEntry: echo_pkt delta, count, then each record with every field kept and delta-coded per session (tick, handle and client_tag against the session's previous values, qty as LEB128). Budget ~5.5 B per record, ~53 B per packet at the peak mix. No tag is stored. A resend is rebuilt from the entry alone, with no lookup of the request.
• Storage. Per session, a chain of 4 KiB pages covering its last 65,536 entries; freed pages return to a free list, and a slow session's open page is flushed every 10 s. Capacity: ≤ 200,704 sessions (49 × MAX_SESSIONS_PER_LANE) × 65,536 × 53 B ≈ 700 GB, so REP_RSP_STORE_BYTES = 1 TB (0.14% of R3's NVMe). Writes: ~110M packets/s × 53 B ≈ 5.8 GB/s at peak plus ≤ 0.1 GB/s of partial pages (§8.1), ~0.6 GB/s at average load. RAM: an open page and an index per session, ≈ 2 GB.
• Serving. A lane whose hot ring no longer holds a ResendReq range hands it to control on core 51, which relays it to R3 on REP_UDP_CTRL. R3 core 6 rebuilds the packets, tags them with K_out (SipHash, ~60 ns each), sets flags.resend and sends them from a data port. Clients accept them by session and MAC, not by source address (04). The cap is REP_RSP_MAX_PPS = 2M packets/s (≤ 4.5 Gbps out, ~0.1 GB/s of page reads). Reject RESEND_GONE is sent only when neither the hot ring nor R3 holds the packet.
• Failure. If R3 is down, R4 takes the role from its own replay at once; older resends get RESEND_GONE until R3 returns. That loses history, never correctness.

9. Sequencer failover#

9.1 Detection#

A voter suspects the sequencer when any one of these holds:

• No frame of the current tenure has arrived on either path for FAILOVER_SILENCE_MS = 2 ms. Heartbeats come every 50 µs, so that is 40 missed beats.
• It holds valid equivocation evidence (§10.1).
• A decoded block failed order_root after repair from the sequencer and from every peer, or REP_E_HASHLAG fired.

On suspicion, the voter freezes its tail. It stops accepting frames of the tenure into its tail state, and buffers any that still arrive. It then broadcasts a signed Silence{epoch, tenure, seq, reason, report}, where report is its TailReport (§9.3). If fewer than Q voters have sent Silence within REP_SUSPECT_ABORT_MS = 5 ms, it unfreezes, applies the buffered frames and repairs the rest. A single isolated voter therefore cannot depose a live sequencer. The standby sends a TailReport too, but its Silence never counts toward Q.

9.2 Deposition#

When a voter holds Silence messages from Q distinct voters, the tenure is deposed at that voter: BASE 3 of 4, CHAIN 4 of 5. INV-REP-5: a deposed voter never again accepts a frame, header or retransmission of (epoch, tenure) into its log. The next Simplex leader in plain round-robin order builds the seal proposal. The self-fence (§4.8) guarantees that the sequencer had stopped releasing before Q voters could freeze (fence ≤ 1.15 ms < FAILOVER_SILENCE_MS), so a voter outside the Q set (for example R1, co-located with the sequencer) may still hold the newest frames. The leader therefore waits for every voter's and the standby's report, but no longer than T* + REP_TAIL_WAIT_MS, where T* is the newest t_last_rx in the reports it holds and REP_TAIL_WAIT_MS = 4.05 ms (fence 1.15 + detection 2.0 + one hop 0.8 + 0.1). It starts fetching what it lacks at the Q-th report, in parallel. After a crash all reports arrive together.

9.3 Tail reports and the prefix rule#

Two substreams. Every frame quantity of this section is a pair, one value per substream: F_i, extra, direct, F*, and F* in REP_SealExt. Each pair is computed and checked on its own. A step is in the prefix only when its home record, and its cut-only record if OTHER is set, are both in it (§3.8).

TailReport (inside Silence)
  epoch u32, tenure u8, reporter u8,
  H_i      u64       highest height whose header i holds with a valid seq_sig, together with
                     every ORDER frame through BlockMark(H_i)
  F_i      u64       highest frame_seq such that i holds every frame 1..F_i of the tenure
  extra    ≤ 64 ranges [a, b] of frames held above F_i
  direct   bitmap over (H_i's mark, F_i] plus extra: 1 = received from a sequencer path
  t_last_rx u64      PTP time at which i received its newest frame of the tenure
  replay_lag_us u32, promotable u8   (§9.5)
  sig      Ed25519 over the above by reporter i (a voter, or the standby under its seq_key)

The leader computes the prefix from every report it holds when the wait of §9.2 ends (≥ Q):

1. Signed part. H* = max_i H_i. One reporter is enough, because seq_sig is verifiable and no voter can forge it. This is the "longest prefix with a verifiable seq_sig".
2. Unsigned tail. F* = the largest k such that every frame 1..k of the tenure is held by at least one reporter, counting F_i and extra. The leader fetches the frames it lacks from their holders. That takes one RTT, and the budget is ≤ 1 ms.
3. Consistency. Each voter compares every frame in (BlockMark(H*), F*] that it holds with the leader's bytes (INV-REP-3, flags excluded). On any difference it refuses to vote and broadcasts TailConflict{frame_seq, its bytes, its direct bit}. The next proposal uses F* := lowest conflicting frame_seq − 1. Only a sequencer that sent different bytes under one frame_seq can cause a conflict, so that is evidence of a faulty sequencer.
4. Data. Every PacketRecord in frames ≤ F* must be resolved at each voter (§5.4). Fetches start REP_DATA_WAIT_US after each record's frame, before the crash is even detected. If the leader still cannot resolve one from any source REP_SEAL_DATA_WAIT_MS = 2 ms after the record's frame arrived (so at the latest when it proposes), it lists the packet as unavailable. A voter accepts an unavailable entry only if it cannot resolve the packet either. If it can, it sends the bytes to the leader and refuses, and the next proposal includes the packet. A record with no HashRun in the prefix is unhashed. Its payload is accepted on (session, pkt_seq) alone (§9.6, case d).

Why this is safe. A failover happens only with Q Silence messages from voters. Under f = 1 there are two cases:

• The sequencer is the faulty box. Then every reporter is honest. Every frame an honest reporter holds came from the sequencer's multicast group (static ACL, §3.1) or was repaired from an honest peer, so the union of the reports contains only frames that the sequencer sent.
• The sequencer is correct and one voter is Byzantine. A deposition then needs Q − 1 honest voters that saw 2 ms of silence. The self-fence (§4.8) stopped every release before that, so every released frame is held by an honest voter that kept receiving, and the leader waits for its report (§9.2). A Byzantine voter that sends Silence alone achieves nothing.

Beyond f = 1 (a dead sequencer plus a lying replica), a forged tail frame would still need to pass the static ACL, and every frame at or below H* is covered by seq_sig. Property: a frame that any reporter holds is in the prefix, unless a lower frame is held by no reporter, or the sequencer equivocated.

9.4 Seal proposal and seal block#

The seal is one Simplex proposal of kind 1:

REP_Proposal { kind: 1, epoch, first_height: parent.last_height + 1, last_height: Hs = H* + 1,
               last_hash: header_hash(seal header), excl_root: over unavailable ∪ CHAIN-excluded }
REP_SealExt  { end_frame_seq: [F*_A, F*_B] [u64; 2], next_seq: u8, next_tenure: u8, n_unavail: u32,
               unavail: [(SessionId, u32)] }   // digest = BLAKE3(proposal ‖ seal ext)
seal header  { height: Hs, epoch, book, first_step: last_step(H*) + 1,
               last_step: last StepId in frames ≤ F*, prev: header_hash(H*),
               order_root: as §6.1, over each substream's ORDER section bytes after its
                           BlockMark(H*) through its F*,
               data_root: as §6.1, with 32 zero bytes for each unavailable packet,
               seq_sig: [0; 64] }

• A kind-1 range may hold up to REP_MAX_SEAL_RANGE = 64 heights: the signed headers (parent.last, H*] plus the seal header.
• The seal block's lane_bounds are the last lane_seq of each lane in frames ≤ F* of its substream. A cut-only record whose home record is not in the prefix is dropped, and so is every later record of either substream's StepRun. PacketRecords above the last kept step's cut are part of the prefix. The next tenure's first cut starts at these bounds, so INV-CUT-MONO holds across the boundary (05 applies those trailing ops as lane-local ops).
• Voters run §6.4 items 1, 3 and 4 on the range and §9.3 items 3 and 4 on the tail. Finalization of the seal proposal makes the seal block final. Its certificate stands in for seq_sig.
• After the seal is final, next_seq emits frame 1 of tenure next_tenure with TenureStart{ first_step: last_step(Hs) + 1, first_height: Hs + 1, prev_header_hash: header_hash(Hs), lane bases, prev_cut = lane_bounds }.

9.5 Next sequencer, promotion, client redirect#

Deterministic choice (01 §14.4). REP_Membership.seq_order lists only non-voting Rust vyrn-seq boxes: the epoch's primary, then the warm standby in F2, then the spares in the fallback order F3, F4, F1 (08 assigns the primary). No voter is ever in seq_order, so no failover changes the voting set or CLIENT_SPLIT. next_seq is the first entry after the deposed one that meets all of these:

• it has not been deposed in this epoch
• it sent a TailReport in this failover
• it reported promotable = 1: at least REP_PROMOTE_HISTORY_S = 10 s replayed since its last snapshot load, and replay_lag_us ≤ 2,000

So the warm standby is chosen first. A cold spare is promotable only after it has warmed (below). Voters recompute the choice and refuse a seal that names another box.

No promotable box. The seal still finalizes the prefix (every response sent is final), with REP_SealExt.next_seq = 0xFF. The book stays halted, and data boxes send SeqChange{new_seq: none}. No voter is promoted, so neither client reaches Q alone. This needs two sequencer faults inside one warm-up window, which is beyond f = 1.

Warming the next standby. When the standby is promoted or fails, the first free spare in fallback order boots a non-voting vyrn-replica beside vyrn-seq standby, joins the data-box destinations, catches up (§7.6, ≤ 3.5 min at peak) and is promotable 10 s later. Trading goes on; only the second-failure margin is missing (REP_A_STANDBY). A spare that replaced a voter is not free.

Promotion (on next_seq, budget 3 ms, B-REP-8).

1. During the seal it keeps replaying. It holds every frame ≤ F* by the time the seal is final (it fetches missing ones from the leader), so replay reaches the seal within ~1 ms.
2. The standby vyrn-seq process adopts the replay engine's arenas, which live in a shared 1 GB hugepage mapping. vyrn-replica stops writing to them first, so INV-WRITER holds across processes. vyrn-seq pins the sequencer core map (01 §10): cut cores 1 and 6, fill TX 3, 5 and 0, stream cores 4 and 7, order hasher 50, control and finality listener 51. The lane and BBO threads keep cores 8–49, 56–62 and 2. The replica threads on 0, 1 and 3–7 stop first, and the data RX threads on 50, 52–55 and 63 stop; core 50 becomes the hasher.
3. It loads its seq_key and the epoch's sequencer session keys. In BASE they are preloaded from the secret store. CHAIN keys are 09's question (see Interfaces).
4. It arms the NIC flow filters (UDP port = lane) on its 4 client ports, which carried data frames until now, and emits TenureStart.
5. Data boxes stop sending data frames to it.

Client redirect. Every data box sends SeqChange{epoch, tenure, new_seq, address} to every session it serves, using the bound addresses it already knows. The new sequencer also sends it on first contact. 04 owns the record. Clients also hold seq_order and addresses from the epoch notice, so a client can switch without waiting.

Resend dedupe by (session, pkt_seq). The new sequencer knows last(session), the highest pkt_seq in the sealed prefix, from replay. For a packet with pkt_seq:

Clients resend every packet that has no response, from their retained buffer.

9.6 The fill-revert condition#

Exact condition. The condition is the same for every response, because every response is gated (§4.3, §4.11): read "fill" as "fill, ack, reject or BUSY", f(s) as the pair of frames, one per substream, up to need_A(s) and need_B(s), and for an ack as its own half's frame that holds the step whose released cut covers the op. Let fill F belong to step s, carried in ORDER frame f(s) of tenure t. F is in the final log if and only if the tenure is never deposed, or the seal has F* ≥ f(s) and no packet that step s depends on is unavailable. So F reverts if and only if t is deposed and one of these holds:

• (a) or (b) Some frame k ≤ f(s) of t is held by no reporter.
• (c) lead < 0 on the wire (§4.4, B-REP-2W) and the sequencer box lost power inside that window. This is the in-box case: the response left and its frame never did.
• (d) A packet of step s was unavailable or excluded. That needs client equivocation plus a dead sequencer, or a CHAIN tag failure (09).
• (e) A tail conflict truncated the prefix below f(s), which needs a Byzantine sequencer.

When does no reporter hold frame k? Either k was lost on both paths for every voter and the standby and was not repaired before the sequencer stopped, or the sequencer was cut off from every box that reports. A box that keeps receiving is a holder, and the leader waits for its report (§9.2), so isolating F1 from F2–F4 reverts nothing while R1 lives. Path loss is at switch ingress, so a loss on a path hits every replica behind that path at once. The paths are independent: separate NICs, switches and fibre. The budget assumptions in the table are inputs to B-REP-11.

Case (b) dominates the crash cases. REP_SELF_FENCE_US = 250 µs. Lowering it below ~200 µs risks spurious fences from ack jitter, and each spurious fence costs a 20 ms failover. The number to publish is that, with both paths healthy, a crash-fault revert of any response, fill or ack, needs the same frame lost on both independent paths followed by sequencer death within 0.5 ms, and the expected count is below 1e-13 per year. An acked order is never lost outside these cases (01 §14.5). Partition exposure is published separately: ≤ 0.85 ms of responses per dual-path isolation of the sequencer, and ~0.2 fills per year at average load. This matches DESIGN §0.3 and §10.

9.7 Failover budget (FAILOVER_TARGET_MS = 20)#

The clock starts at T, the sequencer's last frame. Every number is a max (budget) under a timely network (hop ≤ REP_HOP_MAX_MS = 0.8 ms), with an honest sequencer that crashed.

Detection stays at 2 ms: the seal fits 9 ms with 1.15 ms to spare. A 1 ms detection would need the fence to act first (REP_ACK_LIVE_US + REP_HOP_MAX_MS + REP_SELF_FENCE_US < 1 ms), which leaves ~0.2 ms for the fence timers and makes spurious 20 ms failovers likely. Typical: ~4.5 ms.

Every response already sent, fills and acks alike, is in the prefix. A response in the tail was sent ≤ 1.1 ms before T (its block closed, and its header did not leave), so it is final ≤ 1.1 + 7.85 ≈ 9.0 ms after it was sent: ≤ FINALITY_MAX_MS. Clients resend only packets with no response.

10. Byzantine cases#

10.1 Sequencer equivocation#

Action on a valid SeqEquivocation:

1. The holder broadcasts it, and every participant suspects at once (§9.1, reason EQUIV).
2. The failover runs as in §9. Simplex finalizes at most one of the two headers. If either is already in a notarized proposal, that one stands. The other one never enters the log.
3. The sequencer's index enters the epoch's deposed set, and it leaves seq_order from the next REP_Membership until operators clear it (BASE).
4. In CHAIN, the evidence is submitted as a slashing transaction. The bond is slashed automatically (09 owns the amount and the path).

The sequencer has no vote, so nothing changes in Simplex. The fault budget f = 1 is used up by it for the rest of the failover.

10.2 Byzantine replica#

Frame authenticity (referenced from §3.1). Frames carry no MAC. These are the threats and what stops each one:

• Outsiders and Byzantine replicas cannot inject into G_ord(seq, ·): the VLANs are dedicated, and the static switch ACLs are keyed by ingress port.
• A Byzantine peer that repairs with fake frames is caught by order_root for every signed block. Under f = 1, the unsigned tail is used only when the sequencer is the fault (§9.3), and then no replica is Byzantine.
• A compromised switch together with a dead sequencer is two faults, and only the unsigned tail is exposed. That is accepted for v1. MACsec on the replication links is the upgrade path. It is a COTS NIC and switch feature and needs no change to the frame format.

10.3 Partitions: safety over liveness#

• No conflicting finality, ever. Two quorums intersect in 2Q − n ≥ 2 ≥ f + 1 participants: BASE 2·3 − 4 = 2, CHAIN 2·4 − 5 = 3. No timeout finalizes anything, and no timeout deposes without Q.
• The sequencer is cut off from voters. It self-fences once fewer than 3 voters ack within 250 µs (INV-REP-6, ≤ 1.15 ms). If the voters still have Q among themselves, they fail over (§9). If not, everything stays halted.
• Voters split with no Q on either side (for example BASE 2/2). Nothing finalizes and no failover can start. The finality gate (INV-REP-10, §6.8) halts the book once more than 8 closed blocks are not final: ≤ ~9 ms of fills, none reverted. Voters hold their frames, and those frames finalize when the partition heals. The only exception is a sequencer that dies before then, which is the §9.6 condition.
• Heal. Views resume and the backlog finalizes 16 blocks per view, so a 9-block backlog clears in 1 view, and a longer halt's empty blocks clear at 16 per view.

11. Client diversity (CLIENT_SPLIT)#

11.1 What is diverse#

The sequencer, the standby and the spares are Rust but never vote, so a failover never changes this table (01 §14.4).

A Vx box runs the Rust shell (network, data tier client, consensus, storage, structural checks) and vyrn-vx-client, which contains its own Vx decoder of §3 and its own replay engine. The decision is that the Vx client decodes the raw ORDER bytes itself. Otherwise a bug in the shared Rust decoder would feed both clients the same wrong records, and the roots would agree on a wrong state. The encoding in §3 is therefore implemented twice, and its fuzz corpus is cross-checked (T-REP-2). What stays common-mode is the consensus and network code, whose faults cost liveness (Simplex safety does not depend on the replay), and the frame transport.

11.2 Detection and alarm#

• Signal. REP_A_DIVERGE (INV-REP-9, §7.4): two root digests for one index, each signed by ≥ f + 1, or no root certified within 2 s.
• Latency (budget). The divergent block becomes final at t. Its root height is ≤ 1 s later, and the roots are computed and voted ≤ 10 ms after that. Detection therefore happens at ≤ t + 1.01 s, and the book halts within 1 ms more.
• Alarm. The alarm pages, carrying (index, digests, signer sets). Clients receive BOOK_HALTED. Fills that were already final keep their place in the final order log, and only the state that follows from them is in question.

11.3 Procedure#

1. Halt. This is automatic. Participants stop voting and the sequencer halts the book. An epoch handoff that falls in a halt is postponed (08).
2. Freeze. Every participant keeps its full log and its snapshots. No compaction or archive deletion happens until the procedure closes.
3. Locate. vyrn-sim replays from the last certified root R(i−1) with the Rust engine, vyrn-vx-client and vyrn-vx-oracle, diffs the state after every message, and reports the first divergent (height, StepId | PacketRecord).
4. Classify. The three vote 2 of 3, checked against 05: a Rust bug, a Vx bug, or a spec ambiguity, which is fixed in the spec first.
5. Fix. The faulty client is fixed and released under 11's process. The fixed binary must reproduce the majority root for every root index since the divergence.
6. Resume. Participants restart on the fixed binary and replay from R(i−1). Roots certify and votes resume. If the Rust engine was wrong, the sequencer restarts on the fixed binary through an orderly tenure change, and the fills that differ from the correct semantics are handled under 10's error-trade policy. The order log is never rewritten.

The target for steps 2 to 6 is ≤ 4 h. That is an ops budget, and 11 owns the runbook.

11.4 Why this halts and does not fork#

A fork would need two certified histories. The order has one history, because one Simplex instance finalizes it. A state needs a root certificate, which needs Q signatures, and no single client has Q. A disagreement between clients therefore produces no root certificate. INV-REP-9 then stops new order finality at once, and the gate stops it within 5000 blocks in any case.

12. WAN followers#

• Role. All three homes run a full set at all times (DESIGN §6). The voters, standby and spares of an inactive home are followers. They receive finalized blocks, verify them, store them and replay them. They do not vote in the active home's Simplex.

• Feed. The facility's archive node (§8.4) pulls finalized blocks from its voter and sends block h when h mod n_fac equals its facility index (n_fac = 4 BASE, 5 CHAIN). Each feeder sends to one gateway per follower home, and the gateway fans out inside its metro. A gateway that has missed a height for 5 ms requests it from any other feeder.

• Unit.

  REP_WanBlock { BlockRecord header part (§8.2), FinalityCert or seal cert, order bytes,
                 data in O-order coded with pbz, unavailable list }
  

  The pkt_hash list is dropped, because followers recompute the hashes.

• Codec pbz. Message fields are delta-coded per session: type, flags 1 B; tick as a ZigZag LEB128 delta against the session's previous tick; qty as LEB128; client_tag as a delta. The packet header is implicit from the PacketRecord.

  • Tags: BASE drops them. data_root excludes the tags, so followers still verify every packet. CHAIN keeps them, because K_com tags are publicly re-verifiable after disclosure (09).
  • The ratio budget is 3× on messages (B-REP-12). The cost is ~5 ns per message, spread over the feeder boxes.

• Links. Each pair of homes has REP_WAN_LINKS = 2 × 100G waves on diverse routes.
• Lag (budget). Finality ≤ 5.65 ms + pull and encode ≤ 2 ms + WAN one-way (35–110 ms) + decode and verify ≤ 2 ms gives REP_WAN_LAG = WAN one-way + ≤ 10 ms. The next home in the schedule has priority on shared links. The other follower may lag ≤ 10 min at sustained peak, and it must be caught up 1 h before its epoch (REP_A_WANLAG fires otherwise).
• To 08. At handoff, the next home holds the blocks through last_height within REP_WAN_LAG of their finalization, and its replay is within 2 ms of them. 08 checks the HandoffRecord's state_root and log_hash (§2.2) against the follower's own values. Followers verify certificates against the active home's REP_Membership.

13. Replica recovery and membership#

13.1 Membership record#

REP_Membership { epoch_from: Epoch, home: HomeCode, mode: u8, n: u8, q: u8,
  voters: n × { index u8, vote_key PubKey, client u8 (0 Rust, 1 Vx), facility u8,
                data_addr [18] (IPv6 or v4-mapped, plus port), repl_addr [18] },
  seq_order: [{ index u8, role u8 (0 primary, 1 standby, 2 spare), seq_key PubKey, facility u8,
                client_addr [18], repl_addr [18] }],
  recovery: u8 (voter index of the recovery service, §8.5),
  data_boxes: [{ index u8, facility u8, addr [18] }],
  cert: Q signatures of the current set over BLAKE3("vyrn      v1 membership" ‖ body) }

The rules are these:

• Membership changes only at epoch boundaries, at epoch_from.
• A record must be certified ≥ REP_MEMBERSHIP_LEAD_S = 3600 s before its boundary.
• Validation rejects any record in which one client's voter count reaches Q (CLIENT_SPLIT), q ≠ Q_BASE or Q_CHAIN for the mode, seq_order names a Vx box, any key appears both as a vote_key and a seq_key, the standby is not in F2, or recovery names a Vx voter.
• In CHAIN mode, 09 decides validator-set changes, and 07 only applies them at the boundary.

13.2 Recovering a failed replica (same identity)#

1. Fence the old box. Power it off through the BMC (Redfish or IPMI) and confirm, or disable its switch ports and confirm. Two boxes never hold one vote key at the same time.
2. Start vyrn-replica on the facility's spare box, and load the key from the secret store (06). That spare leaves seq_order for the rest of the epoch.
3. Catch up as in §7.6. Votes resume once live frames and data flow, before the state is caught up.

Budget: fence ≤ 30 s, start ≤ 60 s, votes ≤ 2 min, state ≤ 5 min. Until then there is no fault margin (BASE: 3 of 4 = Q), and a second fault halts the book within 8 blocks (§6.8).

13.3 Adding or replacing a replica (new identity)#

A new identity enters only through a new REP_Membership at the next boundary. Before epoch_from, the new box runs as a follower (§12), so it is caught up when its first epoch starts. A box that is not caught up at epoch_from counts as a faulty voter until it is.

14. Failure matrix#

15. Benchmarks and tests owned here#

Interfaces to other chapters#

Local definitions (prefix REP_)#

Requests to 01#

Resolved by 01 and removed: PacketRecord.flags (§6.6: SHED, ADVANCE), INV-REP-6's condition (§14.16), the mode-neutral FIN_GATE_BLOCKS (§14.9), the data-home rule (§14.16) and AcctTake (§14.17 item 4). Open:

1. §1 Words: add tenure, voter (a replica that votes; N_BASE/N_CHAIN count them), warm standby, spare and substream. The "replica" entry says "votes in consensus", but the standby replays and never votes.
2. §6.6: add a kind payload to StepRecord (Rescan band, Control code, Uncross data; 05 R1, R2); note that StepRecord.cut is the union of two half-cuts, logged as a home record plus at most one cut-only record (§3.8); add AcctTake { account, ver, lane } to the record list.
3. §7: INV-ACK-AFTER-CUT should read "after a released cut includes the op (LaneCut, written by the lane's stream core after its frame's sfence)". Add INV-REP-11 next to it, and list INV-REP-6 by name with §14.16's "R1 alone never keeps it alive".
4. §14.9: state that FIN_GATE_BLOCKS counts closed blocks: block h + 9 opens only when block h is final. 07 §6.8 needs it (7.65 ms after close in BASE), and 09 §4.6 now uses it (7.11 ms after close in CHAIN, with the data home outside F1).
5. §6.4: response codes for the client watermarks of §6.9: Durable { step, height } and Final { height, view }, as two separate records.
6. §5: promote REP_SELF_FENCE_US, REP_FENCE_ACKS, REP_HOP_MAX_MS, REP_ROOT_BLOCKS, REP_HASH_LAG_NS, REP_SEAL_US and REP_ORDER_FRAME_MAX. Other chapters reason with them.
7. §6: promote the frame header (§3.2, with its stream byte), REP_Proposal and REP_FinalityCert (§6.3, §6.5), REP_DurableMark, and the log_hash definition (§2.2).
8. §10 and §14.7: the sequencer's replication cards are 2 × dual-port X2542, one port per stream core per path (06 §2.3); two stream cores on one single-port X2541 would be two CTPIO writers. A replica has 12 × 61.44 TB (§8.1). The standby and spares are built to the vyrn-seq spec and a spare adds 12 NVMe.
9. §14.15 item 3: a stream core runs at ~0.41, not ≤ 0.25, because the per-frame CTPIO cost does not halve with the lanes (§4.7). The gate bound does not depend on it.
10. §14.2: the cross fill at index 9 with the 15 ns step is 947 ns, not 880: the gate opens ≤ 142 ns after the step starts for a step that spans both halves (§4.7).
11. §5: N_LANES_REF = 50 disagrees with the core map's 49 lanes; 07 uses 49 (§3.4).

Open questions#