06. Platform: the Unix host

This chapter specifies the machines and the operating system under every VYRN process: the parts list per target and per role, the BIOS, the Linux kernel and its tuning, the NIC backends behind one Rust trait, the C shim, time, memory, AWS, the consumer box, macOS for development, host security, operating procedures, and the checklist a box passes before it joins a facility.

The engine chapters (05, 07) assume a quiet core, a hardware RX timestamp and a NIC that steers each packet to the lane that owns it. This chapter is where those assumptions are made true, or where the text says exactly which target cannot make them true and what it does instead.

0. Decisions in one table#

1. How this chapter meets the hard constraints#

2. Bill of materials#

2.1 One platform for every REF role#

Decision. The REF sequencer, standby sequencer, replica, validator, data node and observer share one board, one CPU and one memory configuration.

The reason: failover promotes the home's warm standby vyrn-seq within FAILOVER_TARGET_MS, never a voting replica (01 §14.4). The standby is the sequencer's SKU, NICs included, so a failover does not change the latency bound. Replicas, data nodes and observers could use cheaper CPUs, but a separate board doubles the spares pool, the BIOS profiles and the validation runs for a saving of a few thousand dollars per box, and a replica board can be re-carded as a standby.

2.2 Per-role differences on REF#

The archive node is the only box with a different board class: a single-socket SP5 storage chassis with 24 NVMe bays. Its CPU is any Zen 5 EPYC with ≥ 32 cores. It holds the log segments that are older than the replica's on-box window (next paragraph), so that 01 §14.7's 7 days are kept inside the facility. 24 × 122.88 TB is ~2.9 PB (vendor capacity), which is 7 days at up to ~18% average load; a facility whose measured average is higher adds a second archive node. The archive node streams continuously: it pulls each block from its voter as soon as the block is final, and it never waits for a segment to fill (07 §8.4).

Why the replica's log is split from the archive: DESIGN §7 gives ~26 GB/s at sustained peak and ~230 TB per day at 10% average load. Seven days is ~1.6 PB. One single-socket box has 128 PCIe lanes (vendor), and the replica's four NICs take 64 of them, so no replica can hold a week. The replica writes to 12 × 61.44 TB Gen5 TLC drives (~737 TB, vendor capacity; ≥ 60 GB/s of write budget at ~5 GB/s per drive, vendor). That meets 01 §14.16: PLT_ONBOX_HOT_H_PEAK ≥ 6 h at sustained peak (737 TB ÷ 26 GB/s ≈ 7.9 h) and PLT_ONBOX_HOT_H = 12 h at ≤ 20% average load (~225 TB). The 30.72 TB drives of 01 §14.7 (~368 TB) hold only ~3.9 h at sustained peak, which is why the drive changed (Requests to 01). Retention never blocks replication: the oldest block goes first, archived or not (07 §8.3). 07 owns the segment format and the retention rule; this chapter owns the boxes. The recovery service of 01 §14.6, a role on one replica per home, keeps RESP_RETAIN_PKTS per session on these drives: ≤ 3.8 TB at the maximum response size.

2.3 PCIe lane plan (REF sequencer, replica, observer)#

The EPYC 9575F has 128 Gen5 lanes in four I/O-die quadrants of 32 (vendor). Every device is on CPU lanes, with no PCIe switch.

                 EPYC 9575F I/O die: 4 quadrants × 32 lanes, CCD0-7 over Infinity Fabric
        quadrant P0           quadrant P1           quadrant P2           quadrant P3
        x16     x16           x16     x16           x16     x16           x16     x16
SEQ     NIC0    2×M.2 boot    NIC1    NIC4          NIC2    (empty)       NIC3    NIC5
        client  (x8) +x8 free client  X2542 path A  client                client  X2542 path B
        lanes 0-15            lanes 16-24 a: strm-A lanes 25-40           lanes 41-48 a: strm-A
        TX-A (core 3)         TX-C      b: strm-B   TX-B (core 5)         TX-C,     b: strm-B
        → leaf A              → leaf A  → leaf A    → leaf B              pool 42-48 → leaf B
        used: 6 × 16 + 8 = 104 of 128 lanes
REPLICA NIC0    4×NVMe        NIC1    4×NVMe        NIC2    4×NVMe        NIC3    2×M.2 boot
        path A                data                  path B                data    (x8) +x8 free
        + data                                      + data
        → leaf A              → leaf A              → leaf B              → leaf B
        used: 4 × 16 + 12 × 4 + 8 = 120 of 128 lanes
OBSERVER NIC0   2×M.2 boot    NIC1    NIC3          NIC2    4×NVMe        NIC4    NIC5
        CX-6 Dx (x8) +x8 free CX-6 Dx X2541         CX-6 Dx               X2541   X2541
        taps 0,1              taps 2,3 frames A     MD out a, b           frames B MD out
        RX only               RX only  → leaf A     → leaf A, leaf B      → leaf B → leaf B
        used: 6 × 16 + 16 + 8 = 120 of 128 lanes

Rules:

1. Each client NIC is on its own I/O-die quadrant, on CPU lanes, with no PCIe switch. The two replication NICs are on two different quadrants. A retimer is allowed only where the board requires one for that slot. B1 runs per slot, so a slot whose retimer costs latency fails B1 and is not used for a NIC.
2. The X2541 is a PCIe Gen3 x16 card (vendor). At ~25M packets/s per direction per client card (§6.3) this is ~52 Gbps on the wire per direction, inside Gen3 x16. PLT-B-PPS checks it.
3. Lanes split across client cards in the proportions above. The lane → card map is configuration. Clients learn it from 04's session setup as (ip of card, UDP port) per lane. Each client port has one fill TX core (01 §14.17): TX-A writes NIC0, TX-B NIC2, TX-C NIC1 and NIC3. That gives TX-A 16 lanes, TX-B 16 and TX-C 17, of which 7 are the lower-volume pool, so each TX core runs near the ≈ 0.22 load of 01 §14.17. TX-A serves only lanes of half A and TX-B only half B; TX-C serves both halves, and the frame gate does not care, because every TX core checks both halves' release words (07 §4.2). The 7 pool lanes are all on NIC3 (§6.3).
4. Order-stream path A leaves on NIC4 (cabled to leaf A) and path B on NIC5 (cabled to leaf B). This meets REPL_PATHS = 2 on two switches (01 §14.7). NIC4 and NIC5 are dual-port X2542 cards (vendor: 2 × 100GbE, PCIe Gen3 x16). Port a of each card is written only by stream-A (core 4) and port b only by stream-B (core 7), so each stream core writes its own frames to both paths, and each port still has exactly one CTPIO writer (INV-PLT-7): no core can make another's CTPIO write fall back (§6.3). Each replication port carries ~13 Gbps at peak (07 §3.13), and client TX load stays at ~52 Gbps per port. Six single-port X2541 cards could not do this: two stream cores on one replication port would be two CTPIO writers.
5. On a replica, the data-tier copy of the packets (~190 Gbps at peak, 07) is spread over all four ports; NIC0 also takes order-frame path A and NIC2 path B. That is ≤ 70% of 100G per port.
6. The cards are capacity, not redundancy. A failed card or link makes the box unhealthy (§13.2), and 07's failover moves the book. There is no lane re-homing inside a box.
7. Observer (01 §14.16). The four tap fibres carry ~52 Gbps each; two of them share one CX-6 Dx (PCIe Gen4 x16, vendor), ~104 Gbps per card. Full MBO out is ~208 Gbps (10), spread over CX-6 Dx NIC2's two ports and X2541 NIC5, ≤ 70% of each 100G port. Order frames arrive on two X2541 ports, one per path, at ~26% each (07 §3.13). The CX-6 Dx cards run the dpdk backend (§6.4); the taps need hardware RX stamps (§3.3), not CTPIO.

2.4 Network in one facility#

ASCII original

                clients (cross-connects, colo racks)
                    │                         │
             ┌──────┴──────┐  ISL (2×100G) ┌──┴──────────┐
             │   leaf A    │═══════════════│   leaf B    │   merchant silicon,
             │  (PTP BC)   │               │  (PTP BC)   │   cut-through, PTP boundary clock
             └┬───┬───┬──┬─┘               └┬───┬───┬──┬─┘
              │   │   │  └─ GM-1 (primary)  │   │   │  └─ GM-2 (backup)
              │   │   │                     │   │   │
       NIC0 NIC1 NIC4 │              NIC2 NIC3 NIC5 │    vyrn-seq (client 0-3, repl 4-5)
                  │   └─ data nodes, observer   │   └─ data nodes, observer
                  │                             │
       metro path A ──► F2…Fn          metro path B ──► F2…Fn      (07 owns the metro links)

Passive optical taps on the sequencer's four client RX fibres feed the observer (10 §7.1). Each tap's insertion loss must leave the sequencer's optics inside their receive power budget (PLT-V-4 reads the RX power from the module).

Switches. Merchant-silicon L2/L3 switches only. Accepted silicon families: Broadcom Trident 3/4 and Tomahawk 3/4/5, NVIDIA Spectrum-2/3/4. Example products: Arista 7050X3 and 7060X series, NVIDIA SN3000 and SN4000 series. Requirements:

Switch latency is outside LAT_MAX_NS, because REQ-LAT-1 starts at the exchange NIC. It is inside the order-frame path and the finality budget (07).

2.5 PTP grandmaster#

Decision. Two GNSS-disciplined grandmaster appliances per facility, one on each leaf, from the Meinberg LANTIME M-series or Microchip TimeProvider 4100 class, with a rubidium or OCXO holdover oscillator. GM-1 has BMCA priority1 100, GM-2 has 110. We configure them and run no logic on them. Each has its own GNSS antenna feed and a 1PPS output that the lab uses to calibrate.

2.6 CONSUMER#

A ConnectX-6 Dx dual-port 100GbE can replace both NICs (one port per switch path) if it passes B1 on this board with the dpdk backend. Until it does, the X2541 + X2522 pair is the BOM.

2.7 AWS#

Every instance has one management ENI on the kernel ena driver and one or more data ENIs bound to vfio-pci for DPDK (§9.2). .metal is required: nohz_full and isolation need a host kernel that owns the physical cores, and a hypervisor would add exits we cannot see.

2.8 DEV#

Any Apple M-series Mac with ≥ 32 GB of memory and macOS 14 or later. No NIC requirement. §11 lists what runs.

2.9 Excluded parts#

3. BIOS and firmware#

Board vendors name these options differently. The names below are AMD's CBS names. Validation (§14) checks the effect from Linux, never the menu text, so a renamed option cannot pass with the wrong value.

3.1 REF (EPYC 9575F)#

3.2 CONSUMER (Ryzen 9 9950X3D)#

3.3 Firmware pinning#

Every box records five versions in its platform report (§14): BIOS, BMC, each NIC's firmware (sfboot/sfupdate for Solarflare, mlxfwmanager for ConnectX), each NVMe drive's firmware, and the PTP grandmaster's firmware. A facility runs one version of each per role. A firmware change is rolled to one box, which runs the full §14 checklist and B1–B5, before the rest. Firmware changes happen only in a maintenance window (§13.4).

Solarflare X2541 options set with sfboot: PXE boot disabled, SR-IOV disabled, and the firmware variant under which ef_vi CTPIO and hardware RX/TX timestamps both work. The variant is not trusted by name: PLT-V-11 opens a VI with EF_VI_TX_CTPIO | EF_VI_RX_TIMESTAMPS and sends and receives one stamped frame. Where a card SKU gates hardware timestamping behind a vendor licence, the BOM buys the SKU or licence that enables it; a card without it fails PLT-V-11. ConnectX options set with mlxconfig: REAL_TIME_CLOCK_ENABLE=1 (timestamps in ns of PTP time), SRIOV_EN=0.

4. The Linux kernel#

4.1 Version policy#

Decision. Production runs one exact kernel version per facility, taken from a kernel.org long-term branch no older than 6.6. The default in draft 0.1 is the newest 6.12.y that has passed §14 on every role. A distribution kernel (Debian 13, RHEL 10) is allowed when its config meets the list below.

The reasons: LTS branches get security fixes for years without feature churn, and 6.6 is the oldest branch with everything this chapter needs (the isolcpus flags, nohz_full with the residual tick offloaded, XDP RX timestamp hints from 6.3, hugepages=<node>:<count>).

Required config:

CONFIG_NO_HZ_FULL=y          CONFIG_RCU_NOCB_CPU=y        CONFIG_CPU_ISOLATION=y
CONFIG_HUGETLBFS=y           CONFIG_TRANSPARENT_HUGEPAGE=y (set to never at boot)
CONFIG_PTP_1588_CLOCK=y      CONFIG_VFIO_PCI=m            CONFIG_XDP_SOCKETS=y
CONFIG_BPF_SYSCALL=y         CONFIG_BPF_JIT=y             CONFIG_SECCOMP_FILTER=y
CONFIG_SECURITY_LOCKDOWN_LSM=y  CONFIG_MODULE_SIG_FORCE=y CONFIG_IO_URING=y
CONFIG_PREEMPT_VOLUNTARY=y   (not PREEMPT_RT)             CONFIG_X86_AMD_PSTATE=y
CONFIG_AMD_MEM_ENCRYPT=y     CONFIG_EDAC_AMD64=m          CONFIG_SENSORS_K10TEMP=m
CONFIG_PCIE_PTM=y

PREEMPT_RT is not used. It shortens the wake-up latency of woken tasks. No hot thread is ever woken: it never sleeps. RT adds cost to locks on the housekeeping core and nothing to hot cores.

Out-of-tree modules: the Onload/ef_vi kernel driver (sfc, sfc_char, sfc_resource, onload built from the vendored Onload release, 11 §4.2) and nothing else. They are signed with the facility's module key (§12), because the kernel enforces MODULE_SIG_FORCE.

A kernel upgrade follows §3.3: one box, full §14, B1–B5, then the facility, in maintenance windows.

4.2 Boot parameters#

Every REF box (64 cores; core 51 is housekeeping, 01 §10; one command line for every role, so a standby or a re-carded replica needs no reboot to sequence):

isolcpus=managed_irq,domain,0-50,52-63 nohz_full=0-50,52-63 rcu_nocbs=0-50,52-63 rcu_nocb_poll irqaffinity=51
default_hugepagesz=1G hugepagesz=1G hugepages=640 transparent_hugepage=never
nosmt processor.max_cstate=1 cpufreq.default_governor=performance amd_pstate=active
tsc=reliable clocksource=tsc nosoftlockup nowatchdog nmi_watchdog=0 skew_tick=1
numa_balancing=disable iommu=pt pcie_aspm=off mitigations=auto lockdown=integrity
nvme.poll_queues=8 quiet loglevel=3 console=ttyS0,115200

CONSUMER (16 cores; core 0 is housekeeping and control):

isolcpus=managed_irq,domain,1-15 nohz_full=1-15 rcu_nocbs=1-15 rcu_nocb_poll irqaffinity=0
default_hugepagesz=1G hugepagesz=1G hugepages=104 transparent_hugepage=never
nosmt processor.max_cstate=1 cpufreq.default_governor=performance amd_pstate=active
tsc=reliable clocksource=tsc nosoftlockup nowatchdog nmi_watchdog=0 skew_tick=1
numa_balancing=disable iommu=pt pcie_aspm=off mitigations=auto lockdown=integrity
nvme.poll_queues=4 quiet loglevel=3

AWS: generated by vyrn-admin gen-cmdline --target aws, which reads the instance topology and the ENA devices' NUMA node (§9.3). On c6in.metal the result has the form below, where H is the first core of the ENA node, I is the rest of that node, and the other node's cores stay non-isolated for consensus, admin and catch-up work:

isolcpus=managed_irq,domain,<I> nohz_full=<I> rcu_nocbs=<I> rcu_nocb_poll irqaffinity=<H>,<other node>
default_hugepagesz=1G hugepagesz=1G hugepages=<ena node>:<N>,<other node>:16
transparent_hugepage=never nosmt intel_idle.max_cstate=1 processor.max_cstate=1
cpufreq.default_governor=performance tsc=reliable clocksource=tsc nosoftlockup nowatchdog
nmi_watchdog=0 skew_tick=1 numa_balancing=disable iommu=pt mitigations=auto lockdown=integrity

Every parameter and its reason:

4.3 Housekeeping work that must not reach isolated cores#

The boot parameters do not cover everything. vyrn-tune.service (§4.8) does the rest before any vyrn-* process starts:

1. Writes HK (51 on REF, 0 on CONSUMER) or the AWS housekeeping list to /sys/devices/virtual/workqueue/cpumask, so unbound workqueues run on housekeeping.
2. Sets every writable /proc/irq/*/smp_affinity_list to the housekeeping list. A write that fails with EIO is a managed IRQ; it is checked in step 6 instead.
3. Reduces the kernel netdev queues of every fast-path port to 2 (ethtool -L <dev> combined 2) and pins their IRQs to HK. ef_vi VIs and DPDK queues use no interrupts at all.
4. Writes 0 to /sys/kernel/mm/ksm/run and never to /sys/kernel/mm/transparent_hugepage/defrag.
5. Masks irqbalance.service, tuned.service, systemd-oomd.service, fstrim.timer and systemd-timesyncd.service, and stops chronyd on boxes where phc2sys owns the system clock.
6. Verifies, and fails the unit if any check fails: no IRQ in /proc/irq/*/effective_affinity_list names an isolated core, except NVMe queues whose mask is that one core (§4.7); no kthread other than the per-CPU bound ones (ksoftirqd/N, migration/N, cpuhp/N, kworker/N:*) has an isolated core in its affinity.

4.4 CPU vulnerability mitigations#

Decision. mitigations=auto, the kernel default. We do not turn mitigations off.

The reasons:

• They cost the hot path nothing. Spectre v2, Retbleed, SRSO and similar mitigations act on kernel entry, exit and context switches. A hot core makes no syscall, takes no interrupt and never context-switches after seal() (P7). Its user-mode code runs at the same speed with or without them. On Zen 5 the remaining user-visible control, speculative store bypass disable, is in prctl mode by default and is off for our threads.
• HK pays a little. Housekeeping, ptp4l and control-plane syscalls get slower by a few percent. Nothing on HK is on the 1 µs path.
• The box runs code that attackers reach. Validators accept packets from the public internet (09), and replicas run Commonware's p2p stack. A local privilege escalation after a memory-safety bug in a dependency is the threat that mitigations reduce.
• SMT is off, which already removes the cross-thread attacks (MDS, L1TF class) whose mitigations are the expensive ones.

PLT-B-MIT runs B3 with mitigations=auto and with mitigations=off on one lab box. If the cross p99.99 or the maximum differs by more than 1%, this decision is reopened with that number. The expected difference is zero.

4.5 sysctls#

/etc/sysctl.d/90-vyrn.conf, applied before any vyrn-* process starts and never changed while one runs (§4.9 rule 5):

kernel.numa_balancing = 0            # as numa_balancing=disable
kernel.watchdog = 0                  # soft and hard lockup detectors off
kernel.nmi_watchdog = 0
kernel.sched_autogroup_enabled = 0
kernel.timer_migration = 1           # timers queued on isolated cores migrate to housekeeping
vm.stat_interval = 120               # per-CPU vmstat folding every 120 s, not every 1 s
vm.swappiness = 0                    # and no swap device exists
vm.zone_reclaim_mode = 0
vm.compaction_proactiveness = 0      # no proactive compaction
vm.compact_unevictable_allowed = 0   # compaction never migrates mlocked pages (a migration is a fault)
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1
kernel.yama.ptrace_scope = 2         # only CAP_SYS_PTRACE may attach
kernel.unprivileged_bpf_disabled = 1
kernel.kexec_load_disabled = 1
kernel.perf_event_paranoid = 2
kernel.core_pattern = |/bin/false    # no core dumps: they would contain keys
fs.suid_dumpable = 0
net.core.bpf_jit_harden = 2

4.6 cpusets and cgroups#

cgroup v2, managed by systemd:

ASCII original

-.slice
 ├─ init.scope        AllowedCPUs=HK       (REF: 51)
 ├─ system.slice      AllowedCPUs=HK       sshd, journald, ptp4l, phc2sys, vyrn-timemon, vyrn-admin
 ├─ user.slice        AllowedCPUs=HK       operator shells
 └─ vyrn.slice        AllowedCPUs=0-63   vyrn-seq | vyrn-replica | vyrn-data | vyrn-md (one per box)

/etc/systemd/system.conf sets CPUAffinity=HK, so every service starts on HK unless its slice allows more. vyrn.slice allows every core, and the process pins each of its threads to exactly one core with sched_setaffinity from its core map. isolcpus=domain guarantees that nothing else is scheduled there. No cpuset partition (cpuset.cpus.partition=isolated) is used: the boot-time isolation already covers it, and managed_irq has no cgroup equivalent.

4.7 IRQ affinity and the NVMe exception#

After vyrn-tune, the only interrupts in the system go to HK, with one exception that is by design: an NVMe I/O queue whose managed affinity mask is a single isolated core. Linux creates one such queue per CPU, and managed_irq cannot move an IRQ whose mask has no housekeeping core. The queue only interrupts when I/O is submitted from that core. Hot cores never submit I/O, so those IRQ lines stay at zero; the quiet probe (§4.11) and PLT-V-7 check it.

Replica log writers (07) run on WARM cores (§4.9) and use io_uring with IORING_SETUP_IOPOLL on the NVMe poll queues (nvme.poll_queues), with O_DIRECT. Polled queues have no interrupt line at all.

4.8 systemd units#

vyrn-seq.service:

[Unit]
Description=VYRN      sequencer
Requires=vyrn-tune.service vyrn-time.target
After=vyrn-tune.service vyrn-time.target network-online.target

[Service]
Type=notify
User=vyrn
Group=vyrn
Slice=vyrn.slice
ExecStart=/opt/vyrn/bin/vyrn-seq --config /etc/vyrn/seq.toml
# Never restart a sequencer automatically. A restarted sequencer is stale; 07's failover
# has already promoted a replica. vyrn-replica.service uses Restart=on-failure instead.
Restart=no
TimeoutStartSec=900
WatchdogSec=10
LimitMEMLOCK=infinity
LimitCORE=0
LimitNOFILE=65536
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_IPC_LOCK
NoNewPrivileges=yes
RestrictRealtime=yes
OOMScoreAdjust=-1000
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
MemoryDenyWriteExecute=yes
LockPersonality=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_NETLINK AF_XDP
SystemCallArchitectures=native
ReadWritePaths=/var/lib/vyrn /run/vyrn /dev/hugepages1G
DevicePolicy=closed
DeviceAllow=/dev/sfc_char rw
DeviceAllow=/dev/onload rw
DeviceAllow=/dev/vfio/vfio rw
DeviceAllow=char-vfio rw

WatchdogSec is fed by the control thread only. A stalled control thread means the box cannot take part in epochs or drains, so systemd kills it, which is a failover. TimeoutStartSec covers prefaulting 640 GiB (§5.3). RestrictRealtime=yes enforces §4.10.

4.9 Core classes and the rules for each#

Rules that apply to the whole box while a role process is ACTIVE, because each one sends an IPI to every core, isolated or not:

1. No TLB shootdown in the role process. munmap, mprotect, mremap and madvise(MADV_DONTNEED) on memory the process has mapped send a flush IPI to every CPU that runs a thread of that process, which includes every hot core. The WARM-class allocator never returns memory, and a process-wide seccomp rule returns EPERM for these calls after seal() and counts them.
2. No kernel text patching. Enabling ftrace, kprobes, a tracepoint, or anything that flips a static key calls text_poke, which synchronises every CPU with an IPI. perf on an isolated core also uses NMIs. These tools are for lab boxes and for boxes in DRAINED.
3. No module load or unload and no BPF program load. Both can purge lazy vmap areas, which flushes the kernel TLB on every CPU. The AF_XDP program is loaded before seal().
4. No MSR read of another core. /dev/cpu/N/msr and turbostat run rdmsr on core N through an IPI. Temperature comes from k10temp (read through the SMN, no IPI), and frequency is checked at validation time only.
5. No sysctl or sysfs write that changes kernel behaviour, including drop_caches, compact_memory, nr_hugepages and the numa_balancing toggle.
6. No CPU hotplug and no cpufreq governor change.

vyrn-admin refuses these operations on a box whose role process is ACTIVE and logs the attempt.

4.10 Why hot cores do not use SCHED_FIFO#

Decision. Every hot and warm thread runs SCHED_OTHER at nice 0, alone on its core. RestrictRealtime=yes forbids anything else.

• There is nothing to win against. A real-time policy decides who runs when two threads are runnable on one core. isolcpus=domain and pinning guarantee there is only ever one.
• SCHED_FIFO would hurt. The default RT throttle (sched_rt_runtime_us = 950000) takes the core away from a 100%-busy FIFO thread for 50 ms of every second. Turning the throttle off instead lets a FIFO busy-loop starve the per-CPU kthreads that must run occasionally on every core (ksoftirqd/N, migration/N, the stop machine), which turns a rare 2 µs interruption into an RCU stall or a hung CPU-hotplug operation.
• SCHED_OTHER with nohz_full stops the tick once the core has exactly one runnable task. A per-CPU kthread that wakes up on a hot core is a gap above PLT_JITTER_MAX_NS, which the quiet probe fails and the live gap metric reports (§4.11).

4.11 The quiet-core probe#

A core is quiet when nothing but our thread runs on it. The probe measures that directly.

/// Runs on one isolated core, pinned, after seal(). Records every gap between two
/// consecutive TSC reads that is longer than PLT_JITTER_FLOOR_NS.
pub fn quiet_probe(core: CoreId, window: Duration, tsc: &TscMap) -> ProbeReport {
    let floor = tsc.ns_to_ticks(PLT_JITTER_FLOOR_NS);
    let end = rdtsc_ordered() + tsc.ns_to_ticks(window.as_nanos() as u64);
    let mut hist = GapHist::new();          // preallocated: log2 buckets from 16 ns to 1 ms
    let mut prev = rdtsc_ordered();          // lfence; rdtsc
    while prev < end {
        let now = rdtsc_ordered();
        let gap = now - prev;
        if gap > floor { hist.record(tsc.ticks_to_ns(gap)); }
        prev = now;
    }
    ProbeReport { core, hist, irq_delta: read_irq_counts_delta(core) }
}

The same gap measurement runs inside every production hot loop: each idle Core::poll iteration (one that found no work) compares its TSC with the previous iteration's and records gaps above PLT_JITTER_FLOOR_NS in a per-core histogram that the control core exports. So the probe is also a live metric (plt_loop_gap_ns), and it does not count the core's own work, which the budgets already price.

Why the thresholds are maxima and not counts: a gap on a hot core after a packet has passed deadline admission (01 §14.1) makes that packet's response late by the gap. Nothing can shed the packet afterwards. So every gap must fit inside the guard: PLT_JITTER_MAX_NS = 20 ns for gaps, which is all of DEADLINE_GUARD_NS = 20 (01 §14.1). The TSC↔PHC error and the ≤ PLT_RX_STAMP_CAL_NS = 10 ns calibration error of the RX stamp against the first bit (§14.2) are not in the guard; lanes add both to age (§8.4).

Pass thresholds per isolated core, over PLT_JITTER_WINDOW_S = 600 s (budget):

The probe runs on all isolated cores at once, with the role process stopped and with synthetic load on HK (ssh, journalctl -f, a 1 GB/s dd to a boot drive), because housekeeping load is what leaks. rtla osnoise (kernel tools) runs as a second opinion during validation only: it agrees with the probe or the box fails. A gap that the probe sees but /proc/interrupts does not explain is an SMI or a firmware event, and the box goes back to §3.

5. Memory#

5.1 Huge-page layout#

Every hot region is a 1 GiB huge page from one hugetlbfs mount, /dev/hugepages1G (pagesize=1G, mode 0770, group vyrn). Regions are anonymous MAP_PRIVATE | MAP_HUGETLB | MAP_HUGE_1GB mappings, except the trace rings, which are files on that mount so that vyrn-admin trace dump can read them after a crash (11 §3.8).

Budget layout of a REF sequencer (sizes are reservations; 05 and 04 own the contents):

01 §14.6 settles retention. Per-session retention, RESP_RETAIN_PKTS × MAX_SESSIONS_PER_LANE × 49 lanes ≈ 13.2 billion response packets (~527 GB to ~3.7 TB), does not fit in a 768 GB box, so the sequencer keeps a 4 GiB byte ring per lane: ~8 s of every session's responses at ~0.5 GB/s per lane (190 Gbps over 49 lanes, budget at full load). The recovery service keeps the per-session window on a replica's drives (§2.2).

Each hot core touches at most PLT_TLB_PAGES_MAX = 16 distinct 1 GiB pages in steady state (budget), so its working set stays within the L2 TLB's 1 GiB entries. The layout tool checks this from the core map.

5.2 NUMA placement#

Cross-node traffic on AWS is a defect: vyrn-admin reads numastat -p for the role process each minute and alarms if any hot region has pages on the other node.

5.3 Start-up order, prefaulting and seal()#

ASCII original

vyrn-seq start (HK)
  │ 1. parse config, read topology (/sys cpu cache ids, numa nodes, PCI numa_node), check core map
  │ 2. mlockall(MCL_CURRENT | MCL_FUTURE); prctl(PR_SET_THP_DISABLE); prctl(PR_SET_DUMPABLE, 0)
  │ 3. spawn one thread per core-map entry; each pins itself with sched_setaffinity, then:
  ├──► per-core setup (on its own core, in parallel)
  │      a. mmap its regions (1 GiB pages), mbind on AWS
  │      b. touch every 4 KiB of every region with a write of 0  (first touch = fault now)
  │      c. touch its stack to PLT_STACK_TOUCH = 256 KiB
  │      d. open its NIC queues via vyrn-nic (backend open happened once, on HK, before 3)
  │      e. build its structures in the regions; warm its caches by one pass over them
  │      f. report READY
  │ 4. control core: verify VmLck, HugetlbPages, per-thread minflt/majflt baselines
  │ 5. each core: seal() — allocator trap on (11 §3.5), seccomp filter on (§5.4)
  │ 6. sd_notify(READY=1)
  ▼
 steady state: per-thread minflt and majflt must not change (checked every 10 s, §13.2)

Prefaulting in parallel on each owner core matters twice: zeroing 640 GiB from one core takes minutes, and on AWS first touch from the owner places the page on the right node.

The binary is linked with -Wl,-z,now,-z,relro (a request to 11 §4.3), so no lazy PLT binding runs after start, and mlockall locks the text pages.

5.4 The hot-thread seccomp filter#

Decision. Every HOT thread installs a seccomp filter as the last step of seal(). Any syscall outside the allowlist kills the whole process (SECCOMP_RET_KILL_PROCESS).

Allowlist: write with fd = 2 (the abort message), rt_sigprocmask, rt_sigaction, rt_sigreturn, tgkill, getpid, gettid, exit, exit_group. These are exactly what abort() and the panic hook (11 §3.8) need.

• A seccomp filter runs only on syscall entry. A thread that makes no syscalls pays nothing.
• A hot thread that makes a syscall has a bug that also breaks the latency SLO. Killing the process turns it into a failover with a published outage (FAILOVER_TARGET_MS), which is better than a sequencer that runs slow without saying so. This matches 11 §3.5 for allocations.
• The afxdp backend adds sendto on its XSK file descriptor to the allowlist of TX threads (§6.5). The kernel backend installs no filter: it is never a production sequencer backend.
• WARM threads get a different, process-wide filter: munmap, mprotect, mremap and madvise with MADV_DONTNEED/MADV_FREE return EPERM and increment plt_tlb_syscall_denied_total (§4.9 rule 1).

5.5 Cache-line size per target#

vyrn-nic::sys reads the line size at start (sysconf(_SC_LEVEL1_DCACHE_LINESIZE) on Linux, sysctlbyname("hw.cachelinesize") on macOS) and refuses to start if it is larger than the compiled CACHE_LINE. The 128-byte pad for lines that one core writes and another reads is 11's decision; the reason is the adjacent-line prefetcher on Zen and Intel cores, and 11's B7 settles it.

6. NIC backends#

6.1 The trait#

Decision. vyrn-wire defines the per-queue traits RxRing and TxRing (11 §1.3), so the engine never depends on a NIC. vyrn-nic defines one backend trait, NicBackend, whose associated types implement them. The binary picks the backend with one match at start and runs a monomorphised engine for it. No dyn exists on a hot path.

// vyrn-wire::ring — what a lane, the cut core or a TX core sees.
pub const RX_BURST: usize = 16;                       // PLT_RX_BURST

/// One received frame. Same layout as `vyns_rx` in shim.h (asserted below).
#[repr(C)]
#[derive(Clone, Copy)]
pub struct RxPkt {
    data: NonNull<u8>,     // first byte of the Ethernet frame; valid until released
    pub len: u16,          // frame length without FCS
    pub flags: RxFlags,    // TS | TS_SYNC | BAD_CSUM | TRUNC
    pub buf_id: u32,
    pub rx_ts: Nanos,      // TAI ns; source given by RxRing::ts_source()
}
const _: () = assert!(size_of::<RxPkt>() == 24 && offset_of!(RxPkt, rx_ts) == 16);

pub trait RxRing: Send {
    /// Never blocks. Writes up to RX_BURST frames, returns how many. No allocation, no lock,
    /// no syscall (the `kernel` backend excepted).
    fn poll(&mut self, out: &mut [RxPkt; RX_BURST]) -> usize;
    /// Gives back the oldest `n` frames not yet released, in poll order.
    fn release(&mut self, n: usize);
    /// Borrow a frame's bytes. The borrow ends before the next `release`.
    fn bytes<'a>(&'a self, p: &'a RxPkt) -> &'a [u8];
    fn ts_source(&self) -> TsSource;
    fn counters(&self) -> &NicCounters;                // single writer: the polling thread
}

pub trait TxRing: Send {
    /// Sends one complete frame (headers built by vyrn-wire). The bytes are copied or written to
    /// the NIC before it returns. `Err(Full)` means no descriptor is free; the caller reaps.
    fn send(&mut self, frame: &[u8]) -> Result<(), TxFull>;
    /// Processes TX completions; returns how many. Called every PLT_TX_REAP_EVERY sends.
    fn reap(&mut self) -> usize;
    fn counters(&self) -> &NicCounters;
}

#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum TsSource { NicPhc, TscAtPoll, Kernel, Virtual }

// vyrn-nic — what the binary sees at start.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum BackendKind { Efvi, DpdkMlx5, DpdkEna, AfXdp, Kernel, Sim }

#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Steering {
    Exact,            // the NIC delivers lane L's packets only to L's queue
    HashThenForward,  // the NIC hashes; a lane forwards foreign packets to their owner (§6.9)
    Socket,           // the kernel delivers by UDP port
}

pub struct Caps {
    pub steering: Steering,
    pub ts: TsSource,
    pub pio_tx: bool,          // CTPIO-class direct write to the wire
    pub max_rx_queues: u16,
    pub latency_class: LatencyClass,   // §6.8
}

pub struct RxBind { pub lane: LaneId, pub port: PortId, pub ip: Ipv4Addr, pub udp_port: u16 }
#[derive(Clone, Copy)]
pub enum TxClass { Fill, OrderStream, Control }

pub trait NicBackend: Sized {
    type Rx: RxRing;
    type Tx: TxRing;
    const KIND: BackendKind;
    /// Cold path. Opens every port in `cfg`; all vendor errors come back as NicError.
    fn open(cfg: &NicConfig) -> Result<Self, NicError>;
    fn caps(&self) -> Caps;
    fn rx_ring(&mut self, bind: &RxBind) -> Result<Self::Rx, NicError>;
    fn tx_ring(&mut self, port: PortId, class: TxClass) -> Result<Self::Tx, NicError>;
    /// The clock behind this port's timestamps, for vyrn-timemon (§8).
    fn port_clock(&self, port: PortId) -> Result<PortClock, NicError>;
}

#[derive(Debug)]
pub enum NicError {
    Driver { backend: BackendKind, port: PortId, rc: i32, msg: ShimMsg },  // ShimMsg: [u8; 160]
    Unsupported { backend: BackendKind, what: &'static str },
    Config { field: &'static str, why: &'static str },
    Topology { why: &'static str },    // e.g. NIC on a different NUMA node than its lanes
}

// bin/vyrn-seq — the only dynamic choice.
pub fn run(cfg: &Config) -> Result<(), SeqError> {
    match cfg.nic.backend {
        #[cfg(target_os = "linux")] BackendKind::Efvi => engine::<efvi::Efvi>(cfg),
        #[cfg(target_os = "linux")] BackendKind::DpdkMlx5 | BackendKind::DpdkEna =>
            engine::<dpdk::Dpdk>(cfg),
        #[cfg(target_os = "linux")] BackendKind::AfXdp => engine::<afxdp::AfXdp>(cfg),
        BackendKind::Kernel => engine::<kernel::KernelUdp>(cfg),
        BackendKind::Sim => engine::<sim::SimNic>(cfg),
        #[allow(unreachable_patterns)]
        other => Err(SeqError::BackendNotBuilt(other)),
    }
}

Contract rules that every backend meets and the sim backend's tests check:

1. poll returns frames of one queue in arrival order. rx_ts is non-decreasing within one queue: a backend that sees a smaller hardware stamp than the previous one returns the previous value and increments rx_ts_regress. (A regression means the PHC stepped, which §8.2 forbids; the counter is an alarm.)
2. Hot-path errors never return as Err. They are counters. Two of them are fatal: evq_overflow (events were lost, so frames may be lost silently) and fatal_hw. The lane checks counters().fatal() once per poll and fail-stops (abort) if it is set.
3. send either sends the whole frame or returns Full. It never sends part of a frame.
4. A backend opens its devices on HK before the hot threads start, and every queue is created by its owner thread before seal().

6.2 Fast-path frame format#

Decision. Ethernet II + IPv4 (IHL = 5, no options, DF set) + UDP, no VLAN tag, 42 bytes of headers (PLT_FRAME_HDR_BYTES). UDP checksum 0 on transmit. IPv6 is not on the fast path.

• IPv4 allows a zero UDP checksum, and IPv6 does not. CTPIO writes a complete frame with no checksum offload, so a UDP checksum would cost a software pass over every response. The packet MACs (01 §4) already authenticate every byte that matters.
• The IPv4 header checksum is computed in software over 20 bytes; the TX core keeps a per-session header template and updates only the length and checksum.
• RX parses fixed offsets. A frame with a VLAN tag, IP options, fragmentation or a UDP length that disagrees with the frame length is dropped and counted before any MAC check.
• Destination MAC for responses is the source MAC of the request (the leaf's routed port). Destination MACs for order frames are resolved at start from the kernel neighbour table over netlink, and refreshed by the control core; no ARP runs on a hot core.
• Frame sizes: a request is ≤ 42 + MAX_REQ_PAYLOAD = 234 bytes; a response is ≤ 42 + 16 + 16 × 16 + 8 = 322 bytes. At 10 messages per packet, 1B msgs/s is 100M frames/s in, ~206 Gbps on the wire including preamble, IFG and FCS (budget arithmetic: 258 B × 8 × 10^8).

6.3 efvi: AMD Solarflare X2541 with ef_vi#

Why the qualification is a maximum. The lane computes age = now − rx_ts when it dequeues a packet (01 §14.1). Every RX delay after the stamp is inside age, so a slow RX costs a BUSY reply, never a late response. The TX side is after the last decision: once a response is written to CTPIO, nothing can shed it. So B1 bounds the echo as a maximum, and bounds TX on its own as a maximum too: the admission test predicts the path with the 240 ns TX of 01 §14.2, so a card with RX + TX ≤ 560 but TX of 260 would still make an admitted packet late. The TX bound is measured from the CTPIO write to the first bit on an idle port; queueing behind other frames on the same port is the TX core's budget (Open question 12).

A CTPIO fallback is a late frame. The fallback DMA leaves microseconds after the budget, so it is not a tolerated rate. The design removes the causes. Underrun cannot happen: the TX core builds the frame in L1 and writes it in one burst of write-combined stores, on a core that takes no interrupt (§4.11). Contention cannot happen: each port has one CTPIO writer (§2.3 rule 4), and probes go through that writer. A fallback that still occurs is a fault: it increments plt_ctpio_fallback_total, and one fallback on an ACTIVE box makes it ineligible (§13.2).

The pooled lanes (01 §14.15). The pool is lanes 42–48 on cores 56–62, CCD 7. Their VIs map to the cores as follows:

• All 7 pool VIs are on client card NIC3 (§2.3), one VI per pool lane, each with the exact filter PORT_BASE + L of any lane. A pooled session gets a home pool lane (its UDP port) at session open; 10's lane map balances them. One card means one PhcMap line for the whole pool (§8.4) and no cross-NIC skew between pooled clients.
• An ef_vi VI has one consumer at a time: its event queue read pointer and RX ring are not thread-safe. So each pool VI has a consumer lock, an AtomicU32 (owner core or FREE) on its own 128-byte line. A core takes it with one CAS before it polls and drops it after release refills the ring. The home core polls its own VI first in every iteration.
• The steal path. A pool core whose own VI had no event tries the next sibling VI in a fixed ring order, with a try-CAS from FREE. On success it takes at most PLT_POOL_STEAL_FRAMES = 1 frame, so it holds a sibling's VI for one poll. On failure it moves on; it never spins.
• Everything stays in one CCD. The locks, the account tokens (05), the VIs' ef_vi state and the RX descriptor rings are allocated in the pool's own 1 GiB page and touched only by cores 56–62, so every CAS and every line transfer of the steal path is inside CCD 7's L3 (~20 ns, 01 §14.15). The NIC writes RX events and buffers by DMA, as for any lane. vyrn-admin refuses a core map that puts a pool lane outside one CCD, or a pool VI on another card.
• Deadline admission is unchanged: the core that dequeues a frame computes age from that frame's rx_ts and NIC3's line, whichever core is the VI's home. The pool's account takes are logged as AcctTake records (01 §14.17, 07 §3.6), so replay does not depend on which core stole what.

6.4 dpdk: ConnectX via mlx5, ENA via ena#

6.5 afxdp: any Linux NIC#

6.6 kernel: UDP sockets on Linux and macOS#

6.7 sim: in-process#

The sim backend is a pair of preallocated rings per lane and per TX port, owned by vyrn-sim's virtual network. rx_ts is the simulator's virtual time, so a replay is bit-identical for a seed. The simulator injects loss, duplication, reordering between queues, delay, CTPIO fallbacks, evq_overflow and rx_ts regressions, so that every counter path in §6.1 has a test. It makes no syscall, needs no privilege and runs identically on Linux and macOS. Its latency class is S (virtual time; no latency claim).

6.8 Backends by target and latency class#

vyrn-seq refuses to start as a production sequencer unless its backend's class is A, or C with target = aws. The check is on Caps::latency_class, so a misconfigured box fails at start, not at the first SLO report.

6.9 Steering, the address map, and forwarding#

The address map gives each lane L the pair (ip of its home card, PORT_BASE + L). 04 carries it in session setup. PORT_BASE is 04's constant; this chapter assumes a block of N_LANES consecutive ports.

With Exact steering, a packet reaches only its lane, or, for a pooled lane, the pool (§6.3). With HashThenForward (ENA, AF_XDP without ntuple), a packet can land on lane M ≠ L. Lane M sees it in its poll, reads the destination port, and copies the frame (≤ 234 bytes) with its rx_ts into the SPSC ring fwd[M → L] (PLT_FWD_RING_SLOTS = 1024 entries, 256-byte slots). Lane L drains its forward rings once per poll, after its own queue. The rx_ts is the original one, so ordering is unaffected. The cost is one cross-core copy, ~100 ns (budget, DESIGN §4). A full forward ring drops the frame and counts it; the client resends (04).

Forward rings exist only on HashThenForward targets. With PLT_N_LANES_AWS = 24 lanes (§9.3) that is 24 × 23 = 552 rings. A lane checks each of its 23 inbound rings once per poll at ~2 ns per empty ring (one cached load of the producer index), ~50 ns per iteration (budget). AWS accepts that cost. PLT-B-ENA-RSS tests the alternative in §9.5.

7. vyrn-nic-shim: the C shim#

7.1 Why C, and only this much#

ef_vi and DPDK are C libraries whose fast paths are macros and static inline functions in their headers (ef_eventq_poll, ef_vi_receive_init, rte_eth_rx_burst, rte_pktmbuf_alloc). Rust cannot call a macro, and bindgen cannot bind a static inline function except by generating a C wrapper, which is a shim anyway. So the shim exists to turn those headers into real functions with a stable C ABI. With cross-language LTO those functions inline back into the Rust loop, so the boundary costs nothing at run time (11 §4.4 verifies it in the binary).

The shim holds no policy. It does not parse packets, build headers, steer, retry or log. It translates one vendor call sequence into one function and reports what happened. Everything that decides anything is Rust. Size: shim.h is 84 code lines, shim.c ≤ 215, total ≤ 300 (11 §4.5).

7.2 c/vyrn-nic-shim/shim.h in full#

/* shim.h: the C ABI between vyrn-nic (Rust) and ef_vi / DPDK. ABI version 1.
 * Rules (06 §7, 11 §4.5): no global state, no malloc, no threads, no stack arrays.
 * Every handle lives in memory that vyrn-nic allocates, sized by vyns_*_size() and aligned to
 * VYNS_HANDLE_ALIGN. Cold calls return vyns_rc and write a NUL-terminated message into the
 * caller's err buffer. Hot calls never return an error: they count, and two counters
 * (evq_overflow, fatal_hw) make vyrn-nic fail-stop. */
#ifndef VYRN_NIC_SHIM_H
#define VYRN_NIC_SHIM_H

#include <stdatomic.h>
#include <stddef.h>
#include <stdint.h>

#define VYNS_ABI_VERSION   1u
#define VYNS_HANDLE_ALIGN  128u
#define VYNS_ERR_LEN       160u

typedef int32_t vyns_rc;
#define VYNS_OK              0
#define VYNS_E_ARG          (-1)  /* bad argument; the message names it              */
#define VYNS_E_DRIVER       (-2)  /* vendor call failed; the message has rc and errno */
#define VYNS_E_UNSUPPORTED  (-3)  /* the device or PMD cannot do this                 */
#define VYNS_E_NO_TS        (-4)  /* hardware timestamps asked for but unavailable    */
#define VYNS_E_STATE        (-5)  /* call out of order                                */

#define VYNS_RX_TS         0x0001u  /* ts_ns is a NIC hardware timestamp */
#define VYNS_RX_TS_SYNC    0x0002u  /* the NIC clock reported "in sync"  */
#define VYNS_RX_BAD_CSUM   0x0004u
#define VYNS_RX_TRUNC      0x0008u

typedef struct vyns_rx {            /* 24 bytes; identical to vyrn_wire::RxPkt */
    const uint8_t *data;            /* first byte of the Ethernet frame      */
    uint16_t len;                   /* frame length without FCS              */
    uint16_t flags;                 /* VYNS_RX_*                             */
    uint32_t buf_id;
    uint64_t ts_ns;                 /* PHC time, TAI ns; 0 unless RX_TS      */
} vyns_rx;

/* Single writer: the thread that polls the handle (relaxed load + store, never a locked RMW).
 * vyrn-nic's control thread reads them as AtomicU64. */
typedef struct vyns_counters {
    _Atomic uint64_t rx_pkts, rx_discard, rx_no_ts, rx_trunc;
    _Atomic uint64_t tx_pkts, tx_ctpio, tx_fallback, tx_full;
    _Atomic uint64_t evq_overflow;  /* != 0: events were lost. Fatal.        */
    _Atomic uint64_t fatal_hw;      /* != 0: the device reported an error.   */
} vyns_counters;

uint32_t vyns_abi_version(void);

/* ---- ef_vi: AMD Solarflare X2 ------------------------------------------------------ */
#define VYNS_EFVI_RX_TS   0x1u
#define VYNS_EFVI_TX_TS   0x2u
#define VYNS_EFVI_CTPIO   0x4u

typedef struct vyns_efvi vyns_efvi;           /* opaque; storage owned by vyrn-nic */

typedef struct vyns_efvi_cfg {
    const char *ifname;             /* kernel netdev of the port                  */
    uint32_t flags;                 /* VYNS_EFVI_*                                */
    uint32_t rxq_entries;           /* power of two; 0 = TX-only VI               */
    uint32_t txq_entries;           /* power of two; 0 = RX-only VI               */
    uint32_t evq_entries;           /* >= rxq_entries + txq_entries               */
    uint32_t buf_bytes;             /* per packet buffer: 2048                    */
    uint32_t ctpio_ct_bytes;        /* cut-through threshold: 64                  */
    uint8_t *buf_mem;               /* huge-page memory for buffers, 4 KiB aligned */
    size_t buf_mem_len;             /* >= (rxq_entries + txq_entries) * buf_bytes */
} vyns_efvi_cfg;

size_t   vyns_efvi_size(void);
vyns_rc  vyns_efvi_open(vyns_efvi *h, const vyns_efvi_cfg *cfg, char *err, size_t err_len);
vyns_rc  vyns_efvi_filter_udp(vyns_efvi *h, uint32_t ip_be, uint16_t port_be,
                              char *err, size_t err_len);
vyns_rc  vyns_efvi_close(vyns_efvi *h, char *err, size_t err_len);
uint32_t vyns_efvi_rx_poll(vyns_efvi *h, vyns_rx *out, uint32_t max);              /* hot */
void     vyns_efvi_rx_release(vyns_efvi *h, uint32_t n);                           /* hot */
int32_t  vyns_efvi_tx_ctpio(vyns_efvi *h, const uint8_t *frame, uint32_t len);     /* hot: 0 sent, 1 full */
uint32_t vyns_efvi_tx_reap(vyns_efvi *h, uint64_t *tx_ts_ns, uint32_t max);       /* hot */
const vyns_counters *vyns_efvi_counters(const vyns_efvi *h);

/* ---- DPDK: mlx5, ena ----------------------------------------------------------------- */
#define VYNS_DPDK_RX_TS    0x1u     /* RTE_ETH_RX_OFFLOAD_TIMESTAMP              */
#define VYNS_DPDK_ISOLATE  0x2u     /* rte_flow isolated mode (mlx5)             */
#define VYNS_DPDK_MAX_Q    64u

typedef struct vyns_dpdk vyns_dpdk;

typedef struct vyns_dpdk_cfg {
    uint16_t port_id;
    uint16_t n_rxq, n_txq;          /* each <= VYNS_DPDK_MAX_Q                   */
    uint16_t rxq_entries, txq_entries;
    uint32_t mbufs_per_q;
    uint32_t flags;                 /* VYNS_DPDK_*                               */
    int32_t socket_id;              /* NUMA node of the port                     */
} vyns_dpdk_cfg;

vyns_rc  vyns_dpdk_eal_init(int argc, char **argv, char *err, size_t err_len);
vyns_rc  vyns_dpdk_thread_register(char *err, size_t err_len);
size_t   vyns_dpdk_size(void);
vyns_rc  vyns_dpdk_open(vyns_dpdk *h, const vyns_dpdk_cfg *cfg, char *err, size_t err_len);
vyns_rc  vyns_dpdk_flow_udp(vyns_dpdk *h, uint32_t ip_be, uint16_t port_be, uint16_t queue,
                            char *err, size_t err_len);
vyns_rc  vyns_dpdk_read_clock(vyns_dpdk *h, uint64_t *dev_ns, char *err, size_t err_len);
vyns_rc  vyns_dpdk_close(vyns_dpdk *h, char *err, size_t err_len);
uint32_t vyns_dpdk_rx_poll(vyns_dpdk *h, uint16_t q, vyns_rx *out, uint32_t max);  /* hot */
void     vyns_dpdk_rx_release(vyns_dpdk *h, uint16_t q, uint32_t n);               /* hot */
int32_t  vyns_dpdk_tx(vyns_dpdk *h, uint16_t q, const uint8_t *frame, uint32_t len); /* hot: 0 sent, 1 full */
const vyns_counters *vyns_dpdk_counters(const vyns_dpdk *h, uint16_t q);

#endif /* VYRN_NIC_SHIM_H */

7.3 What shim.c does, function by function#

The ops-table question (11 §4.4). ef_eventq_poll and the ef_vi transmit calls dispatch through the VI's ops table, and rte_eth_rx_burst/rte_eth_tx_burst dispatch through rte_eth_fp_ops. LTO cannot inline through those pointers. Decision: keep the indirect calls. Calling the generation-specific functions directly would bind the shim to private symbols of one Onload or DPDK release. The call target never changes after open, so the branch predictor gets it right every time; its cost is inside B1's measured echo. 11's asm gate allowlists exactly these four indirect calls by name.

7.4 Build and link#

11 §4 owns the flags and the CI gates. The platform-relevant points:

ASCII original

c/vyrn-nic-shim/shim.c─clang -flto=thin -march=x86-64-v3──► shim.bc ─┐
vendored libciul (ef_vi), built with clang -flto=thin ────► .bc  ────┤
vendored DPDK static libs + mlx5/ena PMDs (meson, clang, b_lto) ──────┤
rustc -C linker-plugin-lto -C target-cpu=x86-64-v3 ───────► .rlib ────┴─► lld + LLVM LTO ─► vyrn-seq

• ef_vi's library (libciul) and DPDK are built from the vendored sources with the same clang, so that their non-inline functions also take part in LTO. DPDK's PMDs register themselves with constructors, so they are linked with --whole-archive.
• rdma-core is not linked: the mlx5 PMD is built to dlopen it, so the one binary starts on a box with no ConnectX and no rdma-core.
• The shim is compiled only for target_os = "linux". On macOS, vyrn-nic builds the kernel and sim backends only, and the build script does not look for clang.

7.5 Error handling#

• Cold calls (open, filter, flow, close, eal_init) return a vyns_rc and write a message of at most VYNS_ERR_LEN bytes: the vendor function's name, its return code, errno and strerror text, and the argument that failed. vyrn-nic turns it into NicError::Driver. vyrn-seq prints it with the port and the lane and exits non-zero before READY. Nothing is retried in C.
• Hot calls return counts. Errors become counters in vyns_counters, and two of them are fatal (evq_overflow, fatal_hw): the Rust caller checks them once per poll and aborts, which is a failover (11 §3.8). A hot call never writes to err, never touches errno and never logs.
• ABI drift: vyrn-nic calls vyns_abi_version() at start and refuses a mismatch. A compile-time assertion in Rust checks size_of::<RxPkt>() == 24 against the C layout.

8. Time#

8.1 What time is for#

The engine reads one clock: the RX timestamp of each packet, which enters the PacketRecord (01 §11.3). It matters in one place: the level FIFO orders lane FIFOs by (rx_ts, lane id). Two lanes on two different cards stamp with two different PHCs, so the skew between the PHCs of one box is a fairness error in time priority. Determinism does not depend on it, because replicas replay the logged rx_ts. Absolute time (TAI) matters only for reports and market data.

ASCII original

        GNSS                                             facility
          │
   GM-1 ──┤ leaf A (PTP BC) ── NIC0 PHC ◄── ptp4l (client, hardware stamps)
   GM-2 ──┘ leaf B (PTP BC)      │
                                 ├──► phc2sys ──► NIC1 PHC
                                 ├──► phc2sys ──► NIC2 PHC
                                 ├──► phc2sys ──► NIC3 PHC
                                 ├──► phc2sys ──► NIC4, NIC5 PHC (replication; rx_ts unused)
                                 └──► phc2sys ──► CLOCK_REALTIME / CLOCK_TAI ──► TAI map (§8.4)
 vyrn-timemon: two-way timestamped probes NIC0 ⇄ NICk through the leaves → measured skew
 vyrn-timemon: PTM cross-timestamp TSC ⇄ PHC of each client card, 100 Hz → TSC↔PHC lines (§8.4)

8.2 ptp4l and phc2sys#

/etc/vyrn/ptp4l.conf:

[global]
slaveOnly               1
domainNumber            0
network_transport       L2
delay_mechanism         E2E
time_stamping           hardware
logSyncInterval         -4          # 16 Sync/s from the boundary clock
logMinDelayReqInterval  -4
tx_timestamp_timeout    10
clock_servo             pi
first_step_threshold    0.00002     # may step once, at start, by > 20 µs
step_threshold          0.0         # never step afterwards: slew only
summary_interval        0
uds_address             /run/ptp4l
[enp1s0np0]                         # NIC 0's kernel netdev (from the platform config)

phc2sys, one instance per sink: phc2sys -s <nic0> -c <nicK> -O 0 -R 16 -n 0 for K = 1…5 (both sides are PHCs in TAI, so the offset is 0), and phc2sys -s <nic0> -c CLOCK_REALTIME -w -R 16 for the system clock, which also sets the kernel's TAI offset from ptp4l's currentUtcOffset. phc2sys's default step threshold is off, so the NIC PHCs only slew. On the observer, the tap ports have no TX fibre, so ptp4l runs on the X2541 that takes path A (§2.3 NIC3) and phc2sys slaves the other five cards to it; a CX-6 Dx netdev that reports the same PHC index as its sibling port gets no phc2sys of its own (checked as in §14.2).

No PHC steps while ACTIVE (INV-PLT-2). A step would move rx_ts backwards on one card and reorder that card's lanes against the others. ptp4l and phc2sys may step only before vyrn-time.target is reached. If a step is ever needed later (offset > 1 ms), vyrn-timemon declares UNSYNC (§8.6) and the box drains before the step.

8.3 Cross-NIC sync: budget and monitoring#

Why 25 ns: resting orders keep exact time priority among themselves (DESIGN §3). 25 ns is about one packet time on a 100G port (258 bytes on the wire = 20.6 ns), so two orders on two cards can be misordered only if they arrive within about one packet time of each other. The cut already limits passive-against-aggressor order to ~100 ns (CUT_REFRESH_NS), so a tighter budget buys nothing there. PLT-B-SKEW measures what phc2sys achieves on the X2541. Skew is a fairness budget, not a latency one: deadline admission maps each lane's now to its own card's PHC (§8.4), so skew never enters age.

phc2sys measures each PHC through PCIe reads, and its own offset report cannot see a constant asymmetry in those reads. So the skew is measured independently, on the wire:

• vyrn-timemon owns one RX-only ef_vi VI per client port (hardware stamps, a filter on PLT_PROBE_UDP_PORT). It sends probes through the port's fill TX core, which writes them with its own CTPIO VI when no response is pending and returns the TX stamp (§6.3: one writer per port).
• Ten times a second, for every pair (0, k), it sends a probe 0 → k and one k → 0 through the leaf switches in the strict-priority queue, with TX stamps t1, t3 and RX stamps t2, t4.
• skew(0,k) = ((t2 − t1) − (t4 − t3)) / 2, as in PTP. Path asymmetry inside a cut-through leaf at one port speed is a few ns (budget ≤ 5 ns, PLT-B-SKEW calibrates it once per box with a direct cable between two cards).
• It exports plt_nic_skew_ns{pair} and feeds §8.6.

If PLT-B-SKEW shows that phc2sys cannot hold 25 ns on the X2541, vyrn-timemon becomes the servo: it steers NIC k's frequency with clock_adjtime(ADJ_FREQUENCY) on /dev/ptpK from the probe measurement, and the three NIC phc2sys units are disabled. The probe stays the monitor either way.

8.4 rx_ts per backend, and the TSC↔PHC map for deadline admission#

Why a map. Deadline admission (01 §14.1) needs age = now − rx_ts in ns when a lane dequeues a packet, and the BBO core needs it again for forwarded ops. rx_ts is in the PHC time of the card that received the packet. A hot core cannot read a PHC: an MMIO read is a ~1 µs PCIe round trip. It reads the TSC and converts.

The line. One seqlock line per client card K, written only by vyrn-timemon on the HOUSE core:

#[repr(C, align(128))]
pub struct PhcMap {               // one per client card; read by its lanes and by the BBO core
    seq: AtomicU64,               // odd while being written
    tsc0: u64, phc0: Nanos,       // one cross-timestamp: TSC and PHC at the same instant
    mult: u64, shift: u32,        // PHC ns per TSC tick, from the last two cross-timestamps
    err0_ns: u32,                 // error bound at tsc0 (below)
    drift_ns_per_ms: u32,         // growth of the bound with the line's age
}
// On the lane, at dequeue (integer only, ~3 ns with the line in L1):
//   d   = rdtsc() - tsc0
//   now = phc0 + ((d * mult) >> shift)
//   age = now - rx_ts + err0_ns + ticks_to_ms(d) * drift_ns_per_ms
//         + PLT_RX_STAMP_OFFSET_NS + PLT_RX_STAMP_CAL_NS

The lane adds the error bound to age. A mapping error therefore costs BUSY replies and never a late response, as long as the bound is true. The BBO core reads the line of the op's card from the static lane → card map. The line changes every 10 ms, so it stays in each reader's cache; the cost is one cross-CCD miss per reader per refresh.

Cross-timestamping. Every PLT_TSC_MAP_REFRESH_MS = 10 ms, vyrn-timemon calls ioctl(PTP_SYS_OFFSET_PRECISE) on /dev/ptpK. The driver's getcrosststamp uses PCIe PTM, so the device time and the system counter (the TSC) are latched at the same instant, with no PCIe read window in the error. vyrn-timemon converts the returned system time back to TSC ticks with the kernel clocksource's own mult/shift (exact to 1 ns), sets mult/shift from the last two samples, and writes the line. The equivalent where PRECISE is absent, PTP_SYS_OFFSET_EXTENDED, brackets the PHC read between two system reads; its error is half the PCIe read window, ~250–500 ns on Gen3 (budget), so it cannot qualify a sequencer card. Its only use is the lab cross-check below. A client NIC whose driver has no PTM cross-timestamp on this platform fails PLT-V-11.

Error budget (PLT_TSC_MAP_ERR_NS = 10 ns, budget):

The bound is measured, not assumed. Each new cross-timestamp is also a check of the previous line: residual = phc_observed − phc_predicted(tsc_observed). err0_ns is the PTM term plus the largest |residual| of the last 16 refreshes (160 ms), and drift_ns_per_ms is the largest |residual| per ms of line age in that window. So a servo that starts to move the PHC rate widens the bound at once.

Refresh and failure.

The threshold is the guard because the budgets of 01 §14.2 and §14.17 leave ~15 ns of slack at message index 9 against the 980 ns shed threshold. A bound above 20 ns turns most worst-case packets into BUSY, which B3 forbids, and means the cross-timestamp itself is no longer trustworthy.

PLT_RX_STAMP_OFFSET_NS. The X2541 stamps a frame at its MAC, a fixed time after the first bit reaches the port. The B1 tap rig measures that offset per card once (§14.2), and the lane adds it. It also adds the calibration error, ≤ PLT_RX_STAMP_CAL_NS = 10 ns, to age, because DEADLINE_GUARD_NS = 20 is all gap budget now (01 §14.1, §4.11).

AWS: rx_ts is the TSC at RX completion (01 §14.11). The ENA has no hardware stamp we rely on, so the polling core stamps with the TSC. The stamp must not be later than the completion, or age would under-state the time a frame waited in the ring. So rx_ts is the TSC of the most recent poll of that queue that drained it (returned fewer than RX_BURST frames). Every frame returned after that poll completed after it, so rx_ts ≤ the true completion time. On an idle queue the stamp is at most one poll iteration early (~50 ns, budget). Under a backlog the stamp stays old, age grows, and admission answers BUSY, which is the right answer for a lane that is behind. age = rdtsc() − rx_ts uses one TSC (invariant, synchronised across the cores of the ENA socket, which holds every hot core, §9.3), so the AWS age has no mapping error at all. The Nitro card's time before the completion is not ours, as 01 §14.0 states. A forwarded frame (§6.9) keeps its original stamp.

The TAI map. Logs and market data need TAI, not PHC. A second seqlock line (tsc0, tai0, mult, shift) is refreshed at 10 Hz from paired reads of CLOCK_TAI and rdtsc. It converts TscAtPoll stamps to TAI for the PacketRecord; it is never used for admission. A box whose rx_ts source is TscAtPoll says so in its health beacon, and 02 reports its fairness bound separately.

8.5 AWS and the consumer box#

AWS. Time comes from the Amazon Time Sync Service through the ENA PHC on the management ENI (kernel ena driver, PHC enabled), read by chrony with refclock PHC /dev/ptp0 poll 0 delay 0.000010 prefer (vendor configuration). chrony owns CLOCK_REALTIME; the TSC map follows it. The ENA PHC's accuracy is the vendor's, in the microsecond range, reported by chrony as the clock error bound. Since all AWS rx_ts come from one TSC (§8.4), that error moves every stamp together and never reorders two lanes.

CONSUMER. In a facility, the X2541 runs ptp4l exactly as NIC 0 on REF, and it has one TSC↔PHC line (§8.4); the AM5 root port must offer PTM, which PLT-V-11 checks. There is one client card, so there is no cross-NIC skew: the X2522 carries only order frames, whose rx_ts is never used. In a lab with no grandmaster, the X2541 PHC free-runs from a one-time NTP set (phc2sys -s CLOCK_REALTIME -c <nic> once at boot, then stopped) and the time state is FREERUN.

8.6 Time state and eligibility#

ASCII original

             ptp4l s2 and |GM offset| < 100 ns for 60 s and skew < 50 ns
   UNSYNC ──────────────────────────────────────────────────────────────► LOCKED
     ▲  ▲                                                                   │  ▲
     │  │ holdover > PLT_HOLDOVER_MAX_S (300 s)        GM lost / ptp4l ≠ s2 │  │ GM back,
     │  └──────────────────────────── HOLDOVER ◄────────────────────────────┘  │ offset ok
     │                                   └─────────────────────────────────────┘
     │ skew > 100 ns for 1 s (from any state)
   SKEWED ── skew back < 50 ns for 60 s ──► UNSYNC
     │ any client card: map err0 > 20 ns or line > 50 ms old (from any state)
   MAPERR ── every line err0 < 10 ns for 60 s ──► UNSYNC
   FREERUN: lab and standalone CONSUMER only; never in a facility

Replicas replay logged rx_ts and never read a clock for the engine, so their time state does not affect replay or voting. Eligibility applies to sequencer-class boxes: it decides whether a standby may be promoted (01 §14.4, 07) or a box scheduled as the next home's sequencer (08). vyrn-timemon publishes the state in the health beacon (§13.2).

9. AWS#

9.1 Instance class#

Decision. c6in.metal for the sequencer, data and market-data roles; m6idn.metal for replicas, which need local NVMe. Both are x86-64, rated 200 Gbps (vendor), and run the REF binary unchanged. c7gn.metal (Graviton, 200 Gbps, one socket) is the aarch64 alternative; it needs the aarch64 build and is not the default.

Rule for any future instance type: the largest .metal type rated ≥ 200 Gbps whose ENA devices can all be bound from one NUMA node. PLT-V-AWS-1 checks /sys/bus/pci/devices/*/numa_node for every ENA. If an instance's network cards sit on two nodes, vyrn-seq binds only the devices of one node and the box's bandwidth is that node's share. We do not trade one-node placement for bandwidth: on AWS, 1B msgs/s depends on the instance's packet rate (01 §14.0, B16, §9.4), and cross-node placement would put a remote hop inside the 1 µs.

9.2 DPDK on ENA#

• One management ENI stays on the kernel ena driver: SSH, chrony with the ENA PHC (§8.5), consensus traffic of replicas (07), CloudWatch agent. Every data ENI is bound to vfio-pci.
• LLQ needs write-combining on the device's memory BAR. vfio-pci is used with write-combining enabled (the amzn-drivers project ships the procedure). PLT-V-AWS-2 fails a box whose ENA PMD logs that LLQ is disabled, because without LLQ every TX descriptor costs a DMA fetch.
• Queue count per ENI is read from rte_eth_dev_info_get at start; lanes are spread over ENIs so that each lane owns one RX queue.
• Security groups allow the client and replica flows with rules broad enough that the flows are untracked, so connection-tracking allowances never apply (vendor behaviour, checked by conntrack_allowance_exceeded staying 0).

9.3 Core map on c6in.metal#

One socket's 32 cores (vendor: 64 cores in 2 sockets, SMT off with nosmt) on the ENA node:

The other socket runs consensus, catch-up, admin and metrics (WARM/HOUSE). 24 lanes at the REF lane budget give ~480M msgs/s of lane compute (budget, before Ice Lake's lower clock); the packet-rate allowance is the real limit (§9.4).

9.4 Packet-rate limits#

AWS does not publish a packets-per-second limit per instance. The ENA reports when one is hit: pps_allowance_exceeded, bw_in_allowance_exceeded, bw_out_allowance_exceeded, conntrack_allowance_exceeded, linklocal_allowance_exceeded (vendor counters, read through the PMD's xstats or the kernel driver's ethtool -S). Per-flow bandwidth is limited too: 5 Gbps per flow outside a cluster placement group, 10 Gbps inside (vendor). Clients therefore spread sessions over many source ports.

PLT-B-AWS-PPS (B16, to be measured): 8 client instances of the same type in one cluster placement group drive 234-byte UDP frames at the sequencer, stepping the rate by 1M frames/s every 60 s, while the sequencer echoes 322-byte responses. The result is the highest step with zero loss and no increment of any allowance counter, in and out separately, repeated in three regions. That number, times 10 messages per packet, is the AWS book's published ceiling. No target accepts more than MAX_MSGS_PER_PKT messages per packet: a larger packet would raise the ceiling only by exempting itself from LAT_MAX_NS, which 01 §14.0 forbids. The ceiling is stated as the AWS fact of 01 §14.0, not as an exemption.

9.5 RSS and client-chosen source ports#

ENA hashes with Toeplitz over the 5-tuple. If the instance lets vyrn-nic set the RSS key and indirection table (checked at start with rte_eth_dev_rss_hash_update), the server can publish them, and a client can pick, per lane, a source port whose hash lands on that lane's queue. Then forwarding (§6.9) becomes the rare case instead of the common one. PLT-B-ENA-RSS measures the mis-steer rate with and without it. The protocol field belongs to 04 (Open questions).

9.6 Replicas across AZs, and what AWS cannot do#

Facilities map to Availability Zones: each replica in its own AZ, the sequencer and its clients in one AZ in a cluster placement group. A region with fewer AZs than N_BASE (or N_CHAIN) has to put two boxes in one AZ; an outage of that AZ then removes two votes and halts finality (safety holds, liveness does not). That limit is published per region.

Cross-AZ transfer costs $0.01/GB in each direction (vendor). Each replica receives ~26 GB/s at sustained peak (DESIGN §7): 26 GB/s × 86,400 s × $0.02 ≈ $45k per replica per day at peak, and ~$4.5k at 10% average load. This is why REF is colo.

What AWS cannot do, and what the spec says instead:

10. The consumer box#

11 lanes and ~220M msgs/s per book: 11 × 33M × 0.6 ≈ 218M (budget). 01 §5 now has 11 lanes, but 01 §10 still says ~250M; the order hasher needs its own core, and the 16 cores hold only 11 lanes beside it (Requests to 01, item 2). The BBO core has the REF budget, so the cross cap is unchanged; the lanes are the limit.

Core map. vyrn-admin finds the V-cache CCD at start from /sys/devices/system/cpu/cpuN/cache/index3/size (the CCD with the 96 MB L3) and refuses to start if the map below would put the BBO, cut or TX core on the other CCD. Shown with V-cache on CCD 0:

The hasher is not merged into core 0, the cut core or the TX core. On core 0, a housekeeping stall would delay a block's roots and so its finality. On the cut or TX core, a BLAKE3 slice would sit on the ack or fill path.

Merging the two TX roles keeps INV-FRAME-FIRST trivially: one core writes the order frame, then the fill, in program order. It is also the only CTPIO writer on the X2541 port, as §6.3 needs. At ~220M msgs/s the TX core sends ~22M responses/s plus ~5 Gbps of order frames, inside one core's budget (PLT-B-X3D checks it).

Why the V-cache CCD: the BBO core reads level heads in every lane's FIFO and the filled-quantity array, and the cut core reads 11 lane counters. Those lines live in the BBO core's L3 when it is 96 MB. The V-cache CCD clocks a little lower than the other (vendor); PLT-B-X3D runs B3 with the BBO, cut and TX cores on each CCD and keeps the faster one. BIOS: §3.2 (PBO off). NICs: one X2541 for all client traffic and path A, one X2522 for path B (§2.6).

11. macOS for development#

Rule. cargo test --workspace passes on macOS with default features, and no test needs a Linux feature to run. Tests that need Linux (efvi, dpdk, afxdp, isolation, seccomp, huge pages) are behind #[cfg(target_os = "linux")] in vyrn-nic and vyrn-bench, and CI runs them on Linux runners.

CACHE_LINE is 128 on Apple M-series (§5.5); the shared-line pad is 128 everywhere, so layouts are identical.

12. Host security#

13. Operating procedures#

13.1 Boot#

POWER_ON → FIRMWARE (BIOS effects, §14 items 1–4) → KERNEL (cmdline, sysctls, vyrn-tune)
 → TIME (vyrn-time.target: LOCKED or HOLDOVER) → NIC (open, self-test frame per port)
 → QUIET (60 s quiet probe on all isolated cores) → REPLICA_READY (may replay and vote)
 → ELIGIBLE (may be promoted or scheduled to sequence) → ACTIVE (sequencing) / DRAINING → DRAINED

A failed step stops the boot at that state and vyrn-admin reports which check failed and its value. A box enters ELIGIBLE only from REPLICA_READY with time state LOCKED or HOLDOVER, a passing quiet probe, a passing B1-mini (10^6 echoes per client port through the leaf, timed by the box's own RX and TX hardware stamps: maximum RX + TX ≤ NIC_FLOOR_NS, maximum TX ≤ PLT_TX_MAX_NS, zero CTPIO fallbacks), every TSC↔PHC line below PLT_TSC_MAP_ERR_NS, and a platform report less than 30 days old (INV-PLT-5).

13.2 Health checks while running#

vyrn-admin publishes a signed health beacon every 100 ms on the control plane. 07 uses it to pick failover targets and 08 to pick the next home's sequencer.

13.3 Drain#

1. vyrn-admin drain sets DRAINING and the beacon says so.
2. If the box is ACTIVE, it asks 07 for a planned handover at the next block boundary; the home's warm standby vyrn-seq takes over (01 §14.4) and this box becomes the standby.
3. If the box is a replica, it keeps voting until the facility has its full replica set healthy elsewhere. In base mode a drained replica leaves N_BASE − 1 = Q_BASE votes, so only one box per replica set drains at a time and only when the others are healthy.
4. The role process stops; the box is DRAINED. §4.9's banned operations are now allowed.

13.4 Maintenance windows#

A home is active 8 h of 24 (01 §5). Kernel, firmware and BIOS changes happen on a box whose home is inactive for at least the next 12 h, one box per replica set at a time, each followed by the §14 checklist and B1–B5 before the box returns to ELIGIBLE.

14. Validation checklist#

14.1 Before a box joins a facility#

Each line is a PLT-V check; vyrn-admin validate runs all of them and signs the platform report. Thresholds are budgets.

A box that fails any line does not join. A box re-runs lines 3–7, 10–14 (with 12a) and 17 after any kernel, BIOS or firmware change, and the whole list after any hardware change.

14.2 The B1 tap rig#

Passive optical splitters on the box's RX and TX fibres feed a capture box with one dual-port ConnectX-7 whose two ports share one hardware clock (checked: both netdevs report the same PHC index). It stamps the first bit of each frame in and out of the device under test. ASIC NICs only (§2.9).

The rig gives the total, first bit in to first bit out. The box splits it with its own clocks: RX = the TSC of the poll that saw the frame, mapped to the card's PHC (§8.4), minus rx_ts; TX = the card's TX hardware stamp minus the TSC of the CTPIO write, mapped the same way. The rest of the total, rig minus (tx_ts − rx_ts), is the two MAC stamp offsets; the run fixes PLT_RX_STAMP_OFFSET_NS per card from it, with a calibration error ≤ PLT_RX_STAMP_CAL_NS. B1 reports the maximum of each part and of the total, never a percentile alone.

14.3 B-GPU: the only path that can reopen the GPU ban#

B-GPU is a placeholder lab benchmark (11 assigns its number). It runs a GPU-initiated NIC echo: a GPU kernel polls an RX ring placed in GPU memory and posts the TX descriptor and doorbell itself, with no CPU on the path, through the B1 tap rig, at the B1 workload.

Interfaces to other chapters#

Local definitions#

Constants (budgets unless marked):

Types: BackendKind, Steering, Caps, LatencyClass (A, B, C, D, E, S), TsSource, RxBind, TxClass, NicError (§6.1); PhcMap (§8.4); TimeState (UNSYNC, LOCKED, HOLDOVER, SKEWED, MAPERR, FREERUN); PlatformState (§13.1); CoreClass (HOT, WARM, HOUSE).

Invariants:

Benchmarks and checks defined here (11 assigns B-numbers): PLT-B-PPS (§6.3), PLT-B-SW (§2.4), PLT-B-HOP and PLT-B-MEM (§14 items 8–9), PLT-B-FREQ (§3.1 slider), PLT-B-MIT (§4.4), PLT-B-SKEW (§8.3), PLT-B-XTS (§8.4), PLT-B-AWS-PPS (§9.4, B16), PLT-B-ENA-RSS (§9.5), PLT-B-X3D (§10), PLT-B-POOL (§14 item 9), B-GPU (§14.3), PLT-B-INJ and PLT-B-TSME (Open questions); PLT-V-1…18 and 12a (§14.1), PLT-V-AWS-1…2 (§9).

Requests to 01#

Resolved by 01 §14 and removed: the AWS rx_ts rule (§14.11), the X2522 for CONSUMER path B (§14.7), response retention (§14.6), the observer box (§14.16), the frame-gate wording (§14.16), TX ≤ 240 ns as its own maximum (§14.16), N_LANES_CONSUMER = 11 (§5) and the guard (§14.1: DEADLINE_GUARD_NS = 20, all gap budget). Open:

1. §11 rule 3: "the NIC RX timestamp" becomes "the RX timestamp: the NIC hardware stamp where the backend has one, else the TSC of the last poll that drained the queue, mapped to TAI (06 §8.4)". This makes 01 §14.11's "TSC at RX completion" a lower bound, so age is never under-stated (INV-PLT-6).
2. §10 CONSUMER: ~220M msgs/s, not ~250M, at 11 lanes (06 §10). Add the CONSUMER and AWS core maps (06 §10, §9.3) with one cut, one TX and one stream core each, and N_LANES_AWS = 24.
3. §10 REF row and §14.7: the NIC column becomes per role: sequencer and standby 4 × X2541 client plus 2 × dual-port X2542 replication, one port per stream core (06 §2.3 rule 4; two stream cores on single-port cards would break INV-PLT-7); replica 4 × X2541; observer 3 × CX-6 Dx + 3 × X2541. §14.7's replica drives become 12 × 61.44 TB, because 12 × 30.72 TB holds ~3.9 h at sustained peak, short of §14.16's 6 h (06 §2.2).
4. §5: add PLT_TX_MAX_NS = 240 as NIC_TX_MAX_NS, the name for §14.16's TX maximum.
5. §14.1: say that the RX stamp's calibration error (≤ 10 ns) is added to age like the TSC↔PHC error, now that the guard is gap budget only (06 §4.11, §8.4).
6. §7: promote INV-PLT-2 (no clock step while active) to INV-NOSTEP, INV-PLT-6 (age is never under-stated), INV-PLT-7 (one CTPIO writer per port) and INV-PLT-8 (one pool VI consumer at a time, inside the pool's CCD).
7. §5: add PORT_BASE (value from 04) and SKEW_BUDGET_NS = 25, which 02's fairness requirement cites.
8. §9: list the vyrn-nic backends (efvi, dpdk, afxdp, kernel, sim) and note that the ring traits live in vyrn-wire (11 §1.3) while NicBackend lives in vyrn-nic.
9. §5 and §14.3: N_LANES_REF = 50 disagrees with the core map's 49 lanes (0–48: 42 dedicated, 7 pooled), and §14.3 still says cut B reads lanes 25–49 and the hasher is on core 7. 06 uses 49 lanes and the §10 map.
10. §10: name the fill TX core → client port map: TX-A NIC0, TX-B NIC2, TX-C NIC1 and NIC3 (06 §2.3 rule 3). The map's "TX-A lanes 0–24, TX-B lanes 25–48" cannot hold with three TX cores and one CTPIO writer per port.

Open questions#

Resolved by 01 §14 and removed: response retention (§14.6) and the replica on-box window with the archive node (§14.7).