================================================================================ CONSTELLATION RELAY v2 — VECTORIZED + COLD GATES Input dim: 128 = 8 patches × 16 Device: cuda fp8: hardware ================================================================================ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ TEST 1: Single relay pass — CV preservation + gate state ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Relay params: 19,112 Gate init: sigmoid(-3.0) = 0.0474 Before: CV=0.0577 eff_dim=119.8 After: CV=0.0589 eff_dim=119.1 Normed: CV=0.0588 cos_to_orig: 0.995478 Diagnostics: cos=0.4322 margin=0.1091 drift=0.0005 gate=0.0474 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ TEST 2: Depth stability — relay vs attention vs interleaved ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ A) Relay-only: cycle CV CV_n eff_d cos_orig gate 1 0.0587 0.0590 119.3 0.995716 0.047 2 0.0580 0.0579 119.2 0.995005 0.047 4 0.0582 0.0591 119.2 0.994683 0.047 8 0.0595 0.0573 119.1 0.994514 0.047 12 0.0603 0.0590 119.1 0.994425 0.047 16 0.0583 0.0584 119.1 0.994436 0.047 B) Attention-only: cycle CV CV_n eff_d cos_orig 1 0.0607 0.0597 119.3 0.990663 2 0.0724 0.0712 118.6 0.955266 4 0.0802 0.0799 117.8 0.301391 8 0.0713 0.0705 117.8 0.132908 12 0.0711 0.0671 117.8 0.093587 16 0.0699 0.0683 117.8 0.073860 C) Interleaved (attn → relay → ...): step type CV_n eff_d cos_orig 1 attn 0.0598 119.2 0.990823 2 relay 0.0609 118.4 0.986670 4 relay 0.0605 118.4 0.985883 8 relay 0.0598 118.3 0.985115 12 relay 0.0603 118.3 0.984901 16 relay 0.0626 118.3 0.984822 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ TEST 3: Clustered input — relay vs attention ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ spread CV_before CV_relay CV_attn Δ_relay Δ_attn 0.1 0.1482 0.1540 0.1734 +0.0057 +0.0252 0.3 0.0630 0.0617 0.0815 -0.0013 +0.0185 0.5 0.0574 0.0594 0.0662 +0.0020 +0.0088 1.0 0.0583 0.0569 0.0618 -0.0014 +0.0034 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ TEST 4: Throughput — vectorized relay vs attention ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Relay: 0.87ms (19,112 params) Attention: 0.51ms (65,792 params) Speedup: 0.59× ◄ attn faster ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ TEST 5: Patch count sweep ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ patches d_patch params CV_out cos_orig ms 1 128 8,353 0.0577 0.995592 0.87 2 64 9,794 0.0592 0.995592 0.87 4 32 12,868 0.0586 0.995592 0.87 8 16 19,112 0.0588 0.995591 0.86 16 8 18,336 0.0590 0.995591 0.87 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ TEST 6: Scaling — relay at d=256, 512, 768, 1024 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ D patches params CV_in CV_out cos ms 256 16 38,192 0.0399 0.0402 0.997171 0.88 512 32 76,352 0.0280 0.0283 0.997997 0.87 768 48 114,512 0.0226 0.0233 0.998251 0.88 1024 64 152,672 0.0197 0.0199 0.998352 0.88 ================================================================================ SUMMARY ================================================================================ v2 changes: - Vectorized: all patches in single einsum, no Python loop - Gates init at sigmoid(-3.0) ≈ 0.047 (was 0.622) - Patchwork via batched einsum (P, tri, hidden) × (P, hidden, d) Architecture: 128d → 8p × 16d, 16A/patch, 3-phase Params: 19,112 Key comparison at depth 16: Relay-only cos_to_orig: (see Test 2A, cycle 16) Attention-only cos_to_orig: (see Test 2B, cycle 16) Interleaved cos_to_orig: (see Test 2C, step 16) ================================================================================ DONE ================================================================================