|
| 1 | +# ADR-063: 60 GHz mmWave Sensor Fusion with WiFi CSI |
| 2 | + |
| 3 | +**Status:** Proposed |
| 4 | +**Date:** 2026-03-15 |
| 5 | +**Deciders:** @ruvnet |
| 6 | +**Related:** ADR-014 (SOTA signal processing), ADR-021 (vital sign extraction), ADR-029 (RuvSense multistatic), ADR-039 (edge intelligence), ADR-042 (CHCI coherent sensing) |
| 7 | + |
| 8 | +## Context |
| 9 | + |
| 10 | +RuView currently senses the environment using WiFi CSI — a passive technique that analyzes how WiFi signals are disturbed by human presence and movement. While this works through walls and requires no line of sight, CSI-derived vital signs (breathing rate, heart rate) are inherently noisy because they rely on phase extraction from multipath-rich WiFi channels. |
| 11 | + |
| 12 | +A complementary sensing modality exists: **60 GHz mmWave radar** modules (e.g., Seeed MR60BHA2) that use active FMCW radar at 60 GHz to measure breathing and heart rate with clinical-grade accuracy. These modules are inexpensive (~$15), run on ESP32-C6/C3, and output structured vital signs over UART. |
| 13 | + |
| 14 | +**Live hardware capture (COM4, 2026-03-15)** from a Seeed MR60BHA2 on an ESP32-C6 running ESPHome: |
| 15 | + |
| 16 | +``` |
| 17 | +[D][sensor:093]: 'Real-time respiratory rate': Sending state 22.00000 |
| 18 | +[D][sensor:093]: 'Real-time heart rate': Sending state 92.00000 bpm |
| 19 | +[D][sensor:093]: 'Distance to detection object': Sending state 0.00000 cm |
| 20 | +[D][sensor:093]: 'Target Number': Sending state 0.00000 |
| 21 | +[D][binary_sensor:036]: 'Person Information': Sending state OFF |
| 22 | +[D][sensor:093]: 'Seeed MR60BHA2 Illuminance': Sending state 0.67913 lx |
| 23 | +``` |
| 24 | + |
| 25 | +### The Opportunity |
| 26 | + |
| 27 | +Fusing WiFi CSI with mmWave radar creates a sensor system that is greater than the sum of its parts: |
| 28 | + |
| 29 | +| Capability | WiFi CSI Alone | mmWave Alone | Fused | |
| 30 | +|-----------|---------------|-------------|-------| |
| 31 | +| Through-wall sensing | Yes (5m+) | No (LoS only, ~3m) | Yes — CSI for room-scale, mmWave for precision | |
| 32 | +| Heart rate accuracy | ±5-10 BPM | ±1-2 BPM | ±1-2 BPM (mmWave primary, CSI cross-validates) | |
| 33 | +| Breathing accuracy | ±2-3 BPM | ±0.5 BPM | ±0.5 BPM | |
| 34 | +| Presence detection | Good (adaptive threshold) | Excellent (range-gated) | Excellent + through-wall | |
| 35 | +| Multi-person | Via subcarrier clustering | Via range-Doppler bins | Combined spatial + RF resolution | |
| 36 | +| Fall detection | Phase acceleration | Range/velocity + micro-Doppler | Dual-confirm reduces false positives to near-zero | |
| 37 | +| Pose estimation | Via trained model | Not available | CSI provides pose; mmWave provides ground-truth vitals for training | |
| 38 | +| Coverage | Whole room (passive) | ~120° cone, 3m range | Full room + precision zone | |
| 39 | +| Cost per node | ~$9 (ESP32-S3) | ~$15 (ESP32-C6 + MR60BHA2) | ~$24 combined | |
| 40 | + |
| 41 | +### RuVector Integration Points |
| 42 | + |
| 43 | +The RuVector v2.0.4 stack (already integrated per ADR-016) provides the signal processing backbone: |
| 44 | + |
| 45 | +| RuVector Component | Role in mmWave Fusion | |
| 46 | +|-------------------|----------------------| |
| 47 | +| `ruvector-attention` (`bvp.rs`) | Blood Volume Pulse estimation — mmWave heart rate can calibrate the WiFi CSI BVP phase extraction | |
| 48 | +| `ruvector-temporal-tensor` (`breathing.rs`) | Breathing rate estimation — mmWave provides ground-truth for adaptive filter tuning | |
| 49 | +| `ruvector-solver` (`triangulation.rs`) | Multilateration — mmWave range-gated distance + CSI amplitude = 3D position | |
| 50 | +| `ruvector-attn-mincut` (`spectrogram.rs`) | Time-frequency decomposition — mmWave Doppler complements CSI phase spectrogram | |
| 51 | +| `ruvector-mincut` (`metrics.rs`, DynamicPersonMatcher) | Multi-person association — mmWave target IDs help disambiguate CSI subcarrier clusters | |
| 52 | + |
| 53 | +### RuvSense Integration Points |
| 54 | + |
| 55 | +The RuvSense multistatic sensing pipeline (ADR-029) gains new capabilities: |
| 56 | + |
| 57 | +| RuvSense Module | mmWave Integration | |
| 58 | +|----------------|-------------------| |
| 59 | +| `pose_tracker.rs` (AETHER re-ID) | mmWave distance + velocity as additional re-ID features for Kalman tracker | |
| 60 | +| `longitudinal.rs` (Welford stats) | mmWave vitals as reference signal for CSI drift detection | |
| 61 | +| `intention.rs` (pre-movement) | mmWave micro-Doppler detects pre-movement 100-200ms earlier than CSI | |
| 62 | +| `adversarial.rs` (consistency check) | mmWave provides independent signal to detect CSI spoofing/anomalies | |
| 63 | +| `coherence_gate.rs` | mmWave presence as additional gate input — if mmWave says "no person", CSI coherence gate rejects | |
| 64 | + |
| 65 | +### Cross-Viewpoint Fusion Integration |
| 66 | + |
| 67 | +The viewpoint fusion pipeline (`ruvector/src/viewpoint/`) extends naturally: |
| 68 | + |
| 69 | +| Viewpoint Module | mmWave Extension | |
| 70 | +|-----------------|-----------------| |
| 71 | +| `attention.rs` (CrossViewpointAttention) | mmWave range becomes a new "viewpoint" in the attention mechanism | |
| 72 | +| `geometry.rs` (GeometricDiversityIndex) | mmWave cone geometry contributes to Fisher Information / Cramer-Rao bounds | |
| 73 | +| `coherence.rs` (phase phasor) | mmWave phase coherence as validation for WiFi phasor coherence | |
| 74 | +| `fusion.rs` (MultistaticArray) | mmWave node becomes a member of the multistatic array with its own domain events | |
| 75 | + |
| 76 | +## Decision |
| 77 | + |
| 78 | +Add 60 GHz mmWave radar sensor support to the RuView firmware and sensing pipeline with auto-detection and device-specific capabilities. |
| 79 | + |
| 80 | +### Architecture |
| 81 | + |
| 82 | +``` |
| 83 | +┌─────────────────────────────────────────────────────────┐ |
| 84 | +│ Sensing Node │ |
| 85 | +│ │ |
| 86 | +│ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │ |
| 87 | +│ │ ESP32-S3 │ │ ESP32-C6 │ │ Combined │ │ |
| 88 | +│ │ WiFi CSI │ │ + MR60BHA2 │ │ S3 + UART │ │ |
| 89 | +│ │ (COM7) │ │ 60GHz mmWave │ │ mmWave │ │ |
| 90 | +│ │ │ │ (COM4) │ │ │ │ |
| 91 | +│ │ Passive │ │ Active radar │ │ Both modes │ │ |
| 92 | +│ │ Through-wall │ │ LoS, precise │ │ │ │ |
| 93 | +│ └──────┬───────┘ └──────┬───────┘ └─────┬──────┘ │ |
| 94 | +│ │ │ │ │ |
| 95 | +│ └────────┬───────────┘ │ │ |
| 96 | +│ ▼ │ │ |
| 97 | +│ ┌────────────────┐ │ │ |
| 98 | +│ │ Fusion Engine │◄──────────────────────┘ │ |
| 99 | +│ │ │ │ |
| 100 | +│ │ • Kalman fuse │ Vitals packet (extended): │ |
| 101 | +│ │ • Cross-validate│ magic 0xC5110004 │ |
| 102 | +│ │ • Ground-truth │ + mmwave_hr, mmwave_br │ |
| 103 | +│ │ calibration │ + mmwave_distance │ |
| 104 | +│ │ • Fall confirm │ + mmwave_target_count │ |
| 105 | +│ └────────────────┘ + confidence scores │ |
| 106 | +└─────────────────────────────────────────────────────────┘ |
| 107 | +``` |
| 108 | + |
| 109 | +### Three Deployment Modes |
| 110 | + |
| 111 | +**Mode 1: Standalone CSI (existing)** — ESP32-S3 only, WiFi CSI sensing. |
| 112 | + |
| 113 | +**Mode 2: Standalone mmWave** — ESP32-C6 + MR60BHA2, precise vitals in a single room. |
| 114 | + |
| 115 | +**Mode 3: Fused (recommended)** — ESP32-S3 + mmWave module on UART, or two separate nodes with server-side fusion. |
| 116 | + |
| 117 | +### Auto-Detection Protocol |
| 118 | + |
| 119 | +The firmware will auto-detect connected mmWave modules at boot: |
| 120 | + |
| 121 | +1. **UART probe** — On configured UART pins, send the MR60BHA2 identification command (`0x01 0x01 0x00 0x01 ...`) and check for valid response header |
| 122 | +2. **Protocol detection** — Identify the sensor family: |
| 123 | + - Seeed MR60BHA2 (breathing + heart rate) |
| 124 | + - Seeed MR60FDA1 (fall detection) |
| 125 | + - Seeed MR24HPC1 (presence + light sleep/deep sleep) |
| 126 | + - HLK-LD2410 (presence + distance) |
| 127 | + - HLK-LD2450 (multi-target tracking) |
| 128 | +3. **Capability registration** — Register detected sensor capabilities in the edge config: |
| 129 | + |
| 130 | +```c |
| 131 | +typedef struct { |
| 132 | + uint8_t mmwave_detected; /** 1 if mmWave module found on UART */ |
| 133 | + uint8_t mmwave_type; /** Sensor family (MR60BHA2, MR60FDA1, etc.) */ |
| 134 | + uint8_t mmwave_has_hr; /** Heart rate capability */ |
| 135 | + uint8_t mmwave_has_br; /** Breathing rate capability */ |
| 136 | + uint8_t mmwave_has_fall; /** Fall detection capability */ |
| 137 | + uint8_t mmwave_has_presence; /** Presence detection capability */ |
| 138 | + uint8_t mmwave_has_distance; /** Range measurement capability */ |
| 139 | + uint8_t mmwave_has_tracking; /** Multi-target tracking capability */ |
| 140 | + float mmwave_hr_bpm; /** Latest heart rate from mmWave */ |
| 141 | + float mmwave_br_bpm; /** Latest breathing rate from mmWave */ |
| 142 | + float mmwave_distance_cm; /** Distance to nearest target */ |
| 143 | + uint8_t mmwave_target_count; /** Number of detected targets */ |
| 144 | + bool mmwave_person_present;/** mmWave presence state */ |
| 145 | +} mmwave_state_t; |
| 146 | +``` |
| 147 | + |
| 148 | +### Supported Sensors |
| 149 | + |
| 150 | +| Sensor | Frequency | Capabilities | UART Protocol | Cost | |
| 151 | +|--------|-----------|-------------|---------------|------| |
| 152 | +| **Seeed MR60BHA2** | 60 GHz | HR, BR, presence, illuminance | Seeed proprietary frames | ~$15 | |
| 153 | +| **Seeed MR60FDA1** | 60 GHz | Fall detection, presence | Seeed proprietary frames | ~$15 | |
| 154 | +| **Seeed MR24HPC1** | 24 GHz | Presence, sleep stage, distance | Seeed proprietary frames | ~$10 | |
| 155 | +| **HLK-LD2410** | 24 GHz | Presence, distance (motion + static) | HLK binary protocol | ~$3 | |
| 156 | +| **HLK-LD2450** | 24 GHz | Multi-target tracking (x,y,speed) | HLK binary protocol | ~$5 | |
| 157 | + |
| 158 | +### Fusion Algorithms |
| 159 | + |
| 160 | +**1. Vital Sign Fusion (Kalman filter)** |
| 161 | +``` |
| 162 | +mmWave HR (high confidence, 1 Hz) ─┐ |
| 163 | + ├─► Kalman fuse → fused HR ± confidence |
| 164 | +CSI-derived HR (lower confidence) ─┘ |
| 165 | +``` |
| 166 | + |
| 167 | +**2. Fall Detection (dual-confirm)** |
| 168 | +``` |
| 169 | +CSI phase accel > thresh ──────┐ |
| 170 | + ├─► AND gate → confirmed fall (near-zero false positives) |
| 171 | +mmWave range-velocity pattern ─┘ |
| 172 | +``` |
| 173 | + |
| 174 | +**3. Presence Validation** |
| 175 | +``` |
| 176 | +CSI adaptive threshold ────┐ |
| 177 | + ├─► Weighted vote → robust presence |
| 178 | +mmWave target count > 0 ──┘ |
| 179 | +``` |
| 180 | + |
| 181 | +**4. Training Calibration** |
| 182 | +``` |
| 183 | +mmWave ground-truth vitals → train CSI BVP extraction model |
| 184 | +mmWave distance → calibrate CSI triangulation |
| 185 | +mmWave micro-Doppler → label CSI activity patterns |
| 186 | +``` |
| 187 | + |
| 188 | +### Vitals Packet Extension |
| 189 | + |
| 190 | +Extend the existing 32-byte vitals packet (magic `0xC5110002`) with a new 48-byte fused packet: |
| 191 | + |
| 192 | +```c |
| 193 | +typedef struct __attribute__((packed)) { |
| 194 | + /* Existing 32-byte vitals fields */ |
| 195 | + uint32_t magic; /* 0xC5110004 (fused vitals) */ |
| 196 | + uint8_t node_id; |
| 197 | + uint8_t flags; /* Bit0=presence, Bit1=fall, Bit2=motion, Bit3=mmwave_present */ |
| 198 | + uint16_t breathing_rate; /* Fused BPM * 100 */ |
| 199 | + uint32_t heartrate; /* Fused BPM * 10000 */ |
| 200 | + int8_t rssi; |
| 201 | + uint8_t n_persons; |
| 202 | + uint8_t mmwave_type; /* Sensor type enum */ |
| 203 | + uint8_t fusion_confidence;/* 0-100 fusion quality score */ |
| 204 | + float motion_energy; |
| 205 | + float presence_score; |
| 206 | + uint32_t timestamp_ms; |
| 207 | + /* New mmWave fields (16 bytes) */ |
| 208 | + float mmwave_hr_bpm; /* Raw mmWave heart rate */ |
| 209 | + float mmwave_br_bpm; /* Raw mmWave breathing rate */ |
| 210 | + float mmwave_distance; /* Distance to nearest target (cm) */ |
| 211 | + uint8_t mmwave_targets; /* Target count */ |
| 212 | + uint8_t mmwave_confidence;/* mmWave signal quality 0-100 */ |
| 213 | + uint16_t reserved; |
| 214 | +} edge_fused_vitals_pkt_t; |
| 215 | + |
| 216 | +_Static_assert(sizeof(edge_fused_vitals_pkt_t) == 48, "fused vitals must be 48 bytes"); |
| 217 | +``` |
| 218 | +
|
| 219 | +### NVS Configuration |
| 220 | +
|
| 221 | +New provisioning parameters: |
| 222 | +
|
| 223 | +```bash |
| 224 | +python provision.py --port COM7 \ |
| 225 | + --mmwave-uart-tx 17 --mmwave-uart-rx 18 \ # UART pins for mmWave module |
| 226 | + --mmwave-type auto \ # auto-detect, or: mr60bha2, ld2410, etc. |
| 227 | + --fusion-mode kalman \ # kalman, vote, mmwave-primary, csi-primary |
| 228 | + --fall-dual-confirm true # require both CSI + mmWave for fall alert |
| 229 | +``` |
| 230 | + |
| 231 | +### Implementation Phases |
| 232 | + |
| 233 | +| Phase | Scope | Effort | |
| 234 | +|-------|-------|--------| |
| 235 | +| **Phase 1** | UART driver + MR60BHA2 parser + auto-detection | 2 weeks | |
| 236 | +| **Phase 2** | Fused vitals packet + Kalman vital sign fusion | 1 week | |
| 237 | +| **Phase 3** | Dual-confirm fall detection + presence voting | 1 week | |
| 238 | +| **Phase 4** | HLK-LD2410/LD2450 support + multi-target fusion | 2 weeks | |
| 239 | +| **Phase 5** | RuVector calibration pipeline (mmWave as ground truth) | 3 weeks | |
| 240 | +| **Phase 6** | Server-side fusion for separate CSI + mmWave nodes | 2 weeks | |
| 241 | + |
| 242 | +## Consequences |
| 243 | + |
| 244 | +### Positive |
| 245 | +- Near-zero false positive fall detection (dual-confirm) |
| 246 | +- Clinical-grade vital signs when mmWave is present, with CSI as fallback |
| 247 | +- Self-calibrating CSI pipeline using mmWave ground truth |
| 248 | +- Backward compatible — existing CSI-only nodes work unchanged |
| 249 | +- Low incremental cost (~$3-15 per mmWave module) |
| 250 | +- Auto-detection means zero configuration for supported sensors |
| 251 | +- RuVector attention/solver/temporal-tensor modules gain a high-quality reference signal |
| 252 | + |
| 253 | +### Negative |
| 254 | +- Added firmware complexity (~2-3 KB RAM for mmWave state + UART buffer) |
| 255 | +- mmWave modules require line-of-sight (complementary to CSI, not replacement) |
| 256 | +- Multiple UART protocols to maintain (Seeed, HLK families) |
| 257 | +- 48-byte fused packet requires server parser update |
| 258 | + |
| 259 | +### Neutral |
| 260 | +- ESP32-C6 cannot run the full CSI pipeline (single-core RISC-V) but can serve as a dedicated mmWave bridge node |
| 261 | +- mmWave modules add ~15 mA power draw per node |
0 commit comments