Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 66e2fa0

Browse files
authored
feat: ADR-063/064 mmWave sensor fusion + multimodal ambient intelligence (#269)
* docs: ADR-063 mmWave sensor fusion with WiFi CSI 60 GHz mmWave radar (Seeed MR60BHA2, HLK-LD2410/LD2450) fusion with WiFi CSI for dual-confirm fall detection, clinical-grade vitals, and self-calibrating CSI pipeline. Covers auto-detection, 6 supported sensors, Kalman fusion, extended 48-byte vitals packet, RuVector/RuvSense integration points, and 6-phase implementation plan. Based on live hardware capture from ESP32-C6 + MR60BHA2 on COM4. Co-Authored-By: claude-flow <[email protected]> * feat(firmware): ADR-063 mmWave sensor fusion — full implementation Phase 1-2 of ADR-063: mmwave_sensor.c/h: - MR60BHA2 UART parser (60 GHz: HR, BR, presence, distance) - LD2410 UART parser (24 GHz: presence, distance) - Auto-detection: probes UART for known frame headers at boot - Mock generator for QEMU testing (synthetic HR 72±2, BR 16±1) - Capability flag registration per sensor type edge_processing.c/h: - 48-byte fused vitals packet (magic 0xC5110004) - Kalman-style fusion: mmWave 80% + CSI 20% when both available - Automatic fallback to CSI-only 32-byte packet when no mmWave - Dual presence flag (Bit3 = mmwave_present) main.c: - mmwave_sensor_init() called at boot with auto-detect - Status logged in startup banner Fuzz stubs updated for mmwave_sensor API. Build verified: QEMU mock build passes. Co-Authored-By: claude-flow <[email protected]> * fix(firmware): correct MR60BHA2 + LD2410 UART protocols (ADR-063) MR60BHA2: SOF=0x01 (not 0x5359), XOR+NOT checksums on header and data, frame types 0x0A14 (BR), 0x0A15 (HR), 0x0A16 (distance), 0x0F09 (presence). Based on Seeed Arduino library research. LD2410: 256000 baud (not 115200), 0xAA report head marker, target state byte at offset 2 (after data_type + head_marker). Auto-detect: probes MR60 at 115200 first, then LD2410 at 256000. Sets final baud rate after detection. Co-Authored-By: claude-flow <[email protected]> * feat: ADR-063 Phase 6 server-side mmWave + CSI fusion bridge Python script reads both serial ports simultaneously: - COM4 (ESP32-C6 + MR60BHA2): parses ESPHome debug output for HR, BR, presence, distance - COM7 (ESP32-S3): reads CSI edge processing frames Kalman-style fusion: mmWave 80% + CSI 20% for vitals, OR gate for presence. Verified on real hardware: mmWave HR=75bpm, BR=25/min at 52cm range, CSI frames flowing concurrently. Both sensors live for 30 seconds. Co-Authored-By: claude-flow <[email protected]> * docs: ADR-064 multimodal ambient intelligence roadmap 25+ applications across 4 tiers from practical to exotic: - Tier 1 (build now): zero-FP fall detection, sleep monitoring, occupancy HVAC, baby breathing, bathroom safety - Tier 2 (research): gait analysis, stress detection, gesture control, respiratory screening, multi-room activity - Tier 3 (frontier): cardiac arrhythmia, RF tomography, sign language, cognitive load, swarm sensing - Tier 4 (exotic): emotion contagion, lucid dreaming, plant monitoring, pet behavior Priority matrix with effort estimates. All P0-P1 items work with existing hardware (ESP32-S3 + MR60BHA2 + BH1750). Co-Authored-By: claude-flow <[email protected]> * fix(ci): add ESP_ERR_NOT_FOUND to fuzz stubs mmwave_sensor stub returns ESP_ERR_NOT_FOUND which wasn't defined in the minimal esp_stubs.h for host-based fuzz testing. Co-Authored-By: claude-flow <[email protected]>
1 parent 7a97ffd commit 66e2fa0

11 files changed

Lines changed: 1602 additions & 5 deletions
Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
# ADR-063: 60 GHz mmWave Sensor Fusion with WiFi CSI
2+
3+
**Status:** Proposed
4+
**Date:** 2026-03-15
5+
**Deciders:** @ruvnet
6+
**Related:** ADR-014 (SOTA signal processing), ADR-021 (vital sign extraction), ADR-029 (RuvSense multistatic), ADR-039 (edge intelligence), ADR-042 (CHCI coherent sensing)
7+
8+
## Context
9+
10+
RuView currently senses the environment using WiFi CSI — a passive technique that analyzes how WiFi signals are disturbed by human presence and movement. While this works through walls and requires no line of sight, CSI-derived vital signs (breathing rate, heart rate) are inherently noisy because they rely on phase extraction from multipath-rich WiFi channels.
11+
12+
A complementary sensing modality exists: **60 GHz mmWave radar** modules (e.g., Seeed MR60BHA2) that use active FMCW radar at 60 GHz to measure breathing and heart rate with clinical-grade accuracy. These modules are inexpensive (~$15), run on ESP32-C6/C3, and output structured vital signs over UART.
13+
14+
**Live hardware capture (COM4, 2026-03-15)** from a Seeed MR60BHA2 on an ESP32-C6 running ESPHome:
15+
16+
```
17+
[D][sensor:093]: 'Real-time respiratory rate': Sending state 22.00000
18+
[D][sensor:093]: 'Real-time heart rate': Sending state 92.00000 bpm
19+
[D][sensor:093]: 'Distance to detection object': Sending state 0.00000 cm
20+
[D][sensor:093]: 'Target Number': Sending state 0.00000
21+
[D][binary_sensor:036]: 'Person Information': Sending state OFF
22+
[D][sensor:093]: 'Seeed MR60BHA2 Illuminance': Sending state 0.67913 lx
23+
```
24+
25+
### The Opportunity
26+
27+
Fusing WiFi CSI with mmWave radar creates a sensor system that is greater than the sum of its parts:
28+
29+
| Capability | WiFi CSI Alone | mmWave Alone | Fused |
30+
|-----------|---------------|-------------|-------|
31+
| Through-wall sensing | Yes (5m+) | No (LoS only, ~3m) | Yes — CSI for room-scale, mmWave for precision |
32+
| Heart rate accuracy | ±5-10 BPM | ±1-2 BPM | ±1-2 BPM (mmWave primary, CSI cross-validates) |
33+
| Breathing accuracy | ±2-3 BPM | ±0.5 BPM | ±0.5 BPM |
34+
| Presence detection | Good (adaptive threshold) | Excellent (range-gated) | Excellent + through-wall |
35+
| Multi-person | Via subcarrier clustering | Via range-Doppler bins | Combined spatial + RF resolution |
36+
| Fall detection | Phase acceleration | Range/velocity + micro-Doppler | Dual-confirm reduces false positives to near-zero |
37+
| Pose estimation | Via trained model | Not available | CSI provides pose; mmWave provides ground-truth vitals for training |
38+
| Coverage | Whole room (passive) | ~120° cone, 3m range | Full room + precision zone |
39+
| Cost per node | ~$9 (ESP32-S3) | ~$15 (ESP32-C6 + MR60BHA2) | ~$24 combined |
40+
41+
### RuVector Integration Points
42+
43+
The RuVector v2.0.4 stack (already integrated per ADR-016) provides the signal processing backbone:
44+
45+
| RuVector Component | Role in mmWave Fusion |
46+
|-------------------|----------------------|
47+
| `ruvector-attention` (`bvp.rs`) | Blood Volume Pulse estimation — mmWave heart rate can calibrate the WiFi CSI BVP phase extraction |
48+
| `ruvector-temporal-tensor` (`breathing.rs`) | Breathing rate estimation — mmWave provides ground-truth for adaptive filter tuning |
49+
| `ruvector-solver` (`triangulation.rs`) | Multilateration — mmWave range-gated distance + CSI amplitude = 3D position |
50+
| `ruvector-attn-mincut` (`spectrogram.rs`) | Time-frequency decomposition — mmWave Doppler complements CSI phase spectrogram |
51+
| `ruvector-mincut` (`metrics.rs`, DynamicPersonMatcher) | Multi-person association — mmWave target IDs help disambiguate CSI subcarrier clusters |
52+
53+
### RuvSense Integration Points
54+
55+
The RuvSense multistatic sensing pipeline (ADR-029) gains new capabilities:
56+
57+
| RuvSense Module | mmWave Integration |
58+
|----------------|-------------------|
59+
| `pose_tracker.rs` (AETHER re-ID) | mmWave distance + velocity as additional re-ID features for Kalman tracker |
60+
| `longitudinal.rs` (Welford stats) | mmWave vitals as reference signal for CSI drift detection |
61+
| `intention.rs` (pre-movement) | mmWave micro-Doppler detects pre-movement 100-200ms earlier than CSI |
62+
| `adversarial.rs` (consistency check) | mmWave provides independent signal to detect CSI spoofing/anomalies |
63+
| `coherence_gate.rs` | mmWave presence as additional gate input — if mmWave says "no person", CSI coherence gate rejects |
64+
65+
### Cross-Viewpoint Fusion Integration
66+
67+
The viewpoint fusion pipeline (`ruvector/src/viewpoint/`) extends naturally:
68+
69+
| Viewpoint Module | mmWave Extension |
70+
|-----------------|-----------------|
71+
| `attention.rs` (CrossViewpointAttention) | mmWave range becomes a new "viewpoint" in the attention mechanism |
72+
| `geometry.rs` (GeometricDiversityIndex) | mmWave cone geometry contributes to Fisher Information / Cramer-Rao bounds |
73+
| `coherence.rs` (phase phasor) | mmWave phase coherence as validation for WiFi phasor coherence |
74+
| `fusion.rs` (MultistaticArray) | mmWave node becomes a member of the multistatic array with its own domain events |
75+
76+
## Decision
77+
78+
Add 60 GHz mmWave radar sensor support to the RuView firmware and sensing pipeline with auto-detection and device-specific capabilities.
79+
80+
### Architecture
81+
82+
```
83+
┌─────────────────────────────────────────────────────────┐
84+
│ Sensing Node │
85+
│ │
86+
│ ┌──────────────┐ ┌──────────────┐ ┌────────────┐ │
87+
│ │ ESP32-S3 │ │ ESP32-C6 │ │ Combined │ │
88+
│ │ WiFi CSI │ │ + MR60BHA2 │ │ S3 + UART │ │
89+
│ │ (COM7) │ │ 60GHz mmWave │ │ mmWave │ │
90+
│ │ │ │ (COM4) │ │ │ │
91+
│ │ Passive │ │ Active radar │ │ Both modes │ │
92+
│ │ Through-wall │ │ LoS, precise │ │ │ │
93+
│ └──────┬───────┘ └──────┬───────┘ └─────┬──────┘ │
94+
│ │ │ │ │
95+
│ └────────┬───────────┘ │ │
96+
│ ▼ │ │
97+
│ ┌────────────────┐ │ │
98+
│ │ Fusion Engine │◄──────────────────────┘ │
99+
│ │ │ │
100+
│ │ • Kalman fuse │ Vitals packet (extended): │
101+
│ │ • Cross-validate│ magic 0xC5110004 │
102+
│ │ • Ground-truth │ + mmwave_hr, mmwave_br │
103+
│ │ calibration │ + mmwave_distance │
104+
│ │ • Fall confirm │ + mmwave_target_count │
105+
│ └────────────────┘ + confidence scores │
106+
└─────────────────────────────────────────────────────────┘
107+
```
108+
109+
### Three Deployment Modes
110+
111+
**Mode 1: Standalone CSI (existing)** — ESP32-S3 only, WiFi CSI sensing.
112+
113+
**Mode 2: Standalone mmWave** — ESP32-C6 + MR60BHA2, precise vitals in a single room.
114+
115+
**Mode 3: Fused (recommended)** — ESP32-S3 + mmWave module on UART, or two separate nodes with server-side fusion.
116+
117+
### Auto-Detection Protocol
118+
119+
The firmware will auto-detect connected mmWave modules at boot:
120+
121+
1. **UART probe** — On configured UART pins, send the MR60BHA2 identification command (`0x01 0x01 0x00 0x01 ...`) and check for valid response header
122+
2. **Protocol detection** — Identify the sensor family:
123+
- Seeed MR60BHA2 (breathing + heart rate)
124+
- Seeed MR60FDA1 (fall detection)
125+
- Seeed MR24HPC1 (presence + light sleep/deep sleep)
126+
- HLK-LD2410 (presence + distance)
127+
- HLK-LD2450 (multi-target tracking)
128+
3. **Capability registration** — Register detected sensor capabilities in the edge config:
129+
130+
```c
131+
typedef struct {
132+
uint8_t mmwave_detected; /** 1 if mmWave module found on UART */
133+
uint8_t mmwave_type; /** Sensor family (MR60BHA2, MR60FDA1, etc.) */
134+
uint8_t mmwave_has_hr; /** Heart rate capability */
135+
uint8_t mmwave_has_br; /** Breathing rate capability */
136+
uint8_t mmwave_has_fall; /** Fall detection capability */
137+
uint8_t mmwave_has_presence; /** Presence detection capability */
138+
uint8_t mmwave_has_distance; /** Range measurement capability */
139+
uint8_t mmwave_has_tracking; /** Multi-target tracking capability */
140+
float mmwave_hr_bpm; /** Latest heart rate from mmWave */
141+
float mmwave_br_bpm; /** Latest breathing rate from mmWave */
142+
float mmwave_distance_cm; /** Distance to nearest target */
143+
uint8_t mmwave_target_count; /** Number of detected targets */
144+
bool mmwave_person_present;/** mmWave presence state */
145+
} mmwave_state_t;
146+
```
147+
148+
### Supported Sensors
149+
150+
| Sensor | Frequency | Capabilities | UART Protocol | Cost |
151+
|--------|-----------|-------------|---------------|------|
152+
| **Seeed MR60BHA2** | 60 GHz | HR, BR, presence, illuminance | Seeed proprietary frames | ~$15 |
153+
| **Seeed MR60FDA1** | 60 GHz | Fall detection, presence | Seeed proprietary frames | ~$15 |
154+
| **Seeed MR24HPC1** | 24 GHz | Presence, sleep stage, distance | Seeed proprietary frames | ~$10 |
155+
| **HLK-LD2410** | 24 GHz | Presence, distance (motion + static) | HLK binary protocol | ~$3 |
156+
| **HLK-LD2450** | 24 GHz | Multi-target tracking (x,y,speed) | HLK binary protocol | ~$5 |
157+
158+
### Fusion Algorithms
159+
160+
**1. Vital Sign Fusion (Kalman filter)**
161+
```
162+
mmWave HR (high confidence, 1 Hz) ─┐
163+
├─► Kalman fuse → fused HR ± confidence
164+
CSI-derived HR (lower confidence) ─┘
165+
```
166+
167+
**2. Fall Detection (dual-confirm)**
168+
```
169+
CSI phase accel > thresh ──────┐
170+
├─► AND gate → confirmed fall (near-zero false positives)
171+
mmWave range-velocity pattern ─┘
172+
```
173+
174+
**3. Presence Validation**
175+
```
176+
CSI adaptive threshold ────┐
177+
├─► Weighted vote → robust presence
178+
mmWave target count > 0 ──┘
179+
```
180+
181+
**4. Training Calibration**
182+
```
183+
mmWave ground-truth vitals → train CSI BVP extraction model
184+
mmWave distance → calibrate CSI triangulation
185+
mmWave micro-Doppler → label CSI activity patterns
186+
```
187+
188+
### Vitals Packet Extension
189+
190+
Extend the existing 32-byte vitals packet (magic `0xC5110002`) with a new 48-byte fused packet:
191+
192+
```c
193+
typedef struct __attribute__((packed)) {
194+
/* Existing 32-byte vitals fields */
195+
uint32_t magic; /* 0xC5110004 (fused vitals) */
196+
uint8_t node_id;
197+
uint8_t flags; /* Bit0=presence, Bit1=fall, Bit2=motion, Bit3=mmwave_present */
198+
uint16_t breathing_rate; /* Fused BPM * 100 */
199+
uint32_t heartrate; /* Fused BPM * 10000 */
200+
int8_t rssi;
201+
uint8_t n_persons;
202+
uint8_t mmwave_type; /* Sensor type enum */
203+
uint8_t fusion_confidence;/* 0-100 fusion quality score */
204+
float motion_energy;
205+
float presence_score;
206+
uint32_t timestamp_ms;
207+
/* New mmWave fields (16 bytes) */
208+
float mmwave_hr_bpm; /* Raw mmWave heart rate */
209+
float mmwave_br_bpm; /* Raw mmWave breathing rate */
210+
float mmwave_distance; /* Distance to nearest target (cm) */
211+
uint8_t mmwave_targets; /* Target count */
212+
uint8_t mmwave_confidence;/* mmWave signal quality 0-100 */
213+
uint16_t reserved;
214+
} edge_fused_vitals_pkt_t;
215+
216+
_Static_assert(sizeof(edge_fused_vitals_pkt_t) == 48, "fused vitals must be 48 bytes");
217+
```
218+
219+
### NVS Configuration
220+
221+
New provisioning parameters:
222+
223+
```bash
224+
python provision.py --port COM7 \
225+
--mmwave-uart-tx 17 --mmwave-uart-rx 18 \ # UART pins for mmWave module
226+
--mmwave-type auto \ # auto-detect, or: mr60bha2, ld2410, etc.
227+
--fusion-mode kalman \ # kalman, vote, mmwave-primary, csi-primary
228+
--fall-dual-confirm true # require both CSI + mmWave for fall alert
229+
```
230+
231+
### Implementation Phases
232+
233+
| Phase | Scope | Effort |
234+
|-------|-------|--------|
235+
| **Phase 1** | UART driver + MR60BHA2 parser + auto-detection | 2 weeks |
236+
| **Phase 2** | Fused vitals packet + Kalman vital sign fusion | 1 week |
237+
| **Phase 3** | Dual-confirm fall detection + presence voting | 1 week |
238+
| **Phase 4** | HLK-LD2410/LD2450 support + multi-target fusion | 2 weeks |
239+
| **Phase 5** | RuVector calibration pipeline (mmWave as ground truth) | 3 weeks |
240+
| **Phase 6** | Server-side fusion for separate CSI + mmWave nodes | 2 weeks |
241+
242+
## Consequences
243+
244+
### Positive
245+
- Near-zero false positive fall detection (dual-confirm)
246+
- Clinical-grade vital signs when mmWave is present, with CSI as fallback
247+
- Self-calibrating CSI pipeline using mmWave ground truth
248+
- Backward compatible — existing CSI-only nodes work unchanged
249+
- Low incremental cost (~$3-15 per mmWave module)
250+
- Auto-detection means zero configuration for supported sensors
251+
- RuVector attention/solver/temporal-tensor modules gain a high-quality reference signal
252+
253+
### Negative
254+
- Added firmware complexity (~2-3 KB RAM for mmWave state + UART buffer)
255+
- mmWave modules require line-of-sight (complementary to CSI, not replacement)
256+
- Multiple UART protocols to maintain (Seeed, HLK families)
257+
- 48-byte fused packet requires server parser update
258+
259+
### Neutral
260+
- ESP32-C6 cannot run the full CSI pipeline (single-core RISC-V) but can serve as a dedicated mmWave bridge node
261+
- mmWave modules add ~15 mA power draw per node

0 commit comments

Comments
 (0)