Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit c6ad674

Browse files
claudeclaude
committed
docs(adr-018): Add ESP32 development implementation ADR
Documents the concrete 4-layer development sequence for closing the hardware gap: firmware (ESP-IDF C), UDP aggregator (Rust), CsiFrame→CsiData bridge, and Python _read_raw_data() UDP socket replacement. Builds on ADR-012 architecture and existing wifi-densepose-hardware parser crate. Includes testability path for all layers before hardware acquisition. https://claude.ai/code/session_01BSBAQJ34SLkiJy4A8SoiL4
1 parent 5cc2198 commit c6ad674

1 file changed

Lines changed: 312 additions & 0 deletions

File tree

‎docs/adr/ADR-018-esp32-dev-implementation.md‎

Lines changed: 312 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,312 @@
1+
# ADR-018: ESP32 Development Implementation Path
2+
3+
## Status
4+
Proposed
5+
6+
## Date
7+
2026-02-28
8+
9+
## Context
10+
11+
ADR-012 established the ESP32 CSI Sensor Mesh architecture: hardware rationale, firmware file structure, `csi_feature_frame_t` C struct, aggregator design, clock-drift handling via feature-level fusion, and a $54 starter BOM. That ADR answers *what* to build and *why*.
12+
13+
This ADR answers *how* to build it — the concrete development sequence, the specific integration points in existing code, and how to test each layer before hardware is in hand.
14+
15+
### Current State
16+
17+
**Already implemented:**
18+
19+
| Component | Location | Status |
20+
|-----------|----------|--------|
21+
| Binary frame parser | `wifi-densepose-hardware/src/esp32_parser.rs` | Complete — `Esp32CsiParser::parse_frame()`, `parse_stream()`, 7 passing tests |
22+
| Frame types | `wifi-densepose-hardware/src/csi_frame.rs` | Complete — `CsiFrame`, `CsiMetadata`, `SubcarrierData`, `to_amplitude_phase()` |
23+
| Parse error types | `wifi-densepose-hardware/src/error.rs` | Complete — `ParseError` enum with 6 variants |
24+
| Signal processing pipeline | `wifi-densepose-signal` crate | Complete — Hampel, Fresnel, BVP, Doppler, spectrogram |
25+
| CSI extractor (Python) | `v1/src/hardware/csi_extractor.py` | Stub — `_read_raw_data()` raises `NotImplementedError` |
26+
| Router interface (Python) | `v1/src/hardware/router_interface.py` | Stub — `_parse_csi_response()` raises `RouterConnectionError` |
27+
28+
**Not yet implemented:**
29+
30+
- ESP-IDF C firmware (`firmware/esp32-csi-node/`)
31+
- UDP aggregator binary (`crates/wifi-densepose-hardware/src/aggregator/`)
32+
- `CsiFrame` → `wifi_densepose_signal::CsiData` bridge
33+
- Python `_read_raw_data()` real UDP socket implementation
34+
- Proof capture tooling for real hardware
35+
36+
### Binary Frame Format (implemented in `esp32_parser.rs`)
37+
38+
```
39+
Offset Size Field
40+
0 4 Magic: 0xC5110001 (LE)
41+
4 1 Node ID (0-255)
42+
5 1 Number of antennas
43+
6 2 Number of subcarriers (LE u16)
44+
8 4 Frequency Hz (LE u32, e.g. 2412 for 2.4 GHz ch1)
45+
12 4 Sequence number (LE u32)
46+
16 1 RSSI (i8, dBm)
47+
17 1 Noise floor (i8, dBm)
48+
18 2 Reserved (zero)
49+
20 N*2 I/Q pairs: (i8, i8) per subcarrier, repeated per antenna
50+
```
51+
52+
Total frame size: 20 + (n_antennas × n_subcarriers × 2) bytes.
53+
54+
For 3 antennas, 56 subcarriers: 20 + 336 = 356 bytes per frame.
55+
56+
The firmware must write frames in this exact format. The parser already validates magic, bounds-checks `n_subcarriers` (≤512), and resyncs the stream on magic search for `parse_stream()`.
57+
58+
## Decision
59+
60+
We will implement the ESP32 development stack in four sequential layers, each independently testable before hardware is available.
61+
62+
### Layer 1 — ESP-IDF Firmware (`firmware/esp32-csi-node/`)
63+
64+
Implement the C firmware project per the file structure in ADR-012. Key design decisions deferred from ADR-012:
65+
66+
**CSI callback → frame serializer:**
67+
68+
```c
69+
// main/csi_collector.c
70+
static void csi_data_callback(void *ctx, wifi_csi_info_t *info) {
71+
if (!info || !info->buf) return;
72+
73+
// Write binary frame header (20 bytes, little-endian)
74+
uint8_t frame[FRAME_MAX_BYTES];
75+
uint32_t magic = 0xC5110001;
76+
memcpy(frame + 0, &magic, 4);
77+
frame[4] = g_node_id;
78+
frame[5] = info->rx_ctrl.ant; // antenna index (1 for ESP32 single-antenna)
79+
uint16_t n_sub = info->len / 2; // len = n_subcarriers * 2 (I + Q bytes)
80+
memcpy(frame + 6, &n_sub, 2);
81+
uint32_t freq_mhz = g_channel_freq_mhz;
82+
memcpy(frame + 8, &freq_mhz, 4);
83+
memcpy(frame + 12, &g_seq_num, 4);
84+
frame[16] = (int8_t)info->rx_ctrl.rssi;
85+
frame[17] = (int8_t)info->rx_ctrl.noise_floor;
86+
frame[18] = 0; frame[19] = 0;
87+
88+
// Write I/Q payload directly from info->buf
89+
memcpy(frame + 20, info->buf, info->len);
90+
91+
// Send over UDP to aggregator
92+
stream_sender_write(frame, 20 + info->len);
93+
g_seq_num++;
94+
}
95+
```
96+
97+
**No on-device FFT** (contradicting ADR-012's optional feature extraction path): The Rust aggregator will do feature extraction using the SOTA `wifi-densepose-signal` pipeline. Raw I/Q is cheaper to stream at ESP32 sampling rates (~100 Hz at 56 subcarriers = ~35 KB/s per node).
98+
99+
**`sdkconfig.defaults`** must enable:
100+
101+
```
102+
CONFIG_ESP_WIFI_CSI_ENABLED=y
103+
CONFIG_LWIP_SO_RCVBUF=y
104+
CONFIG_FREERTOS_HZ=1000
105+
```
106+
107+
**Build toolchain**: ESP-IDF v5.2+ (pinned). Docker image: `espressif/idf:v5.2` for reproducible CI.
108+
109+
### Layer 2 — UDP Aggregator (`crates/wifi-densepose-hardware/src/aggregator/`)
110+
111+
New module within the hardware crate. Entry point: `aggregator_main()` callable as a binary target.
112+
113+
```rust
114+
// crates/wifi-densepose-hardware/src/aggregator/mod.rs
115+
116+
pub struct Esp32Aggregator {
117+
socket: UdpSocket,
118+
nodes: HashMap<u8, NodeState>, // keyed by node_id from frame header
119+
tx: mpsc::SyncSender<CsiFrame>, // outbound to bridge
120+
}
121+
122+
struct NodeState {
123+
last_seq: u32,
124+
drop_count: u64,
125+
last_recv: Instant,
126+
}
127+
128+
impl Esp32Aggregator {
129+
/// Bind UDP socket and start blocking receive loop.
130+
/// Each valid frame is forwarded on `tx`.
131+
pub fn run(&mut self) -> Result<(), AggregatorError> {
132+
let mut buf = vec![0u8; 4096];
133+
loop {
134+
let (n, _addr) = self.socket.recv_from(&mut buf)?;
135+
match Esp32CsiParser::parse_frame(&buf[..n]) {
136+
Ok((frame, _consumed)) => {
137+
let state = self.nodes.entry(frame.metadata.node_id)
138+
.or_insert_with(NodeState::default);
139+
// Track drops via sequence number gaps
140+
if frame.metadata.seq_num != state.last_seq + 1 {
141+
state.drop_count += (frame.metadata.seq_num
142+
.wrapping_sub(state.last_seq + 1)) as u64;
143+
}
144+
state.last_seq = frame.metadata.seq_num;
145+
state.last_recv = Instant::now();
146+
let _ = self.tx.try_send(frame); // drop if pipeline is full
147+
}
148+
Err(e) => {
149+
// Log and continue — never crash on bad UDP packet
150+
eprintln!("aggregator: parse error: {e}");
151+
}
152+
}
153+
}
154+
}
155+
}
156+
```
157+
158+
**Testable without hardware**: The test suite generates frames using `build_test_frame()` (same helper pattern as `esp32_parser.rs` tests) and sends them over a loopback UDP socket. The aggregator receives and forwards them identically to real hardware frames.
159+
160+
### Layer 3 — CsiFrame → CsiData Bridge
161+
162+
Bridge from `wifi-densepose-hardware::CsiFrame` to the signal processing type `wifi_densepose_signal::CsiData` (or a compatible intermediate type consumed by the Rust pipeline).
163+
164+
```rust
165+
// crates/wifi-densepose-hardware/src/bridge.rs
166+
167+
use crate::{CsiFrame};
168+
169+
/// Intermediate type compatible with the signal processing pipeline.
170+
/// Maps directly from CsiFrame without cloning the I/Q storage.
171+
pub struct CsiData {
172+
pub timestamp_unix_ms: u64,
173+
pub node_id: u8,
174+
pub n_antennas: usize,
175+
pub n_subcarriers: usize,
176+
pub amplitude: Vec<f64>, // length: n_antennas * n_subcarriers
177+
pub phase: Vec<f64>, // length: n_antennas * n_subcarriers
178+
pub rssi_dbm: i8,
179+
pub noise_floor_dbm: i8,
180+
pub channel_freq_mhz: u32,
181+
}
182+
183+
impl From<CsiFrame> for CsiData {
184+
fn from(frame: CsiFrame) -> Self {
185+
let n_ant = frame.metadata.n_antennas as usize;
186+
let n_sub = frame.metadata.n_subcarriers as usize;
187+
let (amplitude, phase) = frame.to_amplitude_phase();
188+
CsiData {
189+
timestamp_unix_ms: frame.metadata.timestamp_unix_ms,
190+
node_id: frame.metadata.node_id,
191+
n_antennas: n_ant,
192+
n_subcarriers: n_sub,
193+
amplitude,
194+
phase,
195+
rssi_dbm: frame.metadata.rssi_dbm,
196+
noise_floor_dbm: frame.metadata.noise_floor_dbm,
197+
channel_freq_mhz: frame.metadata.channel_freq_mhz,
198+
}
199+
}
200+
}
201+
```
202+
203+
The bridge test: parse a known binary frame, convert to `CsiData`, assert `amplitude[0]` = √(I₀² + Q₀²) to within f64 precision.
204+
205+
### Layer 4 — Python `_read_raw_data()` Real Implementation
206+
207+
Replace the `NotImplementedError` stub in `v1/src/hardware/csi_extractor.py` with a UDP socket reader. This allows the Python pipeline to receive real CSI from the aggregator while the Rust pipeline is being integrated.
208+
209+
```python
210+
# v1/src/hardware/csi_extractor.py
211+
# Replace _read_raw_data() stub:
212+
213+
import socket as _socket
214+
215+
class CSIExtractor:
216+
...
217+
def _read_raw_data(self) -> bytes:
218+
"""Read one raw CSI frame from the UDP aggregator.
219+
220+
Expects binary frames in the ESP32 format (magic 0xC5110001 header).
221+
Aggregator address configured via AGGREGATOR_HOST / AGGREGATOR_PORT
222+
environment variables (defaults: 127.0.0.1:5005).
223+
"""
224+
if not hasattr(self, '_udp_socket'):
225+
host = self.config.get('aggregator_host', '127.0.0.1')
226+
port = int(self.config.get('aggregator_port', 5005))
227+
sock = _socket.socket(_socket.AF_INET, _socket.SOCK_DGRAM)
228+
sock.bind((host, port))
229+
sock.settimeout(1.0)
230+
self._udp_socket = sock
231+
try:
232+
data, _ = self._udp_socket.recvfrom(4096)
233+
return data
234+
except _socket.timeout:
235+
raise CSIExtractionError(
236+
"No CSI data received within timeout — "
237+
"is the ESP32 aggregator running?"
238+
)
239+
```
240+
241+
This is tested with a mock UDP server in the unit tests (existing `test_csi_extractor_tdd.py` pattern) and with the real aggregator in integration.
242+
243+
## Development Sequence
244+
245+
```
246+
Phase 1 (Firmware + Aggregator — no pipeline integration needed):
247+
1. Write firmware/esp32-csi-node/ C project (ESP-IDF v5.2)
248+
2. Flash to one ESP32-S3-DevKitC board
249+
3. Verify binary frames arrive on laptop UDP socket using Wireshark
250+
4. Write aggregator crate + loopback test
251+
252+
Phase 2 (Bridge + Python stub):
253+
5. Implement CsiFrame → CsiData bridge
254+
6. Replace Python _read_raw_data() with UDP socket
255+
7. Run Python pipeline end-to-end against loopback aggregator (synthetic frames)
256+
257+
Phase 3 (Real hardware integration):
258+
8. Run Python pipeline against live ESP32 frames
259+
9. Capture 10-second real CSI bundle (firmware/esp32-csi-node/proof/)
260+
10. Verify proof bundle hash (ADR-011 pattern)
261+
11. Mark ADR-012 Accepted, mark this ADR Accepted
262+
```
263+
264+
## Testing Without Hardware
265+
266+
All four layers are testable before a single ESP32 is purchased:
267+
268+
| Layer | Test Method |
269+
|-------|-------------|
270+
| Firmware binary format | Build a `build_test_frame()` helper in Rust, compare its output byte-for-byte against a hand-computed reference frame |
271+
| Aggregator | Loopback UDP: test sends synthetic frames to 127.0.0.1:5005, aggregator receives and forwards on channel |
272+
| Bridge | `assert_eq!(csi_data.amplitude[0], f64::sqrt((iq[0].i as f64).powi(2) + (iq[0].q as f64).powi(2)))` |
273+
| Python UDP reader | Mock UDP server in pytest using `socket.socket` in a background thread |
274+
275+
The existing `esp32_parser.rs` test suite already validates parsing of correctly-formatted binary frames. The aggregator and bridge tests build on top of the same test frame construction.
276+
277+
## Consequences
278+
279+
### Positive
280+
- **Layered testability**: Each layer can be validated independently before hardware acquisition.
281+
- **No new external dependencies**: UDP sockets are in stdlib (both Rust and Python). Firmware uses only ESP-IDF and esp-dsp component.
282+
- **Stub elimination**: Replaces the last two `NotImplementedError` stubs in the Python hardware layer with real code backed by real data.
283+
- **Proof of reality**: Phase 3 produces a captured CSI bundle hashed to a known value, satisfying ADR-011 for hardware-sourced data.
284+
- **Signal-crate reuse**: The SOTA Hampel/Fresnel/BVP/Doppler processing from ADR-014 applies unchanged to real ESP32 frames after the bridge converts them.
285+
286+
### Negative
287+
- **Firmware requires ESP-IDF toolchain**: Not buildable without a 2+ GB ESP-IDF installation. CI must use the official Docker image or skip firmware compilation.
288+
- **Raw I/Q bandwidth**: Streaming raw I/Q (not features) at 100 Hz × 3 antennas × 56 subcarriers = ~35 KB/s/node. At 6 nodes = ~210 KB/s. Fine for LAN; not suitable for WAN.
289+
- **Single-antenna real-world**: Most ESP32-S3-DevKitC boards have one on-board antenna. Multi-antenna data requires external antenna + board with U.FL connector or purpose-built multi-radio setup.
290+
291+
### Deferred
292+
- **Multi-node clock drift compensation**: ADR-012 specifies feature-level fusion. The aggregator in this ADR passes raw `CsiFrame` per-node. Drift compensation lives in a future `FeatureFuser` layer (not scoped here).
293+
- **ESP-IDF firmware CI**: Firmware compilation in GitHub Actions requires the ESP-IDF Docker image. CI integration is deferred until Phase 3 hardware validation.
294+
295+
## Interaction with Other ADRs
296+
297+
| ADR | Interaction |
298+
|-----|-------------|
299+
| ADR-011 | Phase 3 produces a real CSI proof bundle satisfying mock elimination |
300+
| ADR-012 | This ADR implements the development path for ADR-012's architecture |
301+
| ADR-014 | SOTA signal processing applies unchanged after bridge layer |
302+
| ADR-008 | Aggregator handles multi-node; distributed consensus is a later concern |
303+
304+
## References
305+
306+
- [Espressif ESP-CSI Repository](https://github.com/espressif/esp-csi)
307+
- [ESP-IDF WiFi CSI API Reference](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/wifi.html#wi-fi-channel-state-information)
308+
- `wifi-densepose-hardware/src/esp32_parser.rs` — binary frame parser implementation
309+
- `wifi-densepose-hardware/src/csi_frame.rs` — `CsiFrame`, `to_amplitude_phase()`
310+
- ADR-012: ESP32 CSI Sensor Mesh (architecture)
311+
- ADR-011: Python Proof-of-Reality and Mock Elimination
312+
- ADR-014: SOTA Signal Processing

0 commit comments

Comments
 (0)