A Proof of Concept for real-time aircraft telemetry monitoring, simulating IoT-based avionics sensors that generate high-frequency in-flight events 🛩️.
Telemetry is first ingested through Mosquitto MQTT, serving as the lightweight edge protocol for aircraft sensors, and then seamlessly transferred into the streaming pipeline using Kafka Connect.
From there, events flow into Apache Kafka, are processed in real-time by Apache Spark Structured Streaming ⚡, stored in Apache Cassandra 💾, and visualized in operational dashboards 📊.
This project demonstrates how telemetry signals from multiple aircraft systems—such as engine performance, flight dynamics, fuel levels, environmental conditions, and airframe health—can be captured, processed, and acted upon instantly.
- Early detection of system anomalies 🛠️
- Immediate situational awareness 📡 for pilots, analysts, and ground-control teams
- Real-time operational dashboards 📈 for fleet monitoring
- Automated alerts and rule-based monitoring
⚠️ powered by Spark Streaming - High-throughput, fault-tolerant ingestion with MQTT → Kafka Connect → Kafka
This prototype acts as a practical blueprint for designing real-time telemetry pipelines in aviation environments, showing how sensor data can be reliably ingested, processed, stored, and monitored end-to-end.
It demonstrates a complete workflow that can be extended to support richer analytics, larger fleets, or additional aircraft systems as needed.
Aircraft Telemetry Architecture is a Proof of Concept (POC) developed for educational, research, and demonstration purposes only.
Its purpose is to showcase how real-time aircraft telemetry ingestion, event streaming, anomaly detection, and time-series data storage can be combined to build an advanced operational monitoring platform for aviation systems.
This project integrates a modern streaming ecosystem using MQTT, Kafka, Spark Streaming, Cassandra, and Grafana, including support for secure communication and authenticated data pipelines.
It is not intended for production use, All aircraft identifiers, flight data, and telemetry values are fully synthetic and do not correspond to real aircraft, airlines, or aviation systems.
The logo and visual assets for Aircraft Telemetry Architecture were generated by ChatGPT and are free of copyright restrictions.
They are provided exclusively for educational and demonstration purposes, and do not represent any real brand or commercial product.
-
Real-Time Data Streaming 🌊
Telemetry events are ingested from MQTT → Kafka, decoded from Base64, parsed as structured JSON, and processed continuously using Spark Structured Streaming.
The pipeline validates essential fields, converts timestamps, normalizes sensor values, and prepares data for downstream persistence — all with low-latency streaming semantics. -
Dynamic Alert Generation 🚨
The streaming job evaluates several built-in safety rules on each incoming event.
When thresholds are exceeded (e.g., engine overheating 🔥, low fuel ⛽, low oil pressure 🛢️, hydraulic pressure drops 💧, voltage anomalies ⚡, or low cabin pressure 🫁), Spark generates structured alert records with UUIDs, severities, and descriptive messages — in real time.
These alerts are written to Cassandra and also published back to Kafka for external integrations or dashboards. -
Telemetry Storage Pipeline 💾
After cleaning and validating the input, Spark writes enriched telemetry to Apache Cassandra usingforeachBatch.
Data is partitioned by aircraft and event date, enabling large-scale historical storage for analysis, replay, or external queries. -
Operational Metrics & Derived Fields 📊
The system enriches each event with normalized timestamps (ts, event date partitions), maintenance schedule parsing, and validated flight metadata.
These derived fields make downstream analytics and dashboards more accurate and easier to query. -
Configurable Processing Behavior ⚙️
Kafka topics, brokers, Cassandra hosts, and checkpoint locations are configurable through environment variables.
This allows adjusting deployment settings without modifying code, supporting flexible local, containerized, or cloud environments.
The system follows a layered, event-driven architecture, consisting of:
-
Telemetry Sensors Simulation 🛠️
Simulated aircraft sensors generate structured JSON telemetry events, representing flight parameters such as engine temperature, fuel level, hydraulic and electrical metrics, and flight controls.
These events are encoded in Base64 to simulate real-world IoT payloads. -
Message Broker Layer 📨
Telemetry events are first published to MQTT and then ingested into Apache Kafka via Kafka Connect, ensuring reliable, ordered delivery of telemetry streams.
Kafka serves as the backbone for event distribution to multiple downstream consumers. -
Stream Processing Layer ⚡
Apache Spark Structured Streaming continuously consumes Kafka topics, decoding Base64 payloads, parsing JSON into structured events, validating key fields, and converting timestamps.
The stream processing layer also applies real-time alert rules (engine overheating, low fuel, low oil pressure, etc.) and enriches events with derived fields for analytics. -
Storage Layer 💾
Cleaned telemetry and generated alerts are persisted in Apache Cassandra, using a partitioned schema by aircraft ID and event date to allow efficient retrieval.
This distributed storage ensures scalable handling of high-volume telemetry streams and supports historical queries for analytics and dashboards. -
Visualization & Monitoring 📊
Alerts and telemetry metrics are visualized via Grafana dashboards.
Kafka-published alerts allow integration with external monitoring tools or operational dashboards for pilots, ground control, and maintenance teams.
This architecture ensures a seamless flow from simulated sensors to actionable insights, with low-latency streaming, high reliability, and scalability to handle large volumes of aircraft telemetry data.
- Apache Spark Structured Streaming ⚡ – Real-time stream processing, telemetry decoding, data cleaning, and alert generation
- Apache Kafka 📨 – High-throughput event ingestion and distribution for telemetry streams
- MQTT / Mosquitto 🌐 – Lightweight IoT messaging protocol for sensor simulation and ingestion
- Kafka Connect 🔗 – Bridges MQTT telemetry to Kafka topics reliably
- Apache Cassandra 💾 – Distributed, scalable database for raw telemetry, alerts, and aggregated metrics
- Python / PySpark 🐍 – Implements data processing, alert logic, UUID generation, and metrics computation
- Docker & Docker Compose 🐳 – Containerized deployment for consistent development and testing environments
- Logging & Monitoring 📝 – Centralized logging with actionable telemetry insights and optional visualization through Grafana dashboards 📊
All communication between services is secured using mutual TLS (mTLS), ensuring:
- Encrypted data in transit 🔒 to prevent eavesdropping 👀
- Authenticated service-to-service communication 🆔, preventing unauthorized access 🚫
- Centralized certificate management ⚙️ for scalability across multiple services 🏢
This guarantees that telemetry events, alerts, and operational messages remain confidential, authentic, and tamper-proof throughout the system.
- Continuous ingestion of telemetry messages 📩 from simulated aircraft sensors via MQTT, forwarded through Kafka topics
- Payload decoding and parsing 🗃️ to transform Base64-encoded JSON into structured telemetry data
- Data validation and cleaning ✅🧹 to remove missing, zero, or outlier values, ensuring accurate metrics and alerts
- Dynamic alert generation 🚨 in real-time based on configurable thresholds ⚙️ for engine temperature 🔥, fuel level ⛽, oil & hydraulic pressure 💧, and avionics or cabin systems 💻
- Aggregated metrics computation 📊 for short-term monitoring ⏱️ and hourly operational reporting 🕒
- Scalable storage in Apache Cassandra 💾 for raw events, alerts, and metrics, enabling dashboard visualization 🖥️ and historical analysis for predictive maintenance 🧰
This layer provides end-to-end visibility of aircraft systems in real-time, enhancing flight safety 🛡️, operational efficiency ⚡, and proactive maintenance 🧰.
Here are some screenshots that demonstrate the functionality of Real-Time Aircraft Telemetry Architecture:
This project leverages Rake to manage the deployment, lifecycle, and initialization of the Aircraft Telemetry Architecture. Below is a summary of the main tasks and their purposes.
| Task | Description |
|---|---|
aircraft_insights_architecture:deploy |
Deploys the architecture and launches all services and daemons needed for telemetry ingestion, processing, and storage. Runs clean_environment, start, and status internally. |
aircraft_insights_architecture:undeploy |
Undeploys the architecture and stops all running containers. |
aircraft_insights_architecture:start |
Starts all containers after checking Docker, credentials, and deployment files. |
aircraft_insights_architecture:stop |
Stops and removes running containers. |
aircraft_insights_architecture:status |
Shows the current status of all containers. |
aircraft_insights_architecture:clean_environment |
Cleans the Docker environment by pruning unused images and volumes. |
aircraft_insights_architecture:check_docker |
Checks that Docker and Docker Compose are installed and accessible in the PATH. |
aircraft_insights_architecture:login |
Authenticates with Docker using existing credentials. |
aircraft_insights_architecture:check_deployment_file |
Verifies that docker-compose.yml exists in the project directory. |
aircraft_insights_architecture:init_cassandra |
Waits for Cassandra to be ready and initializes the keyspace and tables used for aircraft telemetry storage. |
aircraft_insights_architecture:check_cassandra |
Checks Cassandra tables and retrieves sample records to verify readiness and data integrity. |
This table summarizes the main services and endpoints in the Aircraft Telemetry Proof of Concept (POC). It includes services that can be accessed directly via browser, API, or client, their exposed ports, and a brief description of their role. Internal services such as aircraft nodes or stream processors communicate via MQTT/Kafka and are listed without exposed ports.
| Service / Container | Exposed Port(s) | Role / Description |
|---|---|---|
| ZooKeeper (zookeeper) | 2181 | Coordination service for distributed systems, used by Kafka |
| Kafka (kafka) | 29092 (internal), 9092 (host) | Distributed streaming platform for ingesting and processing aircraft telemetry events |
| AKHQ (akhq) | 8090 | Web UI to explore and manage Kafka topics and monitor streaming data |
| Cassandra (cassandra) | - | Distributed NoSQL database for storing raw telemetry, alerts, and aggregated metrics |
| Grafana (grafana) | 3000 | Visualization dashboards for monitoring telemetry, alerts, and operational metrics |
| Kafka Connect (kafka-connect) | 8083 | Integration layer for Kafka connectors, bridging MQTT messages into Kafka topics |
| Mosquitto Broker (aircraft-broker) | 1883, 8883 | MQTT broker for secure and non-secure communication with simulated aircraft nodes |
| Aircraft Node (aircraft-001 / 002 / 003) | - | Simulated aircraft devices publishing telemetry via MQTT |
| Aircraft Stream Processor (aircraft-stream-processor) | - | Spark Streaming application that validates, enriches, generates alerts, and aggregates telemetry data in real-time |
Aircraft Telemetry Architecture is a Proof of Concept (POC) developed for educational, research, and demonstration purposes only.
Its purpose is to showcase how real-time aircraft telemetry ingestion, event streaming, anomaly detection, and time-series data storage can be combined to build an advanced operational monitoring platform for aviation systems.
This project integrates a modern streaming ecosystem using MQTT, Kafka, Spark Streaming, Cassandra, and Grafana, including support for secure communication and authenticated data pipelines.
It is not intended for production use, All aircraft identifiers, flight data, and telemetry values are fully synthetic and do not correspond to real aircraft, airlines, or aviation systems.
The logo and visual assets for Aircraft Telemetry Architecture were generated by ChatGPT and are free of copyright restrictions.
They are provided exclusively for educational and demonstration purposes, and do not represent any real brand or commercial product.
This project is licensed under the MIT License.