StreamSense — Real-Time Event Streaming Analytics Platform

A production-grade real-time data pipeline that ingests high-throughput e-commerce clickstream events (5,000+ events/sec), processes them through windowed aggregations, and serves a live analytics dashboard. Built to demonstrate distributed systems concepts: consumer-group fault tolerance, partitioned message processing, backpressure handling, and dead-letter queues.

Architecture

┌──────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│   Producer    │────▶│   Apache Kafka       │────▶│   Consumer(s)    │
│  (5k evt/s)  │     │   6 partitions       │     │  Windowed Aggs   │
│  Clickstream │     │   Consumer Groups    │     │  1m / 5m / 15m   │
└──────────────┘     └─────────┬───────────┘     └───────┬──────────┘
                               │                         │
                     ┌─────────▼───────────┐   ┌────────▼─────────┐
                     │   Dead Letter Queue  │   │   Redis (Hot)    │
                     │   Malformed events   │   │   Real-time      │
                     └─────────────────────┘   │   counters        │
                                                └────────┬─────────┘
                                                         │
                     ┌─────────────────────┐   ┌────────▼─────────┐
                     │   PostgreSQL (Cold)  │◀──│   FastAPI         │
                     │   Historical aggs    │   │   REST API        │
                     └─────────────────────┘   └────────┬─────────┘
                                                         │
                                                ┌────────▼─────────┐
                                                │   Next.js         │
                                                │   Live Dashboard  │
                                                │   (polls 2s)      │
                                                └──────────────────┘

Features

• High-Throughput Ingestion — Multi-producer simulation generating 5,000+ synthetic e-commerce events/second with realistic distributions (page views, cart adds, purchases, searches)
• Windowed Aggregations — Rolling 1-minute, 5-minute, and 15-minute windows computing event counts, top products, category breakdown, revenue, region/device distribution
• Consumer-Group Fault Tolerance — Two consumer replicas in the same Kafka consumer group; if one crashes, the other picks up partitions with no data loss
• Dead-Letter Queue — Malformed events (missing required fields) are routed to a separate Kafka topic for inspection instead of crashing the pipeline
• Dual Storage — Redis for sub-second real-time queries (hot path) + PostgreSQL for historical batch analytics (cold path)
• Live Dashboard — Next.js + Recharts frontend polling the FastAPI backend every 2 seconds with area charts, bar charts, pie charts, KPI cards, and consumer health monitoring
• Production-Grade Config — LZ4 compression, batched produces, schema-aware events, configurable backpressure handling

Tech Stack

Layer	Technology
Message Broker	Apache Kafka (Confluent Platform 7.6)
Coordination	ZooKeeper
Stream Processing	Python consumers with windowed aggregation logic
Hot Storage	Redis 7 (real-time materialized views)
Cold Storage	PostgreSQL 16 (historical aggregations)
API	FastAPI (Python) with ORJSON serialization
Dashboard	Next.js 14, TypeScript, Recharts, Tailwind CSS
Orchestration	Docker Compose

Quick Start

Prerequisites

• Docker & Docker Compose
• 8GB+ RAM recommended (Kafka + ZooKeeper are memory-hungry)

Run Everything

# Clone the repo
git clone https://github.com/RohanMukka/StreamSense.git
cd StreamSense

# Start all services
docker compose up --build

# The following services will be available:
# Dashboard:       http://localhost:3000
# API:             http://localhost:8000
# API Docs:        http://localhost:8000/docs
# Kafka:           localhost:9092
# Schema Registry: http://localhost:8081
# Redis:           localhost:6379
# PostgreSQL:      localhost:5432

Verify It's Working

# Check Kafka topics
docker exec ss-kafka kafka-topics --list --bootstrap-server localhost:9092

# Watch producer logs
docker logs -f ss-producer

# Watch consumer logs
docker logs -f ss-consumer

# Hit the API
curl http://localhost:8000/api/metrics/realtime?window=1m | python -m json.tool

API Endpoints

Endpoint	Description
GET /health	Service health check
GET /api/metrics/realtime?window=1m	Current window's aggregated metrics
GET /api/metrics/throughput	Global event throughput stats
GET /api/metrics/timeseries?window=1m&metric=event_type&periods=30	Time-series for charts
GET /api/consumers	Consumer group health and heartbeats
GET /api/kafka/stats	Kafka topic and partition info
GET /api/dlq?limit=50	Dead-letter queue events
GET /api/metrics/historical?window=1m&metric_name=event_type	Historical aggregations from PostgreSQL

Testing Fault Tolerance

# Kill one consumer — watch the other pick up its partitions
docker stop ss-consumer-replica

# Check consumer health endpoint
curl http://localhost:8000/api/consumers

# Restart the consumer
docker start ss-consumer-replica

Design Decisions & Trade-offs

Decision	Rationale
At-least-once delivery	Simpler than exactly-once; idempotent aggregation (counters) handles duplicates gracefully. For financial systems, exactly-once via transactions would be required.
6 Kafka partitions	Balances parallelism with resource constraints. In production, partition count = max consumer parallelism. Would scale to 12-24 for 50k+ events/sec.
Redis for hot path	Sub-millisecond reads for dashboard polling. Auto-expiring keys (TTL = 2× window) eliminate manual cleanup.
PostgreSQL for cold path	ACID guarantees for historical queries and compliance. In production, would migrate to TimescaleDB or ClickHouse for time-series optimization.
LZ4 compression	Best throughput/compression trade-off for Kafka. ~40% size reduction with minimal CPU overhead vs. gzip/snappy.
Leader-only acks (acks=1)	Acceptable for analytics workloads where occasional message loss is tolerable. Financial systems would use acks=all.

Project Structure

StreamSense/
├── producer/              # Kafka event producer (Python)
│   ├── main.py            # High-throughput clickstream generator
│   ├── Dockerfile
│   └── requirements.txt
├── consumer/              # Kafka consumer with windowed aggregations
│   ├── main.py            # Aggregation engine + DLQ routing
│   ├── Dockerfile
│   └── requirements.txt
├── api/                   # FastAPI backend
│   ├── main.py            # REST endpoints for metrics
│   ├── Dockerfile
│   └── requirements.txt
├── dashboard/             # Next.js real-time dashboard
│   ├── src/
│   │   ├── app/
│   │   │   ├── page.tsx   # Main dashboard with charts
│   │   │   ├── layout.tsx
│   │   │   └── globals.css
│   │   └── lib/
│   │       └── api.ts     # API client
│   ├── Dockerfile
│   ├── package.json
│   └── tailwind.config.js
├── scripts/
│   ├── init.sql           # PostgreSQL schema
│   └── create-topics.sh   # Kafka topic creation
├── docker-compose.yml     # Full stack orchestration
├── .env.example
└── README.md

Scaling Considerations

• Horizontal consumer scaling: Add more consumer instances to the same group — Kafka rebalances partitions automatically
• Producer throughput: Current ~5k events/sec is CPU-bound on synthetic generation. Real producers with pre-built payloads can push 50k+/sec per instance
• Dashboard scaling: Next.js is stateless — deploy behind a load balancer with any number of replicas
• Redis → Redis Cluster: For >100k counters, switch to Redis Cluster with hash-slot-based sharding
• PostgreSQL → TimescaleDB: For time-series queries at scale, hypertables with automated partitioning

Author

Rohan Mukka — GitHub | LinkedIn | Portfolio