DataScope — Full-Stack Data Analytics Dashboard

6.25M+ rows cleaned · 7 public datasets (US + India) · 8 ML models · FastAPI + vanilla JS

---

A production-style analytics dashboard demonstrating the full data lifecycle: ingestion → cleaning → preprocessing → ML modeling → aggregation → serving → interactive visualisation → automated findings reports.

Built to showcase skills in data cleaning, preprocessing/post-processing, large-dataset handling, and dashboard engineering.

🗂️ Datasets (~7M raw rows combined)

#	Dataset	Source	Raw rows	Clean rows
1	NYC 311 Service Requests 2025	NYC Open Data	1,000,000	856,275
2	Chicago Crime Reports	Chicago Data Portal	1,149,975	1,137,740
3	NYC TLC Yellow Taxi Trips	TLC Trip Records	3,724,889	3,473,299
4	UCI Online Retail II	UCI ML Repository	1,067,371	779,425
5	India COVID-19 (67 indicators, 2020–24)	Our World in Data	1,682 daily obs	1,682
6	India Air Quality (CPCB, live)	data.gov.in	38 stations	38
7	NIFTY 50 OHLCV (2y daily)	Yahoo Finance	499 sessions	495

🧼 Data Cleaning & Preprocessing (scripts/etl.py)

• Memory-optimised ingestion — column pruning, dtype downcasting (int64→int32, float64→float32), parquet columnar storage (800MB raw → 145MB clean)
• Deduplication on business keys (unique_key, id, trip-composite key)
• Domain-rule validation — TLC trip-recording rules (fare/distance/duration/speed bounds), geo-coordinate bounding boxes, response-time caps
• IQR outlier culling, null imputation, categorical normalisation (title-casing, zip extraction, boolean coercion)
• Feature engineering — response times, trip speed, tip ratio, hour/dow/weekend flags, revenue
• Pre-aggregation into dashboard-optimised JSON assets + an automated ETL quality report

🤖 ML Models (8 total)

Model	Task	Key metric
K-Means (+ PCA, elbow sweep)	Taxi trip segmentation	80% variance in PC1+PC2
Ridge Regression w/ lag features	Taxi demand forecast	MAPE 20.2%
Isolation Forest	Taxi fare/speed anomalies	4,000 flagged (2%)
GradientBoostingRegressor (+ OneHot pipeline)	311 response-time prediction	R² 0.565, MAE 16.1h vs 26.4h baseline
Rolling z-score change-point detection	COVID wave identification (unsupervised)	6 waves, Delta peak 391K/day
K-Means on pollutant fingerprints	AQI station clustering	3 pollution regimes
Isolation Forest	Anomalous AQI stations	6 stations flagged
Ridge (lag/vol features, no lookahead)	NIFTY next-day return	54.5% directional hit-rate vs 50% baseline

📄 Automated Findings Reports

scripts/make_reports.py generates a per-dataset findings report (key findings, pre-processing inventory, algorithm results) from the live ETL/ML report JSONs — rebuilt on every pipeline run and rendered in the dashboard's Reports tab (and machine-readable JSON under data/reports/).

🖥️ Dashboard (frontend/index.html)

7 views — Overview · Taxi · Crime · NYC 311 · Retail & RFM · India 🇮🇳 · Reports — featuring:

• KPI cards computed live from cleaned data
• Server-side filtering: date-range pickers, multi-select category filters, numeric range inputs, free-text search
• Column show/hide picker on every data table
• Sortable, paginated tables (powered by pandas on the backend, millions of rows)
• India tab: COVID waves + vaccination curves, CPCB AQI city/band/pollutant analytics, NIFTY price/MA/volatility/forecast charts
• Reports tab: per-dataset findings reports with preprocessing & algorithm documentation

🏗️ Architecture

data/raw ──▶ scripts/etl.py + etl_india.py ──▶ data/clean (parquet)
                        │                                │
                        ▼                                ▼
             data/aggregates (JSON) ──▶ backend/routers/* (FastAPI)
                        ▲                                │
             scripts/ml.py + ml_india.py ─▶ ml/*.joblib  ▼
                                     data/reports ◀── scripts/make_reports.py
                                                      ▲
                                            frontend/ (Chart.js SPA)

🚀 Run Locally

# 1. environment
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. download raw data + clean + model (≈1.5 min)
python scripts/download_data.py   # optional; or fetch sources manually
python scripts/etl.py
python scripts/ml.py

# 3. serve
uvicorn backend.main:app --reload
# open http://localhost:8000

☁️ Free Hosting

Backend + dashboard (one service):

• Render.com free web service — start command: uvicorn backend.main:app --host 0.0.0.0 --port $PORT
• Railway.app / Fly.io free tiers also work
• ⚠️ Commit only data/clean + data/aggregates (145MB) or add a data-download step to the build; keep raw out of git

Frontend-only demo: deploy frontend/ to GitHub Pages / Netlify / Vercel and set const API = "https://your-render-app.onrender.com" in index.html.

📡 API Highlights

Endpoint	Description
GET /api/overview	Global KPIs, ETL + ML reports
GET /api/311/records?borough=…&q=…&start=…&columns=…	Filter + search + paginate + column selection
GET /api/crimes/records?arrest=true&crime_type=…	Same, over 1.1M incidents
GET /api/taxi/records?hour=…&min_fare=…	3.4M trips, facet-filtered
GET /api/retail/rfm?segment=Champion&min_spend=…	RFM customer segments
GET /docs	Full OpenAPI/Swagger UI

---

Data sources: NYC Open Data, Chicago Data Portal, NYC TLC, UCI — used under their open-license terms.