Unified data pipeline for two production apps. Extracts from Souq (service marketplace, PostGIS) and Affordable Imports (ecommerce, M-Pesa/Paystack), transforms through a Bronze → Silver → Gold medallion architecture, and produces business-ready analytics covering demand zones, provider performance, cross-platform customer LTV, and booking funnel drop-off.
Two production databases feed a single unified pipeline:
Souq (service marketplace) — extracts service requests with PostGIS geometry, bookings, provider profiles, real-time provider location pings, and Paystack/M-Pesa payment records. Geometry columns (GEOMETRY(Point, 4326)) are serialised to WKB hex by psycopg2, converted to lat/lng floats via Shapely, and validated against a Nairobi bounding box before any data reaches Bronze.
Affordable Imports (ecommerce) — extracts orders, order line items, and customers. M-Pesa STK references and Paystack transaction IDs are extracted; raw webhook payloads and credentials are never stored.
Both sources feed a Bronze layer of date-partitioned Parquet files registered as DuckDB virtual tables. Silver applies Polars cleaning and Pydantic v2 schema validation — Kenyan phone numbers are normalised to E.164 format (+254...) for cross-platform customer matching. Gold runs five dbt models producing analytics specific to these two apps. Prefect orchestrates the full chain daily with Slack failure alerts. A FastAPI service exposes pipeline status and a protected trigger endpoint.
These are the five business questions this pipeline answers:
demand_by_zone — Where is service demand highest in Nairobi, at what hours, and how much goes unfulfilled? Snaps request coordinates to a 0.005° grid (~500m cells), aggregates by zone/hour/category, and derives peak demand flags using a windowed 75th percentile. Grain: zone_key + date + hour + service_category.
provider_performance — Which providers are reliably completing jobs, how fast do they respond, and how much have they earned? Joins provider profiles with lifetime booking metrics and payment records. Derives quality_tier (elite/good/standard/new) from completion rate and average customer rating. Grain: provider_id.
cross_platform_customers — Which customers use both Souq and Affordable Imports? Matches on normalised E.164 phone number. Computes combined LTV across both platforms, which platform they joined first, and a cross_sell_segment (souq_upsell, imports_upsell, power_user, standard). This is the model that makes this a unified pipeline rather than two separate ones. Grain: normalised_phone.
booking_funnel — Where are customers dropping off between request and completed service? Tracks request → quote → confirmed → completed with drop-off rates at each stage and average time-to-book by category. Grain: summary_date + service_category.
ecommerce_daily — Daily Affordable Imports metrics by county: GMV, AOV, M-Pesa vs Paystack payment split, new vs returning customers, and cancellation rate. Grain: summary_date + county.
PostGIS geometry extraction without PostGIS in the warehouse — Souq's app database uses GEOMETRY(Point, 4326) columns. The pipeline extracts them as WKB hex via ST_AsEWKB(), converts to lat/lng floats using Shapely (pure Python, no GDAL dependency in the pipeline container), and applies a configurable Nairobi bounding box filter before Bronze. The warehouse never sees raw geometry — it stores plain floats.
Zone snapping in DuckDB, not PostGIS — demand_by_zone derives spatial grid cells by snapping coordinates to a 0.005° grid (round(lat / 0.005) * 0.005). This runs in DuckDB during Silver transformation, produces a zone_key string, and eliminates the need for PostGIS in the warehouse while remaining fully queryable by BI tools. No H3 or S2 library dependency.
Cross-platform phone matching — The cross_platform_customers model joins Souq and Affordable Imports customers by phone number. Both apps store phone numbers in different formats (07xx, +2547xx, 2547xx). Silver normalises all phone numbers to E.164 using the phonenumbers library before any join is attempted. Without this, the join would silently miss most matches.
Read-only source connections — both SouqConnector and ImportsConnector open psycopg2 connections with set_session(readonly=True). The DSN also sets default_transaction_read_only=on at the PostgreSQL session level. A misconfigured pipeline cannot mutate production data.
Payment credential scrubbing — extract_payments() explicitly pops raw_webhook_payload from every row before Bronze. M-Pesa STK references and Paystack transaction IDs are kept (they're analytics-relevant), but anything that could contain signing secrets is removed at the connector level, before Parquet serialisation.
Pydantic v2 domain schemas — Silver schemas reflect the actual business domain: ServiceRequestRecord validates that a request has either a location or a description; BookingRecord enforces that completed bookings have completed_at set; PaymentRecord enforces that every payment links to either a booking_id or an order_id; ImportOrderRecord validates that total_amount == subtotal + shipping_cost within a 1 KES tolerance. These are invariants about the data, not just type checks.
Hybrid CDC + batch extraction — Souq's high-frequency tables (bookings, provider_locations) use PostgreSQL logical replication (pgoutput) for incremental capture. Lower-frequency tables (service_requests, providers, payments) use incremental batch extract filtered by updated_at. Affordable Imports uses append-dominant batch extract with a configurable lookback window (IMPORTS_LOOKBACK_DAYS).
git clone https://github.com/kisugez/souq-etl.git
cd souq-etl
cp .env.example .env
# Add Souq DB, Affordable Imports DB, warehouse credentials, and PIPELINE_API_KEY
docker compose up -dRegister Prefect deployments (run once):
docker compose exec pipeline python scripts/register_deployments.pyTrigger a full pipeline run:
curl -X POST http://localhost:8000/pipeline/trigger \
-H "Authorization: Bearer $PIPELINE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"flow": "full_pipeline"}'Services:
| Service | URL |
|---|---|
| Prefect UI | http://localhost:4200 |
| Pipeline API docs | http://localhost:8000/docs |
| PostgreSQL warehouse | localhost:5432 |
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/health |
None | Service health |
GET |
/pipeline/status |
None | Last-known state per layer |
GET |
/pipeline/runs |
None | Recent run history (proxied from Prefect) |
POST |
/pipeline/trigger |
Bearer token |
Trigger a flow run |
pytest tests/unit/ -v # no infrastructure needed
pytest tests/integration/ -v # mocked DB + Prefect
pytest --cov=src --cov=prefect_flows -v # full suiteUnit tests cover: PostGIS WKB → lat/lng conversion, Nairobi bounding box filtering, Kenyan phone normalisation (E.164), Pydantic schema validation for all seven entity types including cross-validator business rules, and Silver cleaning transformations.
Run dbt tests independently:
cd dbt_project && dbt testsouq-etl/
├── src/
│ ├── ingestion/
│ │ ├── souq_connector.py # Souq PostGIS extractor, WKB → lat/lng, CDC + batch
│ │ └── imports_connector.py # Affordable Imports extractor, payment scrubbing
│ ├── transformations/
│ │ ├── geo.py # WKB parsing, bounding box filter, zone snapping
│ │ ├── cleaner.py # Null standardisation, type casting, dedup
│ │ ├── deduplicator.py # Exact-key and content-hash deduplication
│ │ └── schemas.py # Pydantic v2 schemas: 7 entity types, domain validators
│ ├── api/
│ │ ├── main.py # FastAPI app
│ │ ├── routes.py # /status, /runs, /trigger
│ │ └── auth.py # Constant-time Bearer token validation
│ └── utils/
│ ├── db.py # asyncpg pool, DSN never logged
│ ├── duckdb_utils.py # Staging + analytical DuckDB connections
│ └── logging.py # structlog JSON + credential scrubbing
├── dbt_project/
│ └── models/gold/
│ ├── demand_by_zone.sql # Spatial demand grid, peak hours, fulfilment rate
│ ├── provider_performance.sql # Lifetime metrics, quality/value tiers
│ ├── cross_platform_customers.sql # Phone-matched cross-app LTV
│ ├── booking_funnel.sql # Request → complete funnel, drop-off rates
│ └── ecommerce_daily.sql # GMV, M-Pesa/Paystack split, new vs returning
├── prefect_flows/
│ ├── full_pipeline.py # Bronze → Silver → Gold with Slack alerting
│ ├── souq_bronze_flow.py # Souq ingestion flow
│ ├── imports_bronze_flow.py # Affordable Imports ingestion flow
│ ├── silver_flow.py # Transformation + validation flow
│ └── gold_flow.py # dbt run + test execution
├── tests/
│ ├── unit/
│ │ ├── test_geo.py # WKB parsing, Nairobi filter
│ │ ├── test_cleaner.py # Polars cleaning transformations
│ │ └── test_schemas.py # All 7 Pydantic schemas + cross-validators
│ └── integration/
│ └── test_api.py # FastAPI status + trigger endpoints
├── migrations/
│ └── 001_init.sql # Warehouse schemas, Gold stubs, metadata table
├── docker-compose.yml
└── .env.example # All variables documented inline
For detailed guides and project purpose, API references, and troubleshooting, please explore the Full reference in docs/.