This repository bridges OLTP and OLAP workflows using MongoDB as a NoSQL intermediary, offering a clean local + cloud interface to load and explore Supabase-style JSON exports.

It supports full-stack deployment across DEV (local) and PROD (Render + Streamlit Cloud) environments using a Makefile-based pipeline and environment-driven logic.

📥 Download Demo Video (1.2MB)
Interactive MongoDB dashboard: JSON ingestion → Analytics → Streamlit visualization

� How to watch: Click the thumbnail or download link above to save the video locally and watch the complete demo

git clone https://gitlab.com/stripe_b2/nosql.git
cd nosql

We recommend uv for fast dependency installs:

uv venv
source .venv/bin/activate
uv sync

Run make help to list all available targets.

make all ENV=DEV

This starts:

• MongoDB via Docker
• A data load from GCS (or local)
• FastAPI backend via Uvicorn
• Streamlit dashboard in a new tab

make prod_deploy ENV=PROD

This runs:

• Supabase → MongoDB data ingestion
• Git push to GitHub (for Streamlit Cloud triggers)

Explore your database manually:

make mongosh

To learn manual connection URIs, example aggregation queries, and how to debug your collections:

👉 Read 📄 MongoDB Shell & Query Cheatsheet

The backend is environment-aware (ENV=DEV|PROD) and connects to either local Mongo or Atlas. It exposes:

• /customers, /customers/{id}
• /subscriptions/active
• /charges/fraud
• /payment_intents/3ds

Run locally:

make api

The Streamlit app reads from your backend API and lets you:

• Inspect customers, subscriptions, payment intents, and fraud patterns
• Query by endpoint
• Visualize 3DS usage and suspicious charges

Run locally:

make ui

The primary data ingestion script:

• Downloads the latest Supabase-style db_dump_prod_*.json from GCS
• Parses JSON by collection
• Writes to MongoDB

Run standalone:

ENV=PROD python scripts/gcs_to_mongo.py

At the root of the repo, create a file named .env with:

ENV=DEV
MONGO_URI=mongodb://localhost:27017
GCP_CREDS_FILE=your-raw-json-credentials-content-here

If you're using MongoDB Atlas in production:

ENV=PROD
MONGO_URI=mongodb+srv://<user>:<pass>@cluster.mongodb.net/supabase_snapshot

🧬 These variables are loaded automatically by python-dotenv and are used across scripts and Streamlit/FastAPI.

To enable CI/CD pipelines on GitLab to push to GitHub (for example to trigger Streamlit Cloud redeploys), follow these steps:

Run locally:

ssh-keygen -t rsa -b 4096 -C "gitlab-ci@nosql" -f deploy_key_gitlab_to_github_rsa

This creates two files:

• deploy_key_gitlab_to_github_rsa (private key)
• deploy_key_gitlab_to_github_rsa.pub (public key)

• Go to GitHub → Your repo → Settings → Deploy Keys
• Click "Add deploy key"
• Title: GitLab CI deploy
• Paste the contents of deploy_key_gitlab_to_github_rsa.pub
• ✅ Enable “Allow write access”

• Go to GitLab → Settings → CI/CD → Variables
• Add a new variable:

❗ Must be a file-type variable for it to work with ssh-agent.

Your pipeline should include:

before_script:
  - eval "$(ssh-agent -s)"
  - mkdir -p ~/.ssh
  - echo "$GITHUB_SSH_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa
  - ssh-keyscan github.com >> ~/.ssh/known_hosts

In your Makefile, ensure the push command uses:

git push github HEAD:main

🛡️ SSH setup is only required if you're triggering GitHub deploys from GitLab CI via make all ENV=PROD.

• 🥪 MongoDB Shell & Query Cheatsheet — manual queries & shell usage
• 💃 Integration Guide (Mongo + FastAPI + Streamlit) — fullstack architecture, local & cloud setup

Supabase JSON (GCS/local)
         ↓
      MongoDB
         ↓
    FastAPI Backend
         ↓
   Streamlit Frontend

Want to contribute or adapt this setup to your own OLAP/OLTP bridge? PRs welcome ✨