AmanPay is a cardless biometric banking platform: a person enrolls once with multimodal biometrics (face + fingerprint + voice) and is thereafter recognized and linked to their bank accounts, authorizing payments purely by biometric verification — no physical card, no card numbers. Built-in liveness / anti-spoofing (PAD) and deepfake / injection detection visibly reject fake biometrics.
Ships with ImageNet-pretrained backbones so the full pipeline runs end-to-end out of the box, and trains to real benchmark accuracy on standard datasets. The cardless product plan, data model, and app-store checklist live in results/CARDLESS_PLAN.md; extensibility and the research roadmap live in ARCHITECTURE.md.
flowchart LR
subgraph capture["📲 Capture (on device)"]
F["😀 Face<br/>MTCNN align"]
P["👆 Fingerprint<br/>CLAHE enhance"]
V["🎙️ Voice<br/>log-mel"]
end
subgraph encode["🧠 Encoders — MobileNetV3-Large"]
FE["Face encoder<br/>512-d + liveness"]
PE["FP encoder<br/>512-d + PAD"]
VE["Voice encoder<br/>512-d + PAD"]
end
FUSE["⚖️ Quality-adaptive<br/>fusion → unified 512-d"]
DEC{"🔐 Decision<br/>cosine vs. template<br/>gated by liveness / PAD / deepfake"}
PAY["💳 Cardless authorize<br/>SCA: biometric + device passkey<br/>SPC-signed → network token"]
F --> FE --> FUSE
P --> PE --> FUSE
V --> VE --> FUSE
FUSE --> DEC -->|pass| PAY
DEC -->|fail / spoof| REJECT["🚫 Reject"]
classDef cap fill:#1e3a8a,stroke:#93c5fd,color:#fff;
classDef enc fill:#065f46,stroke:#6ee7b7,color:#fff;
classDef fuse fill:#6d28d9,stroke:#c4b5fd,color:#fff;
classDef dec fill:#b45309,stroke:#fcd34d,color:#fff;
classDef pay fill:#0f766e,stroke:#5eead4,color:#fff;
classDef rej fill:#991b1b,stroke:#fca5a5,color:#fff;
class F,P,V cap;
class FE,PE,VE enc;
class FUSE fuse;
class DEC dec;
class PAY pay;
class REJECT rej;
| Stage | Component |
|---|---|
| Face | MTCNN alignment → MobileNetV3-Large → 512-d embedding + liveness head |
| Fingerprint | CLAHE enhancement → MobileNetV3-Large + minutiae attention → 512-d embedding + PAD head |
| Voice | Log-mel spectrogram → MobileNetV3-Large → 512-d embedding + PAD head |
| Fusion | Quality-adaptive score fusion across the present modalities → unified decision (graceful when one is missing) |
| Decision | Cosine similarity vs. enrolled template, gated by liveness / PAD / deepfake thresholds |
| Cardless wallet | Consent (BIPA/GDPR) → link account as a network token (PAN never stored) → authorize via PSD2 SCA (biometric inherence + device-passkey possession), SPC-signed to the transaction |
The cardless model is industry-aligned — Visa/Mastercard Payment Passkey, Secure Payment Confirmation, Amazon One all use on-device biometric → device passkey → network token. AmanPay's edge is multimodal + cancelable ISO/IEC 24745 templates + deepfake defense vs. incumbents' single-modal palm/face.
Full module map, data flow, and extension points: ARCHITECTURE.md.
git clone https://github.com/MHHamdan/AmanPay.git && cd AmanPay
pip install -r requirements.txt # or: conda env create -f environment.yml
pip install -e .
cp .env.example .env # then edit .env (see Configuration below)
# Get the trained encoder weights (see "Model weights" — pick ONE):
python scripts/hf_weights.py download # from Hugging Face (recommended)
# Launch (serves the web app at / and the API at /docs):
uvicorn api.main:app --host 0.0.0.0 --port 8000Open http://localhost:8000/, go to the Pay tab, click ▶ Load demo — it enrolls a bundled sample identity and fills the form so you can pay in one click. (For camera / microphone / WebAuthn, use the HTTPS launch below.)
No weights? It still runs. On first launch AmanPay auto-downloads the hosted weights (see the fallback chain below); if that's disabled or offline it falls back to ImageNet-pretrained backbones so the whole pipeline still works end-to-end (at untrained accuracy). Trained weights auto-load from
checkpoints/— no code change.
To share a live URL during development (and as the API backend for the future Android/iOS apps), see deploy/DEPLOY.md. The current setup exposes the app on this server through a Cloudflare Tunnel (valid HTTPS, so camera/mic/WebAuthn work), managed by a systemd user service:
systemctl --user start amanpay-quicktunnel # public HTTPS tunnel -> local :8000
./deploy/current-url.sh # prints the current public URLQuick-tunnel URLs rotate on restart; for a stable URL bound to your own domain, use the named-tunnel path in deploy/DEPLOY.md. A ready-to-ship Hugging Face Docker Space bundle (free, always-on, CPU) lives in deploy/space/ as an alternative.
The trained MobileNetV3 encoders (face, fingerprint, voice) + PAD/deepfake heads are ~90 MB and are not committed to git. Get them one of three ways:
# 1) Download from the Hugging Face Hub (public repo, no token needed):
python scripts/hf_weights.py download # -> checkpoints/
# 2) Train them yourself on open benchmarks (see "Train & benchmark" below):
python scripts/exp_face.py --epochs 15
python scripts/exp_socofing.py --epochs 8 --minutiae on
python scripts/train_voice.py --config configs/base.yaml
# 3) Maintainers: push newly-trained checkpoints (needs a WRITE token in .env):
python scripts/hf_weights.py uploadWeights are hosted at MHamdan/amanpay-encoders
(override with AMANPAY_HF_REPO). A public repo downloads with no token; a
private repo or uploading needs HF_TOKEN set in .env.
At startup each encoder takes the best weights available, in order — so AmanPay runs anywhere, no manual step required:
- Local checkpoint in
checkpoints/— your trained weights (highest priority). - Auto-download the hosted weights from
AMANPAY_HF_REPOif a checkpoint is missing (setAMANPAY_HF_AUTO_DOWNLOAD=0to disable). A fresh clone becomes fully functional on first launch with nodownloadstep. - Third-party HF backbone — if an encoder still has no trained weights and
AMANPAY_HF_BACKBONEis set (e.g.timm/mobilenetv3_large_100.ra_in1korrepo:filename), its MobileNetV3-Large backbone is initialised from that Hub repo (shape-matching keys,strict=False). - ImageNet-pretrained backbone (torchvision) — the final fallback, always available.
Copy .env.example → .env. Recognised variables:
| Variable | Purpose | Default |
|---|---|---|
HF_TOKEN |
Hugging Face token (download private / upload weights) | — |
AMANPAY_HF_REPO |
Hub repo holding the weights | MHamdan/amanpay-encoders |
AMANPAY_HF_AUTO_DOWNLOAD |
auto-pull hosted weights when a checkpoint is missing | 1 |
AMANPAY_HF_BACKBONE |
third-party HF MobileNetV3 backbone for untrained encoders (repo or repo:filename) |
— |
AMANPAY_FACE_CKPT / AMANPAY_FP_CKPT / AMANPAY_VOICE_CKPT |
checkpoint paths | checkpoints/… |
AMANPAY_DEEPFAKE_CKPT |
deepfake/injection detector | checkpoints/deepfake_detector.pkl |
AMANPAY_MATCH_MODE |
fusion match mode (score | learned) |
score |
AMANPAY_RP_ID / AMANPAY_ORIGIN |
WebAuthn relying-party (set to your domain in prod) | localhost |
.env is gitignored — never commit real tokens.
import torch
from amanpay.config import AmanPayConfig
from amanpay.models.authenticator import AmanPayAuthenticator
auth = AmanPayAuthenticator(AmanPayConfig()).eval()
face, fp = torch.randn(1, 3, 224, 224), torch.randn(1, 3, 224, 224)
auth.enroll("alice", face, fp)
print(auth.authenticate("alice", face, fp))The FastAPI server serves both the REST API and the web app (the cardless
banking UI at /). Trained encoders auto-load from checkpoints/ if present.
1) Local / HTTP — fastest for API work and desktop testing:
uvicorn api.main:app --host 0.0.0.0 --port 8000
# Web app: http://localhost:8000/
# API docs: http://localhost:8000/docs2) HTTPS — required for the Pay flow's camera / microphone / WebAuthn
(browsers only expose these in a secure context). Uses the self-signed cert in
certs/ (regenerate if absent — include the machine IP in the SAN so mobile
browsers accept it):
mkdir -p certs && openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
-keyout certs/key.pem -out certs/cert.pem -subj "/CN=AmanPay" \
-addext "subjectAltName=IP:172.24.50.21,DNS:localhost"uvicorn api.main:app --host 0.0.0.0 --port 8443 \
--ssl-keyfile certs/key.pem --ssl-certfile certs/cert.pem
# Open https://<your-ip>:8443/ (accept the self-signed cert once)On this host the web app is live at https://172.24.50.21:8443/ — use the machine IP, not
localhost, for the demo (see the access-IP note).
3) Managed (systemd user services) — how it runs on this box; auto-restarts:
systemctl --user restart amanpay-api # HTTP on :8000
systemctl --user restart amanpay-https # HTTPS on :8443
systemctl --user status amanpay-api amanpay-httpsInstall as an app (PWA). The web app is an installable Progressive Web App
(manifest.json + service worker). Open the HTTPS URL on Android/desktop Chrome →
Install / Add to Home Screen; on iOS Safari → Share → Add to Home Screen.
For App Store / Play submission, wrap the same UI with Capacitor to add native
biometric/camera APIs (see CARDLESS_PLAN.md §4).
| Group | Endpoints |
|---|---|
| Auth (2-modal) | POST /enroll · POST /authenticate · POST /verify |
| Unified (tri-modal) | POST /unified/enroll · POST /unified/authenticate |
| Voice factor | POST /voice/enroll · POST /voice/verify |
| Cardless wallet | POST /wallet/consent · POST /wallet/link · GET /wallet/accounts · GET /wallet/transactions · POST /wallet/pay (risk-adaptive: frictionless vs. step-up) |
| Active liveness (Flashmark-style) | POST /pad/challenge · POST /pad/verify |
| Device biometric (Touch ID / Face ID / fingerprint) | POST /device/verify (WebAuthn platform authenticator) |
| Out-of-band confirmation (SMS/email/push, user pref) | POST/GET /notify/prefs · POST /notify/subscribe · POST /notify/respond |
| Passkey / WebAuthn | POST /passkey/{register,challenge,authenticate} · POST /webauthn/{register,authenticate}/{begin,complete} |
| Liveness challenge | POST /liveness/{challenge,verify} |
| Demo | POST /demo/seed (one-click enroll + link + sample payment) |
| Ops | GET /health · GET /report-card · GET /users · DELETE /users/{user_id} |
Images/audio are sent base64-encoded (data-URI accepted). Full schemas at /docs.
python demo/app.py # Gradio UI at http://localhost:7860The registry in amanpay/data/benchmarks.py drives acquisition; see
data/README.md for the data card and layouts.
python scripts/download_data.py --list # status of every dataset
python scripts/download_data.py --fetch socofing # Kaggle (needs token)
python scripts/download_data.py --instructions livdet2015 # license-gated steps| Dataset | Modality / task | Status |
|---|---|---|
| CASIA-WebFace | face recognition (train) | ✅ in use |
| LFW / CFP-FP / AgeDB-30 | face verification (eval) | ✅ in use |
| CASIA-FASD | face PAD | ✅ normalized |
| SOCOFing | fingerprint recognition | ✅ in use |
| FVC2004 DB_B | fingerprint recognition | ✅ in use |
| Replay-Attack · LivDet-2015 · FVC DB_A | PAD / recognition | ⏳ pending upload |
Research/benchmark runs (real datasets, standard protocols):
python scripts/exp_face.py --epochs 15 # CASIA-WebFace → LFW
python scripts/exp_socofing.py --epochs 8 --minutiae on # SOCOFing open-set
python scripts/exp_fvc.py # FVC2004 C2 ablation
python scripts/make_report.py # aggregate results/RESULTS.md
python scripts/run_benchmark.py --dataset lfw --modality face # any registry protocolGeneric single-encoder trainers (ImageFolder layout data/faces/<identity>/*.png):
python scripts/train_face.py --config configs/base.yaml --data_root data/faces
python scripts/train_fingerprint.py --config configs/base.yaml --data_root data/fingerprints
python scripts/train_fusion.py --config configs/base.yaml \
--face_ckpt checkpoints/face_encoder_best.pt --fp_ckpt checkpoints/fingerprint_encoder_best.ptReal runs on open benchmarks (full table in results/RESULTS.md;
tracked in PR #1 on branch research/benchmark-training-paper).
| Task | Benchmark | Result |
|---|---|---|
| Face recognition | CASIA-WebFace → LFW (6000 pairs) | EER 0.311 → 0.0175 (ArcFace, 15 epochs; TAR@FAR1e-2 = 97.4%) |
| C2 minutiae attention | FVC2004 DB_B (real impressions) | minutiae on 0.179 vs off 0.191 EER → Δ +0.013, helps |
| Fingerprint recognition | SOCOFing (open-set) | EER 0.236 → 0.000 (saturates — protocol too easy) |
Paper contributions (current status): C1 multimodal fusion — fusion beats single-modal, validated on genuine NIST BSSR1 (score-level EER 0.6–1.6% vs 4–9%); learned deep fusion does not beat score-level fusion at ~100-subject scale (honest, systematic result). C2 minutiae attention helps on real FVC impressions (Δ EER +0.013). C3 joint liveness — naive CNN overfits cross-subject; see roadmap.
Beyond the research benchmarks, AmanPay ships production-grade, standards-aligned capabilities that differentiate it from single-modal device-bound passkeys (Visa/Mastercard) and server-side liveness vendors:
| Capability | Standard / competitor | Status |
|---|---|---|
| Cancelable BioHash templates — revocable, non-invertible, unlinkable | ISO/IEC 24745 (BioHash is a Table C.1 PI method) | ✅ quantified: D_sys 0.07, bit-entropy 0.998 |
| Passive deepfake / injection detection | FIDO Biometric PAD / ISO 30107-3 | ✅ EER 1.3% → FIDO PAD Level 2 (proxy) |
| Active screen-flash liveness (Flashmark-style) | CEN/TS 18099 IAD target (borrowed from iProov) | ✅ /pad/* — one-time colour challenge, reflection-verified |
| Risk-based step-up — frictionless vs. challenge | PSD2/EMV 3DS SCA exemptions (borrowed from Visa) | ✅ fusion confidence as risk input; auto-escalates to active liveness |
| Device-bound biometric passkey | FIDO2 / WebAuthn | ✅ Ed25519 possession + multimodal inherence |
| On-device inference — no server-side biometric | GDPR / BIPA data-minimization | ✅ ONNX ~18 MB/encoder |
| Quality-aware abstain + calibrated confidence | operational robustness | ✅ |
Standards report card — one call returns recognition + PAD/FIDO level + ISO-24745 conformance + deployment facts:
curl http://<host>:8000/report-cardDevice-bound biometric passkey flow (POST /passkey/{register,challenge,authenticate}):
possession (Ed25519 device key, never leaves the device) + inherence
(multimodal biometric gates signing), single-use challenges block replay. Only the
public key + signature transit — no biometric is stored server-side.
Details & competitor map: results/RESEARCH_competitive_enhancements.md (local).
pytest tests/ -v --cov=amanpaypodman build -t amanpay .
podman run -p 8000:8000 amanpay # REST API
podman run -p 7860:7860 amanpay python demo/app.py # Demo UI
# or: podman-compose upamanpay/evaluation/metrics.py implements EER, TAR@FAR, DET curves, and PAD
rates (BPCER / APCER / ACER) following ISO/IEC 30107-3 conventions.
In scope: face encoder, fingerprint encoder, fusion, liveness/PAD, REST API, demo UI. Out of scope (roadmap): federated learning, blockchain/ZKP, temporal GNN fraud detection.