┌────────────────────────────────────────────────────────────────────────────────────┐
│ INTERNET │
└──────────────────────────────────────┬─────────────────────────────────────────────┘
│ HTTP Requests
▼
┌────────────────────────────────────────────────────────────────────────────────────┐
│ API GATEWAY (NGINX) │
│ ┌──────────────────────────────────────────────────────────────────────────────┐ │
│ │ • Rate Limiting (100 req/s general, 50 req/s chat endpoints) │ │
│ │ • Load Balancing (Least Connections) │ │
│ │ • Health Checks │ │
│ │ • Request Routing │ │
│ └──────────────────────────────────────────────────────────────────────────────┘ │
└────────┬──────────────────────────────────────────────────────────────────┬────────┘
│ │
│ /applications/* │ /applications/:token/chats/*
│ (CRUD operations) │ (High-throughput endpoints)
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ RAILS API │ │ GO CHAT SERVICE │
│ (Ruby on Rails) │ │ (Golang/Fiber) │
│ ┌───────────────────┐ │ │ ┌───────────────────┐ │
│ │ Applications │ │ │ │ Chat Handler │ │
│ │ Controller │ │ │ │ Message Handler │ │
│ │ │ │ │ │ Search Handler │ │
│ │ • Create App │ │ │ │ │ │
│ │ • List Apps │ │ │ │ • Create Chat │ │
│ │ • Get App │ │ │ │ • Create Message │ │
│ │ • Update App │ │ │ │ • Search Messages │ │
│ └─────────┬─────────┘ │ │ └─────────┬─────────┘ │
└────────────┼────────────┘ └────────────┼────────────┘
│ │
│ Read/Write │ Atomic INCR
│ │ Publish
▼ ▼
┌────────────────────────────────────────────────────────────────────────────────────┐
│ DATA LAYER │
│ ┌───────────────────┐ ┌───────────────────┐ ┌────────────────────────────────┐ │
│ │ MySQL │ │ Redis │ │ RabbitMQ │ │
│ │ │ │ │ │ │ │
│ │ • applications │ │ • Counter: │ │ Queues: │ │
│ │ • chats │ │ app:xyz:chats │ │ ┌──────────────────┐ │ │
│ │ • messages │ │ app:xyz:chat:42 │ │ │ chats_queue │ ← Durable│ │
│ │ │ │ :messages │ │ ├──────────────────┤ │ │
│ │ Source of Truth │ │ │ │ │ messages_queue │ ← Durable│ │
│ │ (ACID) │ │ • Delta: │ │ ├──────────────────┤ │ │
│ │ │ │ delta:app:5 │ │ │ indexing_queue │ ← Durable│ │
│ │ Indexed: │ │ :chats │ │ └──────────────────┘ │ │
│ │ • app token │ │ delta:chat:1337 │ │ │ │
│ │ • chat (app, num) │ │ :messages │ │ Persistent Storage │ │
│ │ • msg (chat, num) │ │ │ │ (Survives restarts) │ │
│ └───────────────────┘ │ • Cache: │ │ │ │
│ │ app:token:xyz │ └────────────────────────────────┘ │
│ │ │ │
│ │ • Lock: │ │
│ │ lock: │ │
│ │ reconciliation │ │
│ └───────────────────┘ │
└────────────────────────────────────────────────────────────────────────────────────┘
▲
│ Consume Messages
│ (3 Workers)
│
┌───────────────────────────────────────────────────────────────────────────────────┐
│ GO WORKER SERVICE (Background) │
│ │
│ ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────────┐ │
│ │ Chat Worker │ │ Message Worker │ │ Indexing Worker │ │
│ │ │ │ │ │ │ │
│ │ Consumes: │ │ Consumes: │ │ Consumes: │ │
│ │ chats_queue │ │ messages_queue │ │ indexing_queue │ │
│ │ │ │ │ │ │ │
│ │ Actions: │ │ Actions: │ │ Actions: │ │
│ │ 1. Validate app │ │ 1. Validate app/ │ │ 1. Batch messages │ │
│ │ 2. Create chat │ │ chat │ │ (1000 or 5 sec) │ │
│ │ in MySQL │ │ 2. Create message │ │ 2. Bulk index to ES │ │
│ │ 3. Increment │ │ in MySQL │ │ 3. ACK messages │ │
│ │ delta:app:X │ │ 3. Queue for │ │ │ │
│ │ :chats │ │ indexing │ │ │ │
│ │ 4. ACK message │ │ 4. Increment │ │ │ │
│ │ │ │ delta:chat:Y │ │ │ │
│ │ Idempotent: │ │ :messages │ │ │ │
│ │ Checks if exists │ │ 5. ACK message │ │ │ │
│ │ │ │ │ │ │ │
│ └──────────────────────┘ └──────────────────────┘ └──────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐ │
│ │ Reconciliation Worker (Periodic) │ │
│ │ │ │
│ │ Runs: Every 15 seconds │ │
│ │ Lock: Distributed lock (only 1 instance runs at a time) │ │
│ │ │ │
│ │ Actions: │ │
│ │ 1. Scan Redis for delta:app:*:chats keys │ │
│ │ 2. Atomic GET+DELETE (Lua script) │ │
│ │ 3. UPDATE applications SET chats_count = chats_count + delta │ │
│ │ 4. Scan Redis for delta:chat:*:messages keys │ │
│ │ 5. UPDATE chats SET messages_count = messages_count + delta │ │
│ │ 6. Release lock │ │
│ │ │ │
│ │ Purpose: Sync Redis delta counters to MySQL (eventual consistency) │ │
│ │ Lag: Maximum 15 seconds (acceptable per requirements: < 1 hour) │ │
│ └─────────────────────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────────────┘
│
│ Bulk Index
▼
┌─────────────────────┐
│ Elasticsearch │
│ │
│ Index: messages │
│ │
│ Mapping: │
│ • app_token │
│ • chat_number │
│ • message_number │
│ • content (ngram) │
│ • created_at │
│ │
│ Features: │
│ • Routing by │
│ app:chat │
│ • Fuzzy search │
│ • Partial match │
│ • Pagination │
│ │
└─────────────────────┘
What happens when a client sends a message? Let's trace the journey:
Time Component Action
────────────────────────────────────────────────────────────────────────────
T0 Client POST /applications/xyz/chats/42/messages
Body: {"content": "Hello World"}
T1 API Gateway • Check rate limit (50 req/s)
• Route to Go Chat Service
T2 Go Chat Service • Parse request
• Validate: app_token = "xyz", chat_number = 42
T3 Go Chat → Redis INCR app:xyz:chat:42:messages_count
↓ Returns: 15 (new message number)
T4 Go Chat → RabbitMQ PUBLISH to messages_queue:
{
"app_token": "xyz",
"chat_number": 42,
"message_number": 15,
"content": "Hello World"
}
T5 Go Chat → Client HTTP 200 OK
{"number": 15}
────────────────────────────────────────────────────────────────────────────
Total Time: 5-15ms (User sees response here!)
────────────────────────────────────────────────────────────────────────────
Background Processing (Asynchronous):
T10 Message Worker • Pull message from messages_queue
• Find Application by token "xyz"
• Find Chat by (app_id, number=42)
• Check if message #15 exists (idempotency)
T11 Message Worker → MySQL INSERT INTO messages (chat_id, number, content)
VALUES (1337, 15, "Hello World")
T12 Message Worker → Redis INCR delta:chat:1337:messages
(Track delta for reconciliation)
T13 Message Worker → Queue PUBLISH to indexing_queue:
{
"message_id": 9999,
"application_token": "xyz",
"chat_number": 42,
"message_number": 15,
"content": "Hello World",
"created_at": "2025-11-11T10:30:00Z"
}
T14 Message Worker ACK message (remove from messages_queue)
────────────────────────────────────────────────────────────────────────────
Background Time: 100-300ms
────────────────────────────────────────────────────────────────────────────
Indexing (Further Background):
T20 Indexing Worker • Collect messages in batch (1000 msgs or 5 sec)
• Build bulk index request
T25 Indexing Worker → ES POST /messages/_bulk?routing=xyz:42
(Bulk index 1000 messages)
T26 Indexing Worker ACK all messages in batch
────────────────────────────────────────────────────────────────────────────
Indexing Time: 500-2000ms
Message is now searchable!
────────────────────────────────────────────────────────────────────────────
Reconciliation (Every 15 seconds):
T30 Reconciliation Worker • Acquire distributed lock
• Scan: delta:chat:1337:messages = 10
• Atomic GET+DELETE (Lua script)
T31 Reconciliation → MySQL UPDATE chats
SET messages_count = messages_count + 10
WHERE id = 1337
T32 Reconciliation Worker Release lock
────────────────────────────────────────────────────────────────────────────
Counter in MySQL is now synced!
────────────────────────────────────────────────────────────────────────────
Key Insight: The client gets a response in ~10ms, but full persistence + indexing takes ~2 seconds. Is this acceptable? For a chat system, absolutely! Users don't care if their message is on disk yet, they just want confirmation it was received.
- Docker & Docker Compose
- 8GB RAM (recommended)
- Ports 8080, 3306, 6379, 5672, 9200, 15672 available
# Clone the repository
git clone <your-repo>
cd chat-system
# Start everything with a single command!
docker-compose up
# Wait for all services to be healthy (30-60 seconds)
# You'll see logs from all servicesThat's it! The system is now running on http://localhost:8080
http://localhost:8080
None required for this demo. In production, add JWT/OAuth2.
curl -X POST http://localhost:8080/applications \
-H "Content-Type: application/json" \
-d '{
"application": {
"name": "My Chat App"
}
}'Response:
{
"token": "unique-token-12345",
"name": "My Chat App"
}Note: The token is auto-generated and used to identify this application in all future requests.
curl http://localhost:8080/applications?page=1&per_page=20Response:
{
"data": [
{
"id": 1,
"token": "unique-token-12345",
"name": "My Chat App",
"chats_count": 42,
"created_at": "2025-11-11T10:00:00Z"
}
],
"meta": {
"page": 1,
"per_page": 20,
"total": 1,
"total_pages": 1
}
}curl http://localhost:8080/applications/unique-token-12345Response:
{
"id": 1,
"token": "unique-token-12345",
"name": "My Chat App",
"chats_count": 42,
"created_at": "2025-11-11T10:00:00Z"
}Note: This response is cached in Redis for 30 minutes for faster retrieval.
curl -X POST http://localhost:8080/applications/unique-token-12345/chatsResponse:
{
"number": 1,
"status": "processing"
}Numbering: Chats are numbered sequentially starting from 1 for each application.
Status: "processing" means the chat is queued for persistence. It will be in the database within milliseconds.
curl -X POST http://localhost:8080/applications/unique-token-12345/chats/1/messages \
-H "Content-Type: application/json" \
-d '{
"content": "Hello, World!"
}'Response:
{
"number": 1
}Numbering: Messages are numbered sequentially starting from 1 for each chat.
# Basic search
curl "http://localhost:8080/applications/unique-token-12345/chats/1/messages/search?query=hello"
# With pagination
curl "http://localhost:8080/applications/unique-token-12345/chats/1/messages/search?query=hello&page=1&per_page=20"Response:
{
"total": 156,
"total_pages": 8,
"page": 1,
"per_page": 20,
"messages": [
{
"message_number": 1,
"content": "Hello, World!",
"created_at": "2025-11-11T10:30:00Z"
},
{
"message_number": 5,
"content": "Hello again!",
"created_at": "2025-11-11T10:35:00Z"
}
]
}Search Features:
- Partial matching: "hel" matches "hello", "help", "helicopter"
- Fuzzy matching: "helo" matches "hello" (typo tolerance)
- Case insensitive: "HELLO" matches "hello"
- Pagination: Default per_page = 20
| Service | Technology | Purpose | Port |
|---|---|---|---|
| API Gateway | NGINX | Load balancing, rate limiting, routing | 80 |
| Rails API | Ruby 3.3 + Rails 8 | Application CRUD operations | 3000 |
| Go Chat Service | Go 1.24 + Fiber | High-throughput chat/message creation | 8080 |
| Worker | Technology | Purpose |
|---|---|---|
| Go Worker | Go 1.24 + RabbitMQ | Message queue consumer, background persistence |
| Store | Technology | Purpose |
|---|---|---|
| MySQL 8 | Relational DB | Source of truth, ACID transactions |
| Redis 7 | In-memory cache | Atomic counters, caching, distributed locks |
| RabbitMQ 3 | Message queue | Async job processing, guaranteed delivery |
| Elasticsearch 8.11 | Search engine | Full-text search with fuzzy matching |
- Docker Compose for orchestration
- Health checks for all services
- Persistent volumes for data durability
| Metric | Value |
|---|---|
| Message creation latency | 5-15ms (p95) |
| Search latency | 10-50ms (p95) |
| Messages/second | 1000+ (single worker) |
| Concurrent connections | 10,000+ (Go) |
| Database write throughput | 200 writes/sec (MySQL) |
-
MySQL writes: ~200-500 writes/sec (single instance)
- Solution: Add read replicas, shard by application_id
-
Redis atomic INCR: ~100k ops/sec (single instance)
- Solution: Redis cluster (not needed until 10k+ req/s)
-
Elasticsearch indexing: ~1000 docs/sec (bulk)
- Solution: More shards, more nodes
# 1. Create application
TOKEN=$(curl -s -X POST http://localhost:8080/applications \
-H "Content-Type: application/json" \
-d '{"application": {"name": "Test App"}}' | jq -r '.token')
echo "Application Token: $TOKEN"
# 2. Create chat
CHAT=$(curl -s -X POST http://localhost:8080/applications/$TOKEN/chats | jq -r '.number')
echo "Chat Number: $CHAT"
# 3. Create message
MSG=$(curl -s -X POST http://localhost:8080/applications/$TOKEN/chats/$CHAT/messages \
-H "Content-Type: application/json" \
-d '{"content": "Hello from the test!"}' | jq -r '.number')
echo "Message Number: $MSG"
# 4. Wait for indexing (5 seconds)
sleep 5
# 5. Search
curl "http://localhost:8080/applications/$TOKEN/chats/$CHAT/messages/search?query=hello" | jq# Install Apache Bench
sudo apt-get install apache2-utils
# Create 1000 messages (10 concurrent)
ab -n 1000 -c 10 -p message.json -T application/json \
http://localhost:8080/applications/$TOKEN/chats/$CHAT/messages
# message.json:
# {"content": "Load test message"}Expected Results:
- Throughput: 1000-2000 req/s
- Latency (p50): 5-10ms
- Latency (p95): 15-30ms
- Docker containerization
- Health checks for all services
- Structured logging
- Error handling and retries
- Idempotent workers
- Rate limiting
- Database indexes
- Authentication/Authorization (TODO)
- Metrics (Prometheus) (TODO)
- Tracing (Jaeger) (TODO)
- Automated tests (TODO)
- CI/CD pipeline (TODO)
- Backup strategy (TODO)