Big Data Analytics Infrastructure - Docker Swarm Setup

This setup creates a private Big Data processing cluster with web-based interactive development environment.

Architecture

• Spark: Unified engine for large scale data analytics
• JupyterHub: Multi-user interactive development environment
• Caddy: Reverse proxy with automatic HTTPS and Tailscale integration
• Docker Registry V2: Private image registry with basic authentication
• Registry UI (Joxit): Web interface for browsing and managing registry images
• MinIO: S3-compatible object storage with multi-drive support

URLs

• Registry API: https://registry.custom.dev/v2/
• Registry UI: https://registry-ui.custom.dev
• MinIO API: https://minio.custom.dev
• MinIO Console: https://minio-console.custom.dev

Prerequisites

1. Docker Swarm initialized
2. NFS server running on myhostname Ubuntu host
3. Caddy installed locally (for generating password hashes)

Setup Instructions

1. Register a Custom Domain and Configure DNS

Sidestepping the need for a private DNS server, register a custom domain name (e.g., customdomain.dev) with a domain registrar. Then add DNS A records pointing to the private IP address of the server where the services will be hosted:

# A records pointing to your server's private IP
A    customdomain.dev              → <server-private-ip>
A    registry.customdomain.dev     → <server-private-ip>
A    registry-ui.customdomain.dev  → <server-private-ip>
A    minio.customdomain.dev        → <server-private-ip>
A    minio-console.customdomain.dev → <server-private-ip>

This setup assumes Cloudflare as the DNS provider, thus Caddy is configured to use the Cloudflare DNS challenge for ACME TLS certificate issuance.

2. Configure NFS Share on myhostname Host

On your Ubuntu host myhostname, set up an NFS export:

# Install NFS server
sudo apt update
sudo apt install nfs-kernel-server

# Create directory for registry data
sudo mkdir -p /srv/nfs/docker-registry

# Set permissions
sudo chown -R nobody:nogroup /srv/nfs/docker-registry
sudo chmod 777 /srv/nfs/docker-registry

# Configure NFS export
echo "/srv/nfs/docker-registry *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports

# Apply changes
sudo exportfs -ra
sudo systemctl restart nfs-kernel-server

3. Update NFS Path in docker-compose.yml

Edit docker-compose.yml and update the NFS device path for the registry_data volume:

volumes:
  registry_data:
    driver: local
    driver_opts:
      type: nfs
      o: addr=myhostname,rw,nfsvers=4
      device: ":/srv/nfs/docker-registry"  # Update this path

4. Generate Basic Auth Password Hash

Install Caddy locally if you haven't already:

# On Ubuntu/Debian
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

# Or use Docker
docker run --rm caddy:2-alpine caddy version

Generate bcrypt hash for your password:

# Replace 'your-secure-password' with your actual password
caddy hash-password --plaintext 'your-secure-password'

# Or using Docker
docker run --rm caddy:2-alpine caddy hash-password --plaintext 'your-secure-password'

This will output a bcrypt hash like:

$2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpBnqQkJnEiL3NnhyZ6V6C2

5. Update Caddyfile with Password Hash

Edit Caddyfile and replace the placeholder password hashes with your generated hash:

basicauth {
    admin $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpBnqQkJnEiL3NnhyZ6V6C2
    # Add more users if needed:
    # user2 $2a$14$another_hash_here
}

Update this in both the registry.custom.dev and registry-ui.custom.dev sections.

6. Generate Docker Secrets for MinIO

MinIO uses Docker Swarm secrets for secure credential management.

Create storage directories:

# On myhostname host - create directories for MinIO data (on HDDs)
sudo mkdir -p /mnt/disk1/minio /mnt/disk2/minio

# Create directory for MinIO config (on SSD)
sudo mkdir -p /srv/minio/config

# Set permissions
sudo chown -R nobody:nogroup /mnt/disk1/minio /mnt/disk2/minio /srv/minio/config
sudo chmod -R 755 /mnt/disk1/minio /mnt/disk2/minio /srv/minio/config

Create Docker secrets:

# Create MinIO root user secret
echo -n "admin" | docker secret create minio_root_user -

# Create MinIO root password secret (use a strong password!)
echo -n "your-secure-minio-password" | docker secret create minio_root_password -

# Verify secrets were created
docker secret ls

Important:

• Use -n flag with echo to prevent trailing newline characters
• Only services that declare the secret can access it
• Never commit secrets to version control

To update secrets later:

# Remove old secret (must remove service first or redeploy after)
docker secret rm minio_root_password

# Create new secret
echo -n "new-password" | docker secret create minio_root_password -

# Redeploy stack
docker stack deploy -c docker-compose.yml stack_name

7. Deploy the Stack

# Initialize Docker Swarm (if not already done)
# IMPORTANT: Initializing swarm with Tailscale IP may not survive reboot as Tailscale IP is not available when Docker process start. Use local IPs for all nodes to ensure proper overlay network connectivity. Mixed IP types (local + Tailscale) may cause overlay network routing issues.

# Get your localhost IP:
hostname -I | awk '{print $1}'

# Initialize swarm with Tailscale IP:
docker swarm init --advertise-addr <localhost-ip> --listen-addr <localhost-ip>:2377

# To get the command for joining worker nodes to the swarm:
docker swarm join-token worker

# To get the command for joining manager nodes to the swarm:
docker swarm join-token manager

# When joining worker nodes, also specify the localhost IP. On worker node, get localhost IP first then join with:

docker swarm join --advertise-addr <worker-localhost-ip> --token <token> <manager-localhost-ip>:2377

# Login to the private registry (required for pulling images on worker nodes)
docker login https://registry.custom.dev

# Deploy the stack (--with-registry-auth passes credentials to worker nodes)
docker stack deploy -c docker-compose.yml --with-registry-auth stack_name

# Check service status
docker stack services stack_name
docker stack ps stack_name

# View logs
docker service logs stack_name_caddy
docker service logs stack_name_registry
docker service logs stack_name_registry-ui
docker service logs stack_name_minio

Usage

Access the Registry UI

1. Navigate to https://registry-ui.custom.dev
2. Enter your credentials (username: admin, password: what you set)
3. Browse and manage your Docker images

Push Images to Registry

# Login to the registry
docker login https://registry.custom.dev

# Tag an image
docker tag myimage:latest registry.custom.dev/myimage:latest

# Push the image
docker push registry.custom.dev/myimage:latest

Pull Images from Registry

# Login (if not already logged in)
docker login https://registry.custom.dev

# Pull the image
docker pull registry.custom.dev/myimage:latest

Configure Docker Daemon to Trust the Registry

If you encounter certificate issues, you may need to configure Docker to trust the registry:

# Create daemon config directory
sudo mkdir -p /etc/docker

# Add registry to insecure registries (only if not using valid TLS)
# Note: Not recommended for production
sudo tee /etc/docker/daemon.json <<EOF
{
  "insecure-registries": ["registry.custom.dev"]
}
EOF

# Restart Docker
sudo systemctl restart docker

Access MinIO Console

1. Navigate to https://minio-console.custom.dev
2. Login with your MinIO credentials (from Docker secrets)
3. Create buckets and manage objects through the web interface

Use MinIO S3 API

MinIO is S3-compatible and works with AWS CLI and SDKs:

# Install AWS CLI
pip install awscli

# Configure AWS CLI for MinIO
aws configure --profile minio
# AWS Access Key ID: (get from MinIO Console > Access Keys)
# AWS Secret Access Key: (get from MinIO Console > Access Keys)
# Default region: us-east-1
# Default output format: json

# Set MinIO endpoint
export AWS_ENDPOINT_URL=https://minio.custom.dev

# Create a bucket
aws --profile minio --endpoint-url $AWS_ENDPOINT_URL s3 mb s3://my-bucket

# Upload a file
aws --profile minio --endpoint-url $AWS_ENDPOINT_URL s3 cp file.txt s3://my-bucket/

# List objects
aws --profile minio --endpoint-url $AWS_ENDPOINT_URL s3 ls s3://my-bucket/

# Download a file
aws --profile minio --endpoint-url $AWS_ENDPOINT_URL s3 cp s3://my-bucket/file.txt ./downloaded.txt

# Test MinIO API health
curl -vf https://minio.custom.dev/minio/health/live

MinIO Storage Architecture

• Two HDDs: /mnt/disk1/minio and /mnt/disk2/minio
• No erasure coding: Objects distributed across both drives for capacity
• No redundancy: If one drive fails, objects on that drive are lost
• Config on SSD: /srv/minio/config for better performance

Maintenance

Update Services

# Update a specific service
docker service update --image caddy:2-alpine stack_name_caddy

# Update all services
docker stack deploy -c docker-compose.yml stack_name

Backup Registry Data

# On the myhostname NFS host
sudo tar czf registry-backup-$(date +%Y%m%d).tar.gz /srv/nfs/docker-registry

Remove the Stack

docker stack rm stack_name

# Optionally remove volumes (WARNING: This deletes all registry data)
docker volume rm stack_name_registry_data stack_name_caddy_data stack_name_caddy_config

Troubleshooting

Services not starting

# Check service logs
docker service logs stack_name_caddy
docker service logs stack_name_registry
docker service logs stack_name_registry-ui

# Check service details
docker service ps stack_name_caddy --no-trunc

NFS mount issues

# Verify NFS is accessible
showmount -e myhostname

# Test NFS mount manually
sudo mount -t nfs4 myhostname:/srv/nfs/docker-registry /mnt/test

Cannot push/pull images

1. Verify you're logged in: docker login https://registry.custom.dev
2. Check Caddy logs for authentication errors
3. Verify the registry service is running

UI cannot connect to registry

1. Check that REGISTRY_URL in docker-compose.yml matches your domain
2. Verify network connectivity between services
3. Check registry logs for API errors

Security Notes

• Always use strong passwords for basic authentication
• Keep bcrypt hashes secure and never commit them to public repositories
• Regularly update Docker images for security patches
• Consider implementing rate limiting for production use
• Monitor access logs in /data/access.log (inside Caddy container)
• Use proper TLS certificates for production (Caddy handles this automatically with Let's Encrypt)

Network Architecture

Caddy (port 443, Cloudflare DNS + HTTPS)
         |
         +--- registry.custom.dev
         |         |                                            
         |         +-- /v2/* --> Registry:5000 (basic auth)     
         |                                                      
         +--- registry-ui.custom.dev
         |         |                                            
         |         +-- /* --> Registry UI:80 (basic auth)       
         |                                                      
         +--- minio.custom.dev
         |         |                                            
         |         +-- /* --> MinIO API:9000 (S3 auth)          
         |                                                      
         +--- minio-console.custom.dev
                   |
                   +-- /* --> MinIO Console:9001 (MinIO auth)

Security layers:

• Registry API: Caddy basic auth via registry.custom.dev (Cloudflare DNS + ACME TLS)
• Registry UI: Caddy basic auth via registry-ui.custom.dev (Cloudflare DNS + ACME TLS)
• MinIO API: AWS Signature V4 authentication via minio.custom.dev (Cloudflare DNS + ACME TLS)
• MinIO Console: MinIO's built-in login via minio-console.custom.dev (Cloudflare DNS + ACME TLS)
• All traffic: Encrypted via Caddy ACME TLS (Cloudflare DNS challenge)

Storage:

• Registry data: NFS volume at /srv/nfs/docker-registry
• MinIO data: Direct HDD mounts at /mnt/disk1/minio and /mnt/disk2/minio
• MinIO config: Direct SSD mount at /srv/minio/config
• Caddy data/config: NFS volumes