otaru

Over-Engineering at Its Finest

Bare-metal k3s home lab and technical playground.

The only rule: Raspberry Pi 🍓. Why? Why not ¯\_(ツ)_/¯

Architecture

Current cluster layout:

• Dedicated 192.168.10.0/24 VLAN for cluster nodes and service virtual IPs
• k3s with embedded etcd on three control-plane nodes
• Flannel wireguard-native for pod networking
• MetalLB + Envoy Gateway for service and ingress virtual IPs, with the Envoy Gateway controller in envoy-gateway-system and the ingress proxy in gateway
• Istio ambient mesh with Kiali for service mesh observability

Hardware

Node	Device	Role	Boot	Storage
raspberrypi-00	Raspberry Pi 5 8GB	Control plane	NVMe	Lexar NM620 256GB
raspberrypi-01	Raspberry Pi 5 8GB	Control plane	NVMe	Samsung 980 PRO 2TB (MZ-V8P2T0BW)
raspberrypi-02	Raspberry Pi 5 8GB	Control plane	NVMe	Crucial P3 Plus 4TB
raspberrypi-03	Raspberry Pi 4 Model B 8GB	Worker	SD	SanDisk Max Endurance 32 GB
ucg-ultra	UniFi Cloud Gateway Ultra	Router/Gateway	-	-
usw-ultra	UniFi Switch Ultra	PoE switch	-	-
rackmate-t1	GeeekPi DeskPi RackMate T1	Rack enclosure	-	-
rack-mount	GeeekPi 10" 2U Rack Mount	Pi rack mount	-	-

Three nodes form the control plane. One node remains a worker.

Network Layout

IP	Role
192.168.10.1	VLAN gateway
192.168.10.50	Kubernetes API VIP
192.168.10.51	Internal ingress VIP
192.168.10.60	raspberrypi-00
192.168.10.61	raspberrypi-01
192.168.10.62	raspberrypi-02
192.168.10.63	raspberrypi-03

Cluster Components

Category	Name	Description
Application	Blocky	Stateless DNS server
Application	changedetection.io	Self-hosted website change detection and alerting
Application	CyberChef	The Cyber Swiss Army Knife by GCHQ
Application	Excalidraw	Virtual whiteboard for sketching hand-drawn like diagrams
Application	Home Assistant	Home automation
Application	JSON Crack	JSON, YAML, etc. visualizer and editor
Application	Kubernetes MCP Server	Model Context Protocol server for Kubernetes cluster operations
Application	TeslaMate	Self-hosted data logger for Tesla
Application	冗PowerBot	Telegram bot tracks and counts individual message counts in groups
CI/CD	Argo CD	GitOps, drift detection, and reconciliation
Connectivity	Cloudflare Tunnel	Cloudflare Zero Trust edge
Connectivity	CoreDNS	Cluster DNS managed separately from the disabled bundled k3s add-on
Connectivity	Envoy Gateway	Gateway API ingress controller with controller in envoy-gateway-system and proxy in gateway; TCP/HTTP ingress can use the mesh path, UDP remains outside Istio mTLS
Connectivity	Flannel	wireguard-native encrypted pod networking for k3s
Connectivity	Gateway API	Kubernetes standard CRDs for managing network traffic
Connectivity	httpbin	Generic echo server for health checks
Connectivity	Istio	Service mesh control plane and ambient dataplane (istiod, istio-cni, ztunnel)
Connectivity	k3s-apiserver-loadbalancer	Operator that keeps the kubernetes service exposed as a LoadBalancer
Connectivity	MetalLB	Bare metal LoadBalancer implementation for service virtual IP allocation and L2 advertisement
Database	CloudNativePG	A Kubernetes operator that manages PostgreSQL clusters
Database	CloudNativePG Barman Cloud Plugin	Backup plugin for CloudNativePG
Monitoring	Fluent Bit	Node log collection and shipping to Loki
Monitoring	Grafana	Dashboards and visualization for cluster metrics and logs
Monitoring	Heartbeats	Kubernetes operator for heartbeat monitoring
Monitoring	Kiali	Service mesh observability UI for Istio
Monitoring	Kubernetes Metrics Server	Scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines
Monitoring	Loki	Log aggregation and query backend
Monitoring	Prometheus	Metrics collection and query backend
Scheduling	Descheduler	Evicts pods for optimal cluster node utilisation
Scheduling	k8s-cleaner	Automated failed pod cleanup and periodic workload repaving
Scheduling	KEDA	Event Driven Autoscaler
Scheduling	Reloader	Watch changes in ConfigMap and Secret and do rolling upgrades
Security	1Password Connect	Proxy service for 1Password; acts as a secret provider
Security	amazon-eks-pod-identity-webhook	Amazon EKS Pod Identity Webhook for IRSA in bare metal Kubernetes clusters
Security	cert-manager	Manages TLS certificates via Let's Encrypt and ACME protocol
Security	External Secrets Operator	Extracts secrets from a secret provider
Security	Kyverno	Kubernetes policy engine
Security	oidc-provider	Kubernetes OIDC provider and JWKS endpoint
Storage	Longhorn	Distributed block storage system; backup and restore from/to remote destinations

IaaS, PaaS, and SaaS

Category	Name	Service	Description
CI/CD	Anthropic	Claude Code	AI code review
CI/CD	OpenAI	Codex	AI code review
CI/CD	GitHub	Actions	Repository validation and release automation
CI/CD	Renovate	Dependency Updates	Automated dependency maintenance
Connectivity	Cloudflare	Access	Edge access control
Connectivity	Cloudflare	DNS	Authoritative DNS service
Connectivity	Cloudflare	Tunnel	Edge connectivity
Connectivity	UniFi	Network	Gateway, VLAN, WLAN, and firewall management
Monitoring	Heartbeats	Heartbeats Operator	Kubernetes operator for heartbeat monitoring
Monitoring	WebGazer	Uptime Monitoring	Health check
Security	1Password	Connect	Secrets automation
Security	AWS	IAM	OIDC/IRSA permissions and token exchange
Security	Let's Encrypt	Let's Encrypt	Certificate authority
Security	Snyk	Snyk	Vulnerability detection
Storage	AWS	DynamoDB	Database backend for 冗PowerBot
Storage	AWS	SQS	Queue backend for 冗PowerBot
Storage	AWS	S3	OpenTofu remote state
Storage	Backblaze	B2	Longhorn and CloudNativePG backups

Cluster Bootstrap

1. Install Tooling

   brew install \
     ansible \
     direnv \
     editorconfig-checker \
     go-jsonnet \
     helm \
     kubectl \
     markdownlint-cli2 \
     opentofu \
     terragrunt \
     yq \
     zizmor

2. Add SSH Keys to known_hosts

   KH=~/.ssh/known_hosts && touch "$KH" && for ip in 192.168.10.{60..63}; do ssh-keygen -f "$KH" -R "$ip"; ssh-keyscan "$ip" >> "$KH"; done

3. Set Up Service Credentials

   Create the local files documented in Secrets before running cluster bootstrap. Keep actual credentials outside this repo.

4. Bootstrap Cluster

   make setup

Oopsy

Update host packages and reboot the entire cluster.

make maintenance

Upgrade the k3s Kubernetes version and restart workloads.

make upgrade

Unlock a LUKS node after boot.

make unlock raspberrypi-01

Wipe everything and start from scratch.

make nuke

Set up Raspberry Pi nodes and the k3s cluster.

make setup