🧊 Cube CNI Plugin

A learning-focused Kubernetes Container Network Interface (CNI) plugin built from scratch to understand how container networking really works.

🎯 Project Purpose

Cube CNI is an educational project designed to demystify Kubernetes networking by implementing a fully functional CNI plugin from the ground up. This isn't just another CNI plugin—it's a learning platform with extensive logging, clear code structure, and comprehensive documentation to help you understand:

🔍 How Kubernetes networking actually works under the hood
🌐 How pods get IP addresses and communicate
🔧 How CNI plugins integrate with Kubernetes
📦 How to build production-grade Go projects with Bazel
🧪 How to test network components with property-based testing

Perfect for platform engineers, SREs, and anyone curious about Kubernetes internals!

✨ Features

🎓 Learning-First Design

Extensive Logging: Every operation is logged with context for easy debugging
Clear Code Structure: Well-organized packages with single responsibilities
Comprehensive Documentation: Detailed docs explaining every component
Property-Based Testing: Learn advanced testing techniques

🚀 Production-Ready Components

Bridge-Based Networking: Simple but effective pod-to-pod communication
IPAM (IP Address Management): Automatic IP allocation and tracking
Node Label-Based Subnets: Flexible per-node subnet configuration
CNI Spec Compliance: Implements CNI specification 0.4.0+
Kubernetes Integration: DaemonSet deployment with RBAC

🛠️ Modern Development Stack

Go 1.21+: Modern, efficient, and easy to understand
Bazel Build System: Fast, reproducible builds
Container Images: OCI-compliant images ready for deployment
Cross-Platform Development: Build on macOS, deploy to Linux

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Kubernetes Node                          │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐  │
│  │              Host Network Namespace                   │  │
│  │                                                       │  │
│  │         ┌─────────────────────────────┐              │  │
│  │         │   Linux Bridge (cni0)       │              │  │
│  │         │   IP: 10.244.0.1/24         │              │  │
│  │         └──────────┬──────────────────┘              │  │
│  │                    │                                  │  │
│  │         ┌──────────┼──────────┬──────────┐           │  │
│  │         │          │          │          │           │  │
│  │      veth-a     veth-b     veth-c     veth-d         │  │
│  └────────┼──────────┼──────────┼──────────┼───────────┘  │
│           │          │          │          │               │
│  ┌────────┼──────┐ ┌─┼──────┐ ┌─┼──────┐ ┌─┼──────┐       │
│  │ Pod 1  │      │ │ Pod 2  │ │ Pod 3  │ │ Pod 4  │       │
│  │ ┌──────▼────┐ │ │ ┌──────▼┐│ │ ┌──────▼┐│ │ ┌──────▼┐ │ │
│  │ │ eth0      │ │ │ │ eth0  ││ │ │ eth0  ││ │ │ eth0  │ │ │
│  │ │10.244.0.2 │ │ │ │.0.3   ││ │ │.0.4   ││ │ │.0.5   │ │ │
│  │ └───────────┘ │ │ └───────┘│ │ └───────┘│ │ └───────┘ │ │
│  └───────────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────┘

Component Overview

cube-cni-plugin/
├── cmd/cni-plugin/          # Main CNI plugin binary
│   └── main.go              # Entry point, command routing
├── pkg/
│   ├── ipam/                # IP Address Management
│   │   └── allocator.go     # IP allocation and persistence
│   ├── logging/             # Structured logging
│   │   └── logger.go        # File-based logger
│   ├── netns/               # Network namespace management
│   │   └── manager.go       # Namespace operations
│   ├── network/             # Network configuration
│   │   ├── bridge.go        # Linux bridge management
│   │   ├── veth.go          # Veth pair management
│   │   └── configurator.go # Orchestrates network setup
│   ├── plugin/              # CNI command handlers
│   │   └── handlers.go      # ADD, DEL, CHECK, VERSION
│   └── types/               # Data models
│       └── types.go         # CNI configuration structures
└── deploy/                  # Kubernetes manifests
    ├── daemonset.yaml       # DaemonSet with RBAC
    ├── configmap.yaml       # CNI configuration
    └── install.sh           # Installation script

🚀 Quick Start

Prerequisites

Kubernetes Cluster: Local (kind, minikube) or remote
kubectl: Configured to access your cluster
Bazel: For building (or use pre-built images)
Go 1.21+: For development

Option 1: Use Pre-Built Image (Fastest)

# Clone the repository
git clone https://github.com/cube-k8s/cube-cni-plugin.git
cd cube-cni-plugin

# Install to your cluster
./deploy/install.sh

# Verify installation
kubectl get pods -n kube-system -l app=cube-cni-plugin
kubectl logs -n kube-system -l app=cube-cni-plugin -c install-cni-config

Option 2: Build from Source

# Build everything
./scripts/build.sh all

# Install to cluster
./deploy/install.sh

Verify It Works

# Create test pods
kubectl run test1 --image=busybox -- sleep 3600
kubectl run test2 --image=busybox -- sleep 3600

# Check pod IPs
kubectl get pods -o wide

# Test connectivity
kubectl exec test1 -- ping -c 3 $(kubectl get pod test2 -o jsonpath='{.status.podIP}')

📚 Learning Path

1. Start with the Basics

Read the Requirements → .kiro/specs/kubernetes-cni-plugin/requirements.md

Understand what a CNI plugin must do
Learn about CNI specification compliance

Explore the Design → .kiro/specs/kubernetes-cni-plugin/design.md

See how components fit together
Understand the architecture decisions

2. Dive into the Code

Start with the Entry Point → cmd/cni-plugin/main.go

// See how CNI commands are parsed and routed
func main() {
    // Parse CNI environment variables
    env := parseCNIEnv()
    
    // Route to appropriate handler
    switch env.Command {
    case "ADD":    handleAdd()
    case "DEL":    handleDel()
    case "CHECK":  handleCheck()
    case "VERSION": handleVersion()
    }
}

Follow the ADD Flow → pkg/plugin/handlers.go

// See how a pod gets networking
func HandleAdd(config, env) {
    1. Allocate IP address (IPAM)
    2. Create Linux bridge
    3. Create veth pair
    4. Move veth to pod namespace
    5. Configure IP and routes
    6. Return result to Kubernetes
}

Understand IPAM → pkg/ipam/allocator.go

// See how IPs are allocated and tracked
func (ia *IPAllocator) AllocateIP(containerID) {
    - Find next available IP in subnet
    - Mark as allocated
    - Persist to disk
    - Return IP to caller
}

3. Experiment with Features

Try Node-Specific Subnets → docs/NODE_LABEL_SUBNETS.md

# Assign different subnets to different nodes
kubectl label node worker-1 cni.cube.io/pod-cidr=10.244.1.0/24
kubectl label node worker-2 cni.cube.io/pod-cidr=10.244.2.0/24

# Restart CNI pods to apply
kubectl delete pod -n kube-system -l app=cube-cni-plugin

Watch the Logs

# See every operation in detail
kubectl logs -n kube-system -l app=cube-cni-plugin -c install-cni-config

# Check CNI plugin logs on nodes
kubectl exec -n kube-system -l app=cube-cni-plugin -- cat /var/log/cni/plugin.log

4. Study the Tests

Property-Based Testing → pkg/ipam/allocator_test.go

// Learn advanced testing techniques
Property: "All allocated IPs must be within subnet"
Property: "No two containers get the same IP"
Property: "Allocation persists across restarts"

Run Tests Yourself

./scripts/build.sh test

🎓 Key Concepts Explained

What is a CNI Plugin?

A CNI (Container Network Interface) plugin is a program that Kubernetes calls to:

ADD: Set up networking when a pod starts
DEL: Clean up networking when a pod stops
CHECK: Verify networking is still working
VERSION: Report supported CNI versions

How Does Pod Networking Work?

1. Kubernetes creates a pod
2. Kubelet calls CNI plugin with ADD command
3. CNI plugin:
   - Allocates an IP address
   - Creates a veth pair (virtual ethernet cable)
   - Connects one end to a bridge
   - Moves other end into pod's network namespace
   - Configures IP address and routes
4. Pod can now communicate!

What is a Network Namespace?

A network namespace is like a separate network stack:

Each pod gets its own namespace
Isolated from other pods and the host
Has its own interfaces, IPs, and routes
Connected to host via veth pairs

What is a Linux Bridge?

A Linux bridge is like a virtual network switch:

All pods on a node connect to it
Forwards packets between pods
Enables pod-to-pod communication
Simple but effective for learning

🔧 Advanced Features

Node Label-Based Subnet Allocation

Assign different IP subnets to different nodes:

# Small nodes get /25 (126 IPs)
kubectl label node small-worker cni.cube.io/pod-cidr=10.244.1.0/25

# Large nodes get /23 (510 IPs)
kubectl label node large-worker cni.cube.io/pod-cidr=10.244.2.0/23

# Unlabeled nodes use default: 10.244.0.0/16

How it works:

DaemonSet init container reads node labels
Generates CNI config with appropriate subnet
Writes config to /etc/cni/net.d/
CNI plugin uses that subnet for IP allocation

See docs/NODE_LABEL_SUBNETS.md for details.

Extensive Logging

Every operation is logged for learning and debugging:

=== CNI Configuration Generator ===
Node: worker-1
Querying node labels...
✓ Using subnet from label: 10.244.1.0/24
✓ CIDR validation passed
Generating CNI configuration...
✓ CNI configuration generated successfully

[INFO] CNI ADD command received
[INFO] Container ID: abc123
[INFO] Network namespace: /var/run/netns/pod-ns
[INFO] Allocating IP address...
[INFO] Allocated IP: 10.244.1.5/24
[INFO] Creating bridge: cni0
[INFO] Bridge already exists, skipping creation
[INFO] Creating veth pair: veth-abc123 <-> eth0
[INFO] Moving veth to namespace...
[INFO] Configuring IP address...
[INFO] Adding default route...
[INFO] Network setup complete!

Property-Based Testing

Learn advanced testing with properties that must hold for all inputs:

// Property: All allocated IPs must be within subnet
Property("IP within subnet", func(subnet string) bool {
    allocator := NewIPAllocator(subnet)
    ip := allocator.AllocateIP("container-123")
    return subnet.Contains(ip)
})

// Property: No duplicate IPs
Property("Unique IPs", func(containerIDs []string) bool {
    allocator := NewIPAllocator("10.244.0.0/24")
    ips := make(map[string]bool)
    for _, id := range containerIDs {
        ip := allocator.AllocateIP(id)
        if ips[ip] {
            return false // Duplicate found!
        }
        ips[ip] = true
    }
    return true
})

🛠️ Development

Building

# Build binary for Linux
./scripts/build.sh binary

# Build container image
./scripts/build.sh image

# Run tests
./scripts/build.sh test

# Build everything
./scripts/build.sh all

See BUILD_INSTRUCTIONS.md for detailed build documentation.

Project Structure

cube-cni-plugin/
├── .kiro/specs/             # Design specifications
│   └── kubernetes-cni-plugin/
│       ├── requirements.md  # What the plugin must do
│       ├── design.md        # How it's designed
│       └── tasks.md         # Implementation tasks
├── cmd/                     # Binaries
│   └── cni-plugin/          # Main CNI plugin
├── pkg/                     # Libraries
│   ├── ipam/                # IP address management
│   ├── logging/             # Logging infrastructure
│   ├── netns/               # Network namespace ops
│   ├── network/             # Network configuration
│   ├── plugin/              # CNI handlers
│   └── types/               # Data models
├── deploy/                  # Kubernetes manifests
│   ├── daemonset.yaml       # DaemonSet + RBAC
│   ├── configmap.yaml       # CNI configuration
│   ├── install.sh           # Installation script
│   └── uninstall.sh         # Cleanup script
├── docs/                    # Documentation
│   ├── NODE_LABEL_SUBNETS.md
│   ├── CONTAINER_IMAGE.md
│   └── TESTING_NODE_LABELS.md
├── scripts/                 # Build scripts
│   └── build.sh             # Main build script
├── MODULE.bazel             # Bazel dependencies
├── go.mod                   # Go dependencies
└── README.md                # This file!

Adding Features

Update Requirements → .kiro/specs/kubernetes-cni-plugin/requirements.md
Update Design → .kiro/specs/kubernetes-cni-plugin/design.md
Add Tasks → .kiro/specs/kubernetes-cni-plugin/tasks.md
Implement → Write code in pkg/ or cmd/
Test → Add tests, run ./scripts/build.sh test
Document → Update relevant docs

Running Tests

# All tests
./scripts/build.sh test

# Specific test
bazel test --config=macos //pkg/ipam:ipam_test

# With coverage
bazel coverage --config=macos //pkg/...

📖 Documentation

Quick Start Guide - Get up and running quickly
Requirements - What the plugin must do
Design - Architecture and components
Build Instructions - How to build and test
Node Label Subnets - Per-node subnet configuration
Cross-Node Routing - How pods communicate across nodes
Container Image - Image build and deployment
Testing Guide - Testing node label features
Troubleshooting - Common issues and solutions

🎯 Learning Objectives

After working with this project, you'll understand:

✅ CNI Specification: How CNI plugins integrate with Kubernetes
✅ Network Namespaces: How Linux isolates network stacks
✅ Veth Pairs: How virtual ethernet cables connect namespaces
✅ Linux Bridges: How virtual switches enable pod communication
✅ IPAM: How IP addresses are allocated and tracked
✅ Kubernetes Networking: How pods get IPs and communicate
✅ DaemonSets: How to deploy node-level components
✅ RBAC: How to secure Kubernetes service accounts
✅ Bazel: How to build reproducible Go projects
✅ Property-Based Testing: Advanced testing techniques

🤝 Contributing

This is a learning project! Contributions that improve clarity, add documentation, or enhance the learning experience are welcome.

Ideas for Contributions

📝 More detailed code comments
📚 Additional documentation or tutorials
🧪 More test cases with explanations
🎨 Diagrams explaining concepts
🐛 Bug fixes with learning notes
✨ New features with educational value

How to Contribute

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Make your changes
Add tests and documentation
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

🔍 Troubleshooting

Pods Not Getting IPs

# Check CNI plugin is running
kubectl get pods -n kube-system -l app=cube-cni-plugin

# Check logs
kubectl logs -n kube-system -l app=cube-cni-plugin -c install-cni-config

# Check CNI config on node
kubectl exec -n kube-system -l app=cube-cni-plugin -- cat /host/etc/cni/net.d/10-cube-cni.conflist

Pods Can't Communicate

# Check bridge exists
kubectl exec -n kube-system -l app=cube-cni-plugin -- ip link show cni0

# Check IP allocation
kubectl exec -n kube-system -l app=cube-cni-plugin -- cat /var/lib/cni/networks/cube-cni/allocations.json

# Check routes
kubectl exec test1 -- ip route

Build Issues

# Clean and rebuild
./scripts/build.sh clean
./scripts/build.sh all

# Check dependencies
go mod tidy
bazel mod tidy

See BUILD_INSTRUCTIONS.md for more troubleshooting.

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

CNI Project: For the excellent specification and reference implementations
Kubernetes Community: For comprehensive networking documentation
Bazel Team: For the amazing build system
Go Community: For the fantastic language and ecosystem

📬 Contact & Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: Project Wiki

🎓 Additional Resources

Learn More About CNI

Learn More About Kubernetes Networking

Learn More About Linux Networking

Built with ❤️ for learning Kubernetes networking

⭐ Star this repo if it helped you learn something new!

Report Bug · Request Feature · Documentation

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.kiro/specs/kubernetes-cni-plugin		.kiro/specs/kubernetes-cni-plugin
cmd/cni-plugin		cmd/cni-plugin
deploy		deploy
docs		docs
pkg		pkg
scripts		scripts
.bazelrc		.bazelrc
.gitignore		.gitignore
BUILD.bazel		BUILD.bazel
BUILD_FIX_SUMMARY.md		BUILD_FIX_SUMMARY.md
BUILD_INSTRUCTIONS.md		BUILD_INSTRUCTIONS.md
MODULE.bazel		MODULE.bazel
MODULE.bazel.lock		MODULE.bazel.lock
README.md		README.md
WORKSPACE		WORKSPACE
go.mod		go.mod
go.sum		go.sum

Folders and files

Latest commit

History

Repository files navigation

🧊 Cube CNI Plugin

🎯 Project Purpose

✨ Features

🎓 Learning-First Design

🚀 Production-Ready Components

🛠️ Modern Development Stack

🏗️ Architecture

Component Overview

🚀 Quick Start

Prerequisites

Option 1: Use Pre-Built Image (Fastest)

Option 2: Build from Source

Verify It Works

📚 Learning Path

1. Start with the Basics

2. Dive into the Code

3. Experiment with Features

4. Study the Tests

🎓 Key Concepts Explained

What is a CNI Plugin?

How Does Pod Networking Work?

What is a Network Namespace?

What is a Linux Bridge?

🔧 Advanced Features

Node Label-Based Subnet Allocation

Extensive Logging

Property-Based Testing

🛠️ Development

Building

Project Structure

Adding Features

Running Tests

📖 Documentation

🎯 Learning Objectives

🤝 Contributing

Ideas for Contributions

How to Contribute

🔍 Troubleshooting

Pods Not Getting IPs

Pods Can't Communicate

Build Issues

📜 License

🙏 Acknowledgments

📬 Contact & Support

🎓 Additional Resources

Learn More About CNI

Learn More About Kubernetes Networking

Learn More About Linux Networking

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages