Thanks to visit codestin.com
Credit goes to github.com

Skip to content

NODE_LABEL_SUBNETS.md

Latest commit

 

History

History
381 lines (281 loc) · 11.6 KB

NODE_LABEL_SUBNETS.md

File metadata and controls

381 lines (281 loc) · 11.6 KB

Node Label-Based Subnet Allocation

Overview

The Cube CNI plugin supports per-node subnet allocation through Kubernetes node labels. This allows you to assign different pod CIDR ranges to different nodes, enabling flexible network segmentation and IP address management.

How It Works

When the CNI plugin DaemonSet starts on a node, the init container:

  1. Queries the node's labels for cni.cube.io/pod-subnet
  2. If the label exists, decodes its value by replacing - with / to get CIDR notation
  3. If the label doesn't exist, uses the default subnet from the ConfigMap (10.244.0.0/16)
  4. Validates the decoded CIDR format
  5. Generates the final CNI configuration file with the resolved subnet
  6. Writes the configuration to /etc/cni/net.d/10-cube-cni.conflist

Label Format

Label Key: cni.cube.io/pod-subnet

Label Value: A dash-encoded CIDR notation (e.g., 10.244.1.0-24)

Why Dash Encoding?

Kubernetes label values have strict validation rules and cannot contain forward slashes (/). Since CIDR notation uses / (e.g., 10.244.1.0/24), we must encode the CIDR when storing it in a label.

Encoding Scheme:

  • Replace / with - (dash) when setting the label
  • The init container automatically decodes by replacing - with /
  • Example: 10.244.1.0/24 → 10.244.1.0-24 (for label) → 10.244.1.0/24 (decoded)

Valid vs Invalid Examples:

✅ Correct - Dash-encoded format:

kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0-24
kubectl label node worker-2 cni.cube.io/pod-subnet=10.244.2.0-25
kubectl label node worker-3 cni.cube.io/pod-subnet=10.244.3.0-26

❌ INVALID - Contains forward slash (will be rejected by Kubernetes):

kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0/24

CIDR Requirements

  • Must be in valid dash-encoded format: X.X.X.X-Y
  • After decoding, must be valid CIDR: X.X.X.X/Y
  • Prefix length must be between /1 and /32
  • Must not overlap with other node subnets (your responsibility to manage)
  • Should be appropriately sized for the expected number of pods on the node

Recommended Subnet Sizes

  • /24 (254 usable IPs) - Standard nodes with moderate pod density
  • /25 (126 usable IPs) - Smaller nodes or controlled environments
  • /26 (62 usable IPs) - Minimal pod deployments
  • /23 (510 usable IPs) - Large nodes with high pod density

Labeling Nodes

Label a Single Node

# Label a specific node with a custom subnet (dash-encoded)
kubectl label node <node-name> cni.cube.io/pod-subnet=10.244.1.0-24

Example:

kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0-24
kubectl label node worker-2 cni.cube.io/pod-subnet=10.244.2.0-24
kubectl label node worker-3 cni.cube.io/pod-subnet=10.244.3.0-24

Encoding Examples:

# /24 subnet (254 usable IPs)
10.244.1.0/24  →  10.244.1.0-24

# /25 subnet (126 usable IPs)
10.244.2.0/25  →  10.244.2.0-25

# /26 subnet (62 usable IPs)
10.244.3.0/26  →  10.244.3.0-26

# /23 subnet (510 usable IPs)
10.244.4.0/23  →  10.244.4.0-23

Label Multiple Nodes

# Label all nodes matching a selector
kubectl label nodes -l node-role.kubernetes.io/worker cni.cube.io/pod-subnet=10.244.0.0-16

View Node Labels

# View all labels on a node
kubectl get node <node-name> --show-labels

# View only the pod-subnet label (encoded format)
kubectl get node <node-name> -o jsonpath='{.metadata.labels.cni\.cube\.io/pod-subnet}'

# View all node subnets in a table
kubectl get nodes -o custom-columns=NAME:.metadata.name,SUBNET:.metadata.labels.cni\\.cube\\.io/pod-subnet

Remove a Label

# Remove the pod-subnet label (node will use default subnet after restart)
kubectl label node <node-name> cni.cube.io/pod-subnet-

Configuration Lifecycle

Initial Installation

  1. Before installation: Label nodes with desired subnets (optional, dash-encoded)

    kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0-24

  2. Install CNI plugin: Run the installation script

    ./deploy/install.sh

  3. Verify configuration: Check the generated config on each node

    kubectl exec -n kube-system -l app=cube-cni-plugin -- cat /host/etc/cni/net.d/10-cube-cni.conflist

Changing Node Subnets

Important: Configuration is generated at DaemonSet pod startup. To apply label changes:

  1. Update the node label (dash-encoded):

    kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.5.0-24 --overwrite

  2. Restart the DaemonSet pod on that node:

    # Delete the pod on the specific node (it will be recreated)
kubectl delete pod -n kube-system -l app=cube-cni-plugin --field-selector spec.nodeName=worker-1

  3. Verify the new configuration:

    kubectl exec -n kube-system -l app=cube-cni-plugin --field-selector spec.nodeName=worker-1 -- cat /host/etc/cni/net.d/10-cube-cni.conflist

  4. Restart existing pods on that node to get IPs from the new subnet (optional but recommended)

Verification

Check Generated Configuration

# View the generated CNI config on a specific node
kubectl exec -n kube-system -l app=cube-cni-plugin --field-selector spec.nodeName=<node-name> -- cat /host/etc/cni/net.d/10-cube-cni.conflist

Check Init Container Logs

# View logs from the config generation init container
kubectl logs -n kube-system -l app=cube-cni-plugin -c install-cni-config

Expected output for labeled node:

=== CNI Configuration Generator ===
Node: worker-1
Querying node labels...
✓ Found encoded label value: 10.244.1.0-24
✓ Decoded to CIDR notation: 10.244.1.0/24
✓ Using subnet from label: 10.244.1.0/24
✓ CIDR validation passed
Generating CNI configuration...
✓ CNI configuration generated successfully

Expected output for unlabeled node:

=== CNI Configuration Generator ===
Node: worker-2
Querying node labels...
✓ No label found, using default subnet: 10.244.0.0/16
✓ CIDR validation passed
Generating CNI configuration...
✓ CNI configuration generated successfully

Verify Pod IP Allocation

# Create test pods on specific nodes
kubectl run test-pod-1 --image=busybox --overrides='{"spec":{"nodeName":"worker-1"}}' -- sleep 3600
kubectl run test-pod-2 --image=busybox --overrides='{"spec":{"nodeName":"worker-2"}}' -- sleep 3600

# Check pod IPs (should be from the respective node's subnet)
kubectl get pods -o wide

Troubleshooting

Configuration Not Applied

Symptom: Node label changed but pods still get IPs from old subnet

Solution: Restart the DaemonSet pod on that node

kubectl delete pod -n kube-system -l app=cube-cni-plugin --field-selector spec.nodeName=<node-name>

Invalid CIDR Format Error

Symptom: Init container fails with "Invalid CIDR format" or "Invalid encoded value" error

Check the label value:

kubectl get node <node-name> -o jsonpath='{.metadata.labels.cni\.cube\.io/pod-subnet}'

Common mistakes:

# ❌ Wrong - using forward slash instead of dash
kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0/24

# ✅ Correct - using dash encoding
kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0-24

# ❌ Wrong - missing prefix length
kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0

# ✅ Correct - includes prefix length
kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0-24

Fix: Ensure the label value is in valid dash-encoded format

kubectl label node <node-name> cni.cube.io/pod-subnet=10.244.1.0-24 --overwrite
kubectl delete pod -n kube-system -l app=cube-cni-plugin --field-selector spec.nodeName=<node-name>

Validation command:

# Check if label is properly encoded (should contain dash, not slash)
LABEL=$(kubectl get node <node-name> -o jsonpath='{.metadata.labels.cni\.cube\.io/pod-subnet}')
if echo "$LABEL" | grep -q '/'; then
  echo "ERROR: Label contains forward slash - use dash encoding"
else
  echo "OK: Label is properly encoded"
fi

RBAC Permission Errors

Symptom: Init container fails with "forbidden" or permission errors

Check RBAC:

kubectl get clusterrole cube-cni-plugin -o yaml
kubectl get clusterrolebinding cube-cni-plugin -o yaml

Verify the ServiceAccount has permissions:

kubectl auth can-i get nodes --as=system:serviceaccount:kube-system:cube-cni-plugin
kubectl auth can-i list nodes --as=system:serviceaccount:kube-system:cube-cni-plugin

Configuration File Not Generated

Symptom: /etc/cni/net.d/10-cube-cni.conflist doesn't exist on the node

Check init container logs:

kubectl logs -n kube-system -l app=cube-cni-plugin -c install-cni-config

Check volume mounts:

kubectl describe daemonset cube-cni-plugin -n kube-system

Subnet Overlap Issues

Symptom: Pods on different nodes have IP conflicts

Prevention: Plan your subnet allocation carefully

  • Use non-overlapping CIDR ranges for each node
  • Document your subnet allocation scheme
  • Consider using a subnet calculator tool

Example allocation for 3 nodes:

kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0-24  # 10.244.1.1 - 10.244.1.254
kubectl label node worker-2 cni.cube.io/pod-subnet=10.244.2.0-24  # 10.244.2.1 - 10.244.2.254
kubectl label node worker-3 cni.cube.io/pod-subnet=10.244.3.0-24  # 10.244.3.1 - 10.244.3.254

Best Practices

  1. Plan subnet allocation before deployment

    • Document your subnet scheme
    • Ensure no overlaps between node subnets
    • Leave room for future node additions

  2. Label nodes before installing CNI

    • Avoids need to restart DaemonSet pods
    • Ensures correct configuration from the start

  3. Use consistent subnet sizing

    • Makes management easier
    • Simplifies troubleshooting

  4. Monitor IP allocation

    • Check /var/lib/cni/networks/ on nodes
    • Ensure subnets aren't exhausted

  5. Document your allocation

    • Keep a record of which subnets are assigned to which nodes
    • Update documentation when making changes

Examples

Example 1: Small Cluster (3 nodes)

# Assign /24 subnets to each node (dash-encoded)
kubectl label node control-plane cni.cube.io/pod-subnet=10.244.0.0-24
kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.1.0-24
kubectl label node worker-2 cni.cube.io/pod-subnet=10.244.2.0-24

# Install CNI
./deploy/install.sh

# Verify
kubectl get nodes -o custom-columns=NAME:.metadata.name,SUBNET:.metadata.labels.cni\\.cube\\.io/pod-subnet

Example 2: Mixed Node Sizes

# Large nodes get /23 (510 IPs) - dash-encoded
kubectl label node large-worker-1 cni.cube.io/pod-subnet=10.244.0.0-23
kubectl label node large-worker-2 cni.cube.io/pod-subnet=10.244.2.0-23

# Standard nodes get /24 (254 IPs) - dash-encoded
kubectl label node worker-1 cni.cube.io/pod-subnet=10.244.4.0-24
kubectl label node worker-2 cni.cube.io/pod-subnet=10.244.5.0-24

# Small nodes get /25 (126 IPs) - dash-encoded
kubectl label node small-worker-1 cni.cube.io/pod-subnet=10.244.6.0-25
kubectl label node small-worker-2 cni.cube.io/pod-subnet=10.244.6.128-25

Example 3: Using Default for Most Nodes

# Only label nodes that need custom subnets (dash-encoded)
kubectl label node special-worker-1 cni.cube.io/pod-subnet=10.244.100.0-24
kubectl label node special-worker-2 cni.cube.io/pod-subnet=10.244.101.0-24

# All other nodes will use the default: 10.244.0.0/16
# (Note: This may cause IP conflicts if not managed carefully)

Related Documentation