drenv: Use minikube on macOS #2356
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Nir Soffer <[email protected]>
With the new vmnet network[1] and rosetta[2] in minikube we can use
minikube on macOS. This change replace macOS defaults to use minikube
with vfkit driver and vmnet-shared network.
With this change the vm.yaml environment starts in 16 seconds, 5 times
faster than lima. The regional-dr environment starts in 324 seconds,
about 1.3 times faster than lima.
Example run with minimal environment:
% drenv start envs/vm.yaml
2025-12-13 18:58:13,469 INFO [vm] Starting environment
2025-12-13 18:58:13,547 INFO [cluster] Starting minikube cluster
2025-12-13 18:58:29,953 INFO [cluster] Cluster started in 16.41 seconds
2025-12-13 18:58:29,955 INFO [cluster/0] Running addons/example/start
2025-12-13 18:58:41,240 INFO [cluster/0] addons/example/start completed in 11.28 seconds
2025-12-13 18:58:41,241 INFO [cluster/0] Running addons/example/test
2025-12-13 18:58:41,471 INFO [cluster/0] addons/example/test completed in 0.23 seconds
2025-12-13 18:58:41,471 INFO [vm] Environment started in 28.00 seconds
Example run with full environment:
% drenv start envs/regional-dr.yaml
2025-12-13 19:11:14,075 INFO [rdr] Starting environment
2025-12-13 19:11:14,116 INFO [dr2] Starting minikube cluster
2025-12-13 19:11:14,116 INFO [dr1] Starting minikube cluster
2025-12-13 19:11:14,121 INFO [hub] Starting minikube cluster
2025-12-13 19:11:29,211 INFO [dr1] Cluster started in 15.09 seconds
...
2025-12-13 19:11:32,399 INFO [hub] Cluster started in 18.28 seconds
...
2025-12-13 19:11:36,296 INFO [dr2] Cluster started in 22.18 seconds
...
2025-12-13 19:16:38,669 INFO [rdr] Dumping ramen e2e config to '/Users/nir/.config/drenv/rdr'
2025-12-13 19:16:38,761 INFO [rdr] Environment started in 324.69 seconds
Note: This change requires local minikube build since the rosetta
support is not merged yet.
[1] kubernetes/minikube#20501
[2] kubernetes/minikube#22140
Signed-off-by: Nir Soffer <[email protected]>
Running ramenctl test we see log of pod restarts or crashing on the hub,
and top shows up to 99% memory usage. We know that minikube minimal
memory usage increased in latest version from 2.2g to 3g. Add 2g memory
to all clusters.
Testing shows that start time reduced from 329 to 260 seconds (1.26
times faster), and pod restart during ramenctl test are gone.
Before - pod with issues:
% kubectl get pod -A --context hub | grep ago
argocd argocd-dex-server-b7c4c4d89-9m5mc 1/1 Running 2 (49m ago) 49m
argocd argocd-repo-server-56579d8b85-6mdfm 1/1 Running 1 (105s ago) 49m
kube-system storage-provisioner 1/1 Running 7 (100s ago) 52m
olm catalog-operator-6cd86dbdb-5hnz7 1/1 Running 1 (20m ago) 51m
olm olm-operator-94ddfc887-7r444 1/1 Running 1 (20m ago) 51m
olm operatorhubio-catalog-nlfsh 1/1 Running 3 (31s ago) 51m
olm packageserver-58c944fdc6-858bw 1/1 Running 4 (31s ago) 51m
olm packageserver-58c944fdc6-tqgg2 1/1 Running 4 (2m39s ago) 51m
open-cluster-management-hub cluster-manager-addon-manager-controller-7f48c4564b-j28kl 1/1 Running 6 (31s ago) 51m
open-cluster-management-hub cluster-manager-placement-controller-78b89ddf9c-4j88v 1/1 Running 6 (31s ago) 51m
open-cluster-management-hub cluster-manager-registration-controller-7fdbc97ddc-q8m7t 1/1 Running 5 (31s ago) 51m
open-cluster-management-hub cluster-manager-registration-webhook-6ff7f84c59-t96s4 1/1 Running 1 (20m ago) 51m
open-cluster-management-hub cluster-manager-work-webhook-685df67558-blfrn 1/1 Running 2 (18m ago) 51m
open-cluster-management cluster-manager-8b858c74-6vnvg 1/1 Running 3 (33s ago) 52m
open-cluster-management cluster-manager-8b858c74-7wp5f 1/1 Running 2 (18m ago) 52m
open-cluster-management cluster-manager-8b858c74-jjd2b 1/1 Running 3 (32s ago) 52m
open-cluster-management governance-policy-propagator-7c4756948-rmhp4 1/1 Running 7 (100s ago) 51m
open-cluster-management ocm-controller-6fbf75c98c-j4jfh 0/1 CrashLoopBackOff 8 (32s ago) 51m
ramen-system ramen-hub-operator-5f5d7d6f76-8sb5l 2/2 Running 8 (52s ago) 27m
submariner-operator submariner-operator-64b7475466-jhkjb 0/1 CrashLoopBackOff 7 (32s ago) 51m
After - start:
% drenv start envs/regional-dr.yaml
2025-12-13 20:19:01,200 INFO [rdr] Starting environment
2025-12-13 20:19:01,241 INFO [dr2] Starting minikube cluster
2025-12-13 20:19:01,245 INFO [hub] Starting minikube cluster
2025-12-13 20:19:01,246 INFO [dr1] Starting minikube cluster
2025-12-13 20:19:17,339 INFO [hub] Cluster started in 16.09 seconds
...
2025-12-13 20:19:22,093 INFO [dr2] Cluster started in 20.85 seconds
...
2025-12-13 20:19:24,153 INFO [dr1] Cluster started in 22.91 seconds
...
2025-12-13 20:23:21,655 INFO [rdr] Dumping ramen e2e config to '/Users/nir/.config/drenv/rdr'
2025-12-13 20:23:21,745 INFO [rdr] Environment started in 260.55 seconds
After - pod with issues:
% kubectl get pods -A --context hub | grep ago
kube-system storage-provisioner 1/1 Running 1 (30m ago) 31m
Signed-off-by: Nir Soffer <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
drenv: Use minikube with vfkit driver on macOS
With the new vmnet network[1] and rosetta[2] in minikube we can use
minikube on macOS. This change replace macOS defaults to use minikube
with vfkit driver and vmnet-shared network.
Minikube requires more memory, so cluster memory is increased by 2g per
cluster. This avoids issues with crashing pods and improves start time
significantly.
With this change the vm.yaml environment starts in 16 seconds, 5 times
faster than lima. The regional-dr environment starts in 260 seconds,
about 1.6 times faster than lima.
Example run with minimal environment
Example run with full environment
Issues
Building minikube with rosetta support