deployKF effortlessly integrates Kubeflow and leading MLOps tools on Kubernetes, compose your open ML platform today!

_{Crafted with ❤️ by the developers of Kubeflow.}

deployKF is the best way to build reliable ML Platforms on Kubernetes.

• deployKF supports leading MLOps & Data tools from both Kubeflow, and other projects
• deployKF has a Helm-like interface, with values for configuring all aspects of the deployment (no need to edit Kubernetes YAML)
• deployKF does NOT install resources directly in your cluster, instead it generates ArgoCD Applications to provide native GitOps support

deployKF currently supports MLOps tools from the Kubeflow ecosystem like Kubeflow Pipelines and Kubeflow Notebooks, for the full list of current tools, please see the supported tools page.

We are actively adding support for other popular tools such as MLflow Model Registry, Apache Airflow, and Feast. For a more complete list of planned tools, please see the future tools page.

deployKF was originally created by Mathew Wicks (GitHub: @thesuperzapper), a Kubeflow lead and maintainer of the popular Apache Airflow Helm Chart. However, deployKF is now a community-led project that welcomes contributions from anyone who wants to help.

For commercial services related to deployKF, please see the support page.

deployKF is a new project, and we are still building our community. If you are using deployKF, please consider adding your organization to our list of adopters.

Kubeflow and deployKF are two different but related projects:

• deployKF is a tool for deploying Kubeflow and other MLOps tools on Kubernetes as a cohesive platform.
• Kubeflow is a project that develops MLOps tools, including Kubeflow Pipelines, Kubeflow Notebooks, Katib, and more.

For more details, see our comparison between Kubeflow and deployKF.

The deployKF project is a welcoming community of contributors and users. We encourage participation from anyone who shares our mission of making it easy to build open ML Platforms on Kubernetes.

For more details, see our community page.

Intro / Demo - Kubeflow Community Call - July 2023

For full details on how to get started with deployKF, please see the getting started page.

• your local machine has the deploykf CLI installed
• a Kubernetes cluster (WARNING: we strongly recommend using a dedicated Kubernetes cluster for deployKF)
• the Kubernetes cluster has ArgoCD installed
• a private git repo for your generated manifests (NOTE: not required when using the deployKF ArgoCD Plugin)

deployKF is configured using YAML files containing configs named "values" which behave similarly to those in Helm.

deployKF has a very large number of configurable values (more than 1500), but you can start by defining a few important ones, and then grow your values file over time.

We recommend you start by copying the sample-values.yaml file, which includes reasonable defaults that should work on any Kubernetes cluster.

If you are not using the deployKF ArgoCD Plugin, you will need to set the following values:

We are actively working on detailed "production usage" guides, but for now, here are some other values you might want to change:

For information about other values, you can refer to the following resources:

• docstrings in generator/default_values.yaml, which contains defaults for all values
• the values reference page, which contains a list of all values with links to their docstrings
• the "topics" section of the website, which has information about achieving specific goals, such as using an external S3-compatible object store

TIP: for a refresher on YAML syntax, we recommend Learn YAML in Y minutes and YAML Multiline Strings

You must generate your manifests and commit them to a git repo before ArgoCD can deploy them to your cluster.

TIP: the deployKF ArgoCD Plugin removes the need to generate and commit manifests to a git repo

The generate command of the deploykf CLI creates a manifests folder for a specific version of deployKF and one or more values files:

deploykf generate \
    --source-version "0.1.0" \
    --values ./custom-values.yaml \
    --output-dir ./GENERATOR_OUTPUT

After running deploykf generate, you will likely want to commit the changes to your repo:

# for example, to directly commit changes to the 'main' branch of your repo
git add GENERATOR_OUTPUT
git commit -m "my commit message"
git push origin main

WARNING: any manual changes made in the --output-dir will be overwritten each time the deploykf generate command runs, so please only make changes in your --values files. If you find yourself needing to make manual changes, this indicates we might need a new value, so please raise an issue to help us improve the project!

TIP: the --source-version can be any valid deployKF version, see the changelog for a list of versions

TIP: if you specify --values multiple times, they will be merged with later ones taking precedence (note, YAML lists are not merged, they are replaced in full)

The only manifest you need to manually apply is the ArgoCD app-of-apps, which creates all the other ArgoCD applications.

The app-of-apps.yaml manifest is generated at the root of your --output-dir folder, so you can apply it with:

kubectl apply --filename GENERATOR_OUTPUT/app-of-apps.yaml

If this is the first time you are using ArgoCD, you will need to retrieve the initial password for the admin user:

echo $(kubectl -n argocd get secret/argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

This kubectl command will port-forward the argocd-server Service to your local machine:

kubectl port-forward --namespace "argocd" svc/argocd-server 8090:https

You should now see the ArgoCD interface at https://localhost:8090, where you can log in with the admin user and the password you retrieved in the previous step.

You can now sync the ArgoCD applications which make up deployKF.

WARNING: you must sync each "group" of applications in the same order as the table to avoid dependency issues

WARNING: some applications, specifically dkf-dep--cert-manager and dkf-core--deploykf-profiles-generator may fail to sync on the first attempt, simply wait a few seconds and try the sync again

TIP: you may also sync the applications with the argocd CLI, but we recommend syncing with the web interface when you are first getting started so you can debug any issues:

# expose ArgoCD API server
kubectl port-forward svc/argocd-server -n argocd 8090:https

# get the admin password (if you have not changed it)
argocd "admin" initial-password -n argocd

# log in to ArgoCD
ARGOCD_PASSWORD="<YOUR_PASSWORD_HERE>"
argocd login localhost:8090 --username "admin" --password "$ARGOCD_PASSWORD" --insecure

# sync the apps
argocd app sync "deploykf-app-of-apps"
argocd app sync -l "app.kubernetes.io/component=deploykf-dependencies"
argocd app sync -l "app.kubernetes.io/component=deploykf-core"
argocd app sync -l "app.kubernetes.io/component=deploykf-opt"
argocd app sync -l "app.kubernetes.io/component=deploykf-tools"
argocd app sync -l "app.kubernetes.io/component=kubeflow-dependencies"
argocd app sync -l "app.kubernetes.io/component=kubeflow-tools"

If you have not configured a public Service for your deploykf-istio-gateway, you may access the deployKF web interface with kubectl port-forwarding.

First, you will need to add some lines to your /etc/hosts file (this is needed because Istio uses the "Host" header to route requests to the correct VirtualService).

For example, if you have set the deploykf_core.deploykf_istio_gateway.gateway.hostname value to "deploykf.example.com", you would add the following lines:

127.0.0.1 deploykf.example.com
127.0.0.1 argo-server.deploykf.example.com
127.0.0.1 minio-api.deploykf.example.com
127.0.0.1 minio-console.deploykf.example.com

This kubectl command will port-forward the deploykf-gateway Service to your local machine:

kubectl port-forward \
  --namespace "deploykf-istio-gateway" \
  svc/deploykf-gateway 8080:http 8443:https

You should now see the deployKF dashboard at https://deploykf.example.com:8443/, where you can use one of the following credentials (if you have not changed them):

WARNING:

Changing the owner of a profile requires manual steps! Therefore, it's common to leave [email protected] as the owner of all profiles and simply give it a strong password. (Even once you integrate your identity provider)

The [email protected] user does not have access to the "MinIO Console" or "Argo Workflows Server" interfaces, this is because it is not a "member" of any profile in the default values.

Now that you have a working ML Platform, you might want to dive into some of the following topics:

This error has been discussed in the upstream kubeflow repo, to resolve it you will need to increase your system's open/watched file limits.

On linux, you may need to increase the fs.inotify.max_user_* sysctl values, here are some values which users have reported to work:

1. Modify /etc/sysctl.conf to include the following lines:
   • fs.inotify.max_user_instances = 1280
   • fs.inotify.max_user_watches = 655360
2. Reload sysctl configs by running sudo sysctl -p