Red Hat OpenShift AI (RHOAI) builds on the capabilities of Red Hat OpenShift to provide a single, consistent, enterprise-ready hybrid AI and MLOps platform. It provides tools across the full lifecycle of AI/ML experiments and models including training, serving, monitoring, and managing AI/ML models and AI-enabled applications. This is my personal repository to test and play with some of its most important features.

• 1. Red Hat Training
• 2. RHOAI Architecture
  • 2.1. Deprecated Components
• 3. Installation
  • 3.1. Installation on non-4.20 OCP
  • 3.2. Let’s install!!
• 4. Things you should know!
  • 4.1. NVIDIA GPU nodes
  • 4.2. NVIDIA GPU Partitioning
  • 4.3. Data Connection Pipelines S3 Bucket Secret
  • 4.4. Managing distributed workloads
  • 4.5. Model Registry
• 5. Monitoring, Safety and Evaluation
• 6. Deploying an Inference Server
• 7. Extra Components
  • 7.1. OpenShift Lightspeed
  • 7.2. LLS Playground
  • 7.3. MinIO
  • 7.4. Open WebUI
  • 7.5. Milvus
• 8. Extra documentation
  • 8.1. Useful links
  • 8.2. Study sources
  • 8.3. Nice demos and projects to take a look:

RHOAI is a product under continuous improvement, so this repo will be outdated at some point in time. I recommend you to refer to the Official documentation to check the latest features or you can try the official trainings.

Red Hat OpenShift AI (RHOAI) is a platform for data scientists, AI practitioners, developers, machine learning engineers, and operations teams to prototype, build, deploy, and monitor AI models. This is a wide variety of audience that needs different kinds of training. For that reason, there are several courses that will help you to understand RHOAI from all angles:

• AI262 - Introduction to Red Hat OpenShift AI: About configuring Data Science Projects and Jupyter Notebooks.

• AI263 - Red Hat OpenShift AI Administration: About installing RHOAI, configuring users and permissions and creating Custom Notebook Images.

• AI264 - Creating Machine Learning Models with Red Hat OpenShift AI: About training models and enhancing the model training.

• AI265 - Deploying Machine Learning Models with Red Hat OpenShift AI: About serving models on RHOAI.

• AI266 - Automating AI/ML Workflows with Red Hat OpenShift AI: About creating Data Science Pipelines, and Elyra and Kubeflow Pipelines.

• AI267 - Developing and Deploying AI/ML Applications on Red Hat OpenShift AI: All the previous courses altogether.

The following diagram depicts the general architecture of a RHOAI deployment, including the most important components:

• aipipelines: This enables you to build portable machine learning workflows. It is based on Kubeflow Pipelines and you don’t need to install OCP Pipelines operator. (Formerly known as datasciencepipelines)

• dashboard: Provides the RHOAI dashboard.

• feastoperator: Feast is an open-source feature store for machine learning. It helps manage, store, and serve features for training and inference, enabling feature reuse across ML projects. NOTE: Removed by default, must be explicitly enabled.

• kserve: RHOAI uses Kserve to serve large language models that can scale based on demand.

• kueue: Kueue is a Kubernetes-native job queueing system that provides quota management, resource sharing, and prioritization for batch workloads. Supports configuration of default cluster and local queue names. NOTE: It is installed separately as an OpenShift component, that’s why it should be set to Unmanaged.

• llamastackoperator: Llama Stack is a set of APIs and tools for building AI applications with Meta’s Llama models. This operator helps deploy and manage Llama-based AI applications on OpenShift. NOTE: Removed by default, must be explicitly enabled.

• modelregistry: Model Registry provides a central repository for data scientists to store, version, and manage machine learning models. It enables model governance, lineage tracking, and sharing across teams. Requires specifying a registriesNamespace for model registry deployments.

• ray: Component to run the data science code in a distributed manner.

• trainingoperator: Training Operator provides Kubernetes-native support for distributed training of machine learning models using popular frameworks like PyTorch, TensorFlow, and MPI.

• trustyai: TrustyAI provides AI explainability and fairness monitoring capabilities. It helps data scientists and ML engineers understand model behavior, detect bias, and ensure AI transparency and accountability.

• workbenches: Workbenches are containerized and isolated working environments for data scientists to examine data and work with data models. Data scientists can create workbenches from an existing notebook container image to access its resources and properties. Workbenches are associated to container storage to prevent data loss when the workbench container is restarted or deleted.

Installing RHOAI is not as simple as installing and configuring other operators on OpenShift. This product provides integration with hardware like NVIDIA and Intel GPUs, automation of ML workflows and AI training, and deployment of LLMs. For that reason, I’ve created an auto-install.sh script that will do everything for you:

1. If the installation is IPI AWS, it will create MachineSets for nodes with NVIDIA GPUs (Currently, g5.4xlarge).

2. Install all the operators that RHOAI depends on:

   • Red Hat OpenShift Leader Worker Set Operator to manage leader worker sets. This is a requirement for llm-d feature.

   • Red Hat Build of Kueue to manage distributed workloads. This is a requirement for GPUaaS feature.

   • Node Feature Discovery and Nvidia GPU Operator to discover and configure nodes with GPU.

   • Authorino, to enable token authorization for models deployed with RHOAI.

   • Note: Single-Model Serving (Model Mesh Serving) is not supported anymore in RHOAI 3.0, so Service Mesh 2.x and Serverless operators are not installed as prerequisites.

3. Optionally install and configure OpenShift Data Foundation (ODF) in Multicloud Object Gateway (MCG) mode. This is a lightweight alternative that allows us to use the AWS S3 object storage the same way that we will then use Object storage on Baremetal using ODF.

4. Installs the actual RHOAI operator and configures the installation with some defaults, enabling NVIDIA acceleration and llm-d feature.

5. Deploys a new Data Science Project called RHOAI Playground enabling pipelines and deploying a basic Notebook for testing.

• Monitoring: To ensure the transparency, fairness, and reliability of your data science models in OpenShift AI for Bias and Data Drift. Configure and set up TrustyAI for your project, and then perform the following checks:

  • Bias: Check for unfair patterns or biases in data and model predictions to ensure your model’s decisions are unbiased.

  • Data drift: Detect changes in input data distributions over time by comparing the latest real-world data to the original training data. Comparing the data identifies shifts or deviations that could impact model performance, ensuring that the model remains accurate and reliable.

• Safety: To ensure that your machine-learning models are transparent, fair, and reliable. The TrustyAI Guardrails Orchestrator service is a tool to invoke detections on text generation inputs and outputs, as well as standalone detections.

• Evaluation: To ensure your OpenShift AI models for accuracy, relevance, and consistency. Evaluate your AI systems to generate an analysis of your model’s ability by using the following TrustyAI tools:

  • LM-Eval: Use TrustyAI to monitor your LLM against a range of different evaluation tasks and to ensure the accuracy and quality of its output. Features such as summarization, language toxicity, and question-answering accuracy are assessed to inform and improve your model parameters.

  • RAGAS: Use Retrieval-Augmented Generation Assessment (RAGAS) with TrustyAI to measure and improve the quality of your RAG systems in OpenShift AI. RAGAS provides objective metrics that assess retrieval quality, answer relevance, and factual consistency.

  • Llama Stack: Use Llama Stack components and providers with TrustyAI to evaluate and work with LLMs.

As the Model Registry is still Tech Preview, we still keep documentation about how to sync manually models using an OCP Job and then serve it with OpenShift AI. You can use the following Application that points to a Helm Chart that automates it: