bdereims’ hacks

OCI - one more Cloud

2022-09-02T08:00:00+00:00

For an interne project I have to use Oracle Cloud Infrastructure. I was very skeptical about it and I’m wondering why OCI? Anyway, it was time to discover what it is and automate things to avoid to do again boring tasks. Today there are so many Infrastructure as Code solutions (pulumi, terraform and others) but it always good to go with The Hard Way to discover a solution. This is why I decided to write some bash scripts executing oci cli commands covering my needs:

create a compartment, it’s like a tenant
create VCN, like a VPC
create Public & Private Subnets
create internet gateway
create ingress firewall rules
create instance of my Linux sixty9 jumpBox, configure iptables, wireguard VPN and DNS
possibility to deploy VMs as master/worker kubernetes nodes and bare metal worker kubernetes node with GPU in same VCN
use kubespray to setup kubernetes from sixty9
obviously destroy all the things

These scripts give me flexibility to test things, to add nodes when it’s needed. A perfect use case for me. I share it with you in a git repo a subpart of cloud-garage repo.

OCI portal:

VCN configuration:

With wireguard, I can directly interact with kubernetes from my laptop without exposing it publicly:

Finally OCI is an excellent solution. I don’t regret discovering Oracle Cloud Infrastructure.

Enjoy!

Run NGC container in Windows 11

2022-07-21T13:37:15+00:00

I’m a big server Linux fan and have used laptop macOS for over 10 years now for its unix like shell. Windows has now the famous Windows Subsystem for Linux, time to give it a shot! Furthermore, I want also escape Docker Desktop fees… Let’s try something fancy.

First step is to activate WSL on your Win11 box. So, to do that you need to enable 2 features like below:

Followed in command prompt: wsl --install --distribution Ubuntu-20.04

To run containers, we need to install podman. It’s a free open source docker engine alternative. If you hope to use docker-ce in wsl you wrong, it needs Docker Desktop that is not free if you’re working for large enterprise.

Within WSL shell:

# source /etc/os-release
# sh -c "echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"
# wget -nv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_${VERSION_ID}/Release.key -O- | apt-key add -
# apt-get update -qq -y
# apt-get -qq --yes install podman
# podman version

You should get:

You have to set default registry: # echo 'unqualified-search-registries = ["docker.io"]' | sudo tee /etc/containers/registries.conf

Verify that nvidia driver is working propely: # nvidia-smi, you should get something like:

Install nvidia-container-toolkit (cf. https://github.com/henrymai/podman_wsl2_cuda_rootless )

Modify file /etc/nvidia-container-runtime/config.toml like this:

disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
#accept-nvidia-visible-devices-as-volume-mounts = false

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
#no-cgroups = false
no-cgroups = true
#user = "root:video"
ldconfig = "@/sbin/ldconfig.real"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
debug = "~/.local/nvidia-container-runtime.log"
log-level = "info"

# Specify the runtimes to consider. This list is processed in order and the PATH
# searched for matching executables unless the entry is an absolute path.
runtimes = [
    "docker-runc",
    "runc",
]

mode = "auto"

[nvidia-container-runtime.modes.csv]
mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

Configure hook with this lines:

Configure hook

sudo -i
mkdir -p /usr/share/containers/oci/hooks.d/
cat << EOF | sudo tee /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json
{
    "version": "1.0.0",
    "hook": {
        "path": "/usr/bin/nvidia-container-toolkit",
        "args": ["nvidia-container-toolkit", "prestart"],
        "env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ]
    },
    "when": {
        "always": true,
        "commands": [".*"]
    },
    "stages": ["prestart"]
}
EOF

Last step for configuration is to increase limits defined in /etc/security/limits.conf like below, replace [your-user] by what returns whoami:

[your-user] soft memlock unlimited
[your-user] hard memlock unlimited
[your-user] soft stack 65536
[your-user] hard stack 65536

If you not yet familiar with that is NGC, it’s time to visite NGC Catalog and the getting started. Create your account and install ngc cli.

Let’s try this example.

Launch example with: podman run -it --gpus all -p 8888:8888 -v $PWD:/projects --network=host nvcr.io/nvidia/tensorflow:20.08-tf1-py3 and follow jupyter notebook. You should able to run example:

Conclusion:

podman on wsl works great
with ngc container running localy, it’s easy to learn and test ML/AI topics
first step before to work with large scale environnement like Base Command

Ubunru 20.04 LTS Jumpbox

2022-07-19T08:00:00+00:00

It’s always good to have a VM Jumbox or a simple way to configure your machine! For this I wrote a bunch of automated scripts, for this annoying task. You will get the necessary setup and tools to perform your daily tasks. It comes with: kubectl, stern, docker engine, docker-compose, helm, govc, kubectx, kubens, extend-rootfs.sh, curl, jq, sshpass, unzip, tmux, bash-completion, startup-script.

Use this repo and execute as root bootsrap.sh. Only validated for freshly installed Ubuntu 20.04 LTS, as VM or bare metal.

Enjoy!

Nested SDDC for Testing/Lab/Demo

2022-06-24T17:00:00+00:00

cPodFactory is one of my projects during my VMware years. The main purpose is to automatically and dynamically create nested SDDC environments. It’s means you have the capability to create a separated but nested vSphere SDDC fully routed (dedicated L2 but L3 routed) configured and ready to use.

It comes with cpodctl command line giving you all necessary command to manage cPod lifecycle. A cPod is a lab or if you prefer the nested vSphere. You can:

Create a cPod: naked one (without esx but all networking facilities) or specifying number of esx
Delete a cPod entirely, caution - it’s fast without roll back
Deploy & configure vCenter accordingly your cPod
List all cPods
Deploy VCF cloudbuilder if you want to transform cPod into VCF cluster(s)
Retrieve the auto generated cPod password
Override some variables to customize your cPod
Use completion in bash to facilitate life with tab key

cpodctl cli:

A large Availability Zone running physical layer by VCF:

cPodFactory also offers:

Multi availability zones and connect them via wireguard: it’s possible to move workload across different az or to connect services on different cPods
DNS resolution for all az
VPN for client with OpenVPN, Windows/Mac/Linux compatible
DHCP for each L2 connectivity
Dynamic L3 routing with BGP, with the possibility to peer NSX Edge in a cPod
Nested NSX and nested VSAN
It comes with cPodEdge and cPodRouter VM template for connectivity and network services (based on PhotonOS Linux)
Utilities to automaticaly deploy OVA and Windows/Linux jumpHost in cPod

It needs for the physical layer:

at least 2 or 3 esx 6.7 + vCenter or VCF with 4 esx
vSAN for storage or any shared datastore (NFS, iSCSI, FC)
VLAN backend network or NSX overlay (NSX is not mandatory at all)

I ran it for 4 years to help colleagues to have serious labs and also used it for all my customer workshops. We have never had severe issue of availability or performance - thanks to vSphere for its capabilities to run heavy workloads. It has been deployed in several country interconnected via wireguard: Paris(FR), OVH(FR), Madrid(SP), Munich(DE), Dubai(UAE).

To give you some ideas of use cases:

Large Proof Of Concept with Tanzu running Telco NVFs and consuming +480Gb memory in kubernetes, 9 esx cPod
Demo cPod dedicated to vRealize Automation with NSX and vSAN connected with Public Cloud deploying hybrid App
multi VCF cPods in different regions (Paris <-> Dubai) moving VM workloads via HCX
Cloud Migration from cPod in OVH to VMC in AWS, moving VM via HCX, fully routed without application interruption, keeping on-prem DB
All new vSphere 8 cPod with Tanzu Grid 2.0 for training purpose
nVIDIA AI Enterprise naked cPod bringing agile AI for enterprise running on vSphere

This project is open source. You can use it, improve it but not in production. Nested is not supported/allowed for production.

Check out git repo - you find here all the necessary to install yours.

Enjoy!

Tanzu Konw It All

2022-06-24T16:59:00+00:00

If you want to quickly discover Tanzu or practice a lot kubernetes in order to prepare CKA or CKAD certification, I have thing for you…

Check out my TKIA git repo, you get here links,docs and usefull scripts that I made during so many customer workshops.

Enjoy!