Kubernetes Custom Dev Environment - Home directory is formatted #16344

rhythm-devopspro · 2024-06-27T09:19:04Z

rhythm-devopspro
Jun 27, 2024

Setup

I have installed coder on my kubernetes cluster
I have created a template for kubernetes deployment environments
I created a custom docker image using codercom/enterprise-base:ubuntu as base image, to install some pre-required things like nvm, gvm, pyenv and infisical
When I run the docker image directly using docker run command, everything works exactly as expected, I have nodejs, golang, and python installed.
But when I set that image in kubernetes template on coder, when I create a workspace, the user's home directory /home/coder gets formatted, and all the installed apps like nvm, pyenv and gvm and .bashrc file, all gets removed, thus not usable in the environment
But apps installed using apt are there like infisical, that's not removed.

Expected Result

The custom software I installed and configured in the docker image should persist when used in coder workspace.

Dockerfile Used

FROM codercom/enterprise-base:ubuntu

WORKDIR /home/coder
ENV HOME=/home/coder
ENV NODE_VERSION=20

USER root
# Update libs
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Asia/Kolkata

RUN apt-get update && \
    apt-get install -y locales && \
    locale-gen en_IN.UTF-8 && \
    sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
    sudo sh -c "echo $TZ > /etc/timezone"

ENV LANG=en_IN.UTF-8
ENV LANGUAGE=en_IN:en
ENV LC_ALL=en_IN.UTF-8

USER coder

RUN sudo apt update
RUN sudo apt install -y \
    libbz2-dev \
    libncurses5-dev \
    libncursesw5-dev \
    libffi-dev \
    libreadline-dev \
    libssl-dev \
    zlib1g-dev \
    libsqlite3-dev \
    liblzma-dev \
    tk-dev \
    libgdbm-dev \
    libdb-dev \
    libexpat1-dev \
    libpcap-dev \
    libgmp-dev \
    build-essential \
    checkinstall

# Install Infisical
RUN curl -1sLf \
    'https://dl.cloudsmith.io/public/infisical/infisical-cli/setup.deb.sh' \
    | sudo -E bash
RUN sudo apt-get update && sudo apt-get install -y infisical

# Install NVM and Node
RUN curl https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

#Install Yarn and Turbo
RUN . $HOME/.nvm/nvm.sh && npm install -g yarn turbo

# Install pyenv
RUN curl https://pyenv.run | bash && \
    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \
    echo 'export PATH="$PYENV_ROOT/bin:$PYENV_ROOT/shims:$PATH"' >> ~/.bashrc && \
    echo 'eval "$(pyenv init -)"' >> ~/.bashrc

# Install Python 3.12.4 using pyenv
RUN $HOME/.pyenv/bin/pyenv install 3.12.4 && $HOME/.pyenv/bin/pyenv global 3.12.4 && $HOME/.pyenv/bin/pyenv rehash  

# Install GVM
RUN sudo apt-get install -y bison
RUN bash -c "curl -s -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer | bash"
RUN bash -c "source $HOME/.gvm/scripts/gvm && \
    gvm install go1.21.0 -B && \
    gvm use go1.21.0 --default"

ENV GVM_ROOT=$HOME/.gvm
ENV PATH=$GVM_ROOT/bin:$PATH
ENV GOROOT_BOOTSTRAP=$GVM_ROOT/gos/go1.21.0
ENV GOPATH=$HOME/go
ENV PATH=$GOPATH/bin:$GOROOT_BOOTSTRAP/bin:$PATH

RUN echo "export INFISICAL_TOKEN=\$(infisical login --method=universal-auth --client-id=\$INFISICAL_CLIENT_ID --client-secret=\$INFISICAL_CLIENT_SECRET --plain --silent)" >> ~/.bashrc

Public Docker Image

devopsprosamples/development-environment:coder
1st try to run image directly using command docker run -it devopsprosamples/development-environment:coder bash
Then try to run node -v, python --version, go version, nvm -v.
Now use this image as coder workspace image, then create a workspace, and use the terminal to run the same commands again.

matifali · 2025-01-30T14:33:38Z

matifali
Jan 30, 2025
Maintainer

Hi @rhythm-devopspro. Sorry for not seeing this before.

Coming to the issue is that the template mounts a volume over the home directory, which shadows the file system of the container at that path. You can either install the tools outside the home or install them in the startup_script.

1 reply

imaxun Sep 1, 2025

Can't this be deleted? I think this is very unfriendly.

imaxun · 2025-09-01T06:59:56Z

imaxun
Sep 1, 2025

I have the same problem now, this is my dockerfile

ARG base
FROM ${base}

ARG JAVA_VERSION

# Dazzle does not rebuild a layer until one of its lines are changed. Increase this counter to rebuild this layer.
ENV TRIGGER_REBUILD=1
ENV SDKMAN_DIR=/usr/local/lib/.sdkman

RUN curl -fsSL "https://get.sdkman.io" | bash \
    && bash -c ". $SDKMAN_DIR/bin/sdkman-init.sh \
             && sed -i 's/sdkman_selfupdate_feature=true/sdkman_selfupdate_feature=false/g' $SDKMAN_DIR/etc/config \
             && sdk install gradle \
             && sdk install maven \
             && sdk flush archives \
             && sdk install java ${JAVA_VERSION} \
             && sdk install java 8.0.462-kona \
             && sdk default java ${JAVA_VERSION} \
             && sdk flush temp \
             && mkdir $HOME/.m2"

RUN mkdir -p $HOME/.bashrc.d $HOME/.zshrc.d $HOME/.m2
COPY 99-java $HOME/.bashrc.d/99-java
COPY 99-java $HOME/.zshrc.d/99-java
COPY maven-settings.xml $HOME/.m2/settings.xml

1 reply

matifali Sep 1, 2025
Maintainer

@imaxun, The issue is with how Docker volumes work. A volume overshadows the filesystem in the image. You should either install all tools outside of /home/coder or install them in your startup_script

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Kubernetes Custom Dev Environment - Home directory is formatted #16344

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Kubernetes Custom Dev Environment - Home directory is formatted #16344

Uh oh!

rhythm-devopspro Jun 27, 2024

Replies: 2 comments · 2 replies

Uh oh!

Uh oh!

matifali Jan 30, 2025 Maintainer

Uh oh!

imaxun Sep 1, 2025

Uh oh!

Uh oh!

imaxun Sep 1, 2025

Uh oh!

matifali Sep 1, 2025 Maintainer

rhythm-devopspro
Jun 27, 2024

Replies: 2 comments 2 replies

matifali
Jan 30, 2025
Maintainer

imaxun
Sep 1, 2025

matifali Sep 1, 2025
Maintainer