Investigate GPU support

[bpmct]

Some users will want to mount a GPU to an envbuilder-backed workspace. Can we investigate in which scenarios (if any) this works today and if/how we can patch upstream Kaniko to improve the experience?

Related

• Kaniko issue: Symlink issue on k8s GPU node with /lib/firmware/nvidia/525.xx GoogleContainerTools/kaniko#3006
• Internal Slack message

[mtojek]

@BrunoQuaresma We may need to write a mini-RFC describing the status quo.

[BrunoQuaresma]

After talk to @mtojek I think I have a good plan:

• Try to run envbuilder in a regular environment
  • Spin up a regular k8s cluster on Google Cloud
  • Try to run envbuilder with a hello world image and see if it works
• Try to reproduce the user error by running envbuilder with a GPU
  • Spin up a k8s cluster using a NVidia GPU on Google Cloud
  • Try to run envbuilder with a hello world image and see if it works
  • Try to find a workaround
  • Investigate possible solutions using diff builders besides kaniko

[BrunoQuaresma]

@bpmct I tried to use envbuilder in a GPU environment and it worked as expected. Here is how I made it:

• Spin up a k8s cluster with GPU support on GKE
  • GKE version 1.27.13-gke.1000000
  • Machine type n1-standard-4
  • GPU accelerators (per node) 2 x NVIDIA T4
• Setup a test repo with devcontainer using a Nvidia test image
  • Example: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/k8s/containers/cuda-sample
  • NVidia example image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
• Set the following envbuilder config
  • GIT_URL as https://github.com/BrunoQuaresma/envbuilder-gpu-test
  • INIT_SCRIPT as /tmp/vectorAdd

This is the output:

envbuilder - Build development environments from repositories in a container
#1: 📦 Cloning https://github.com/BrunoQuaresma/envbuilder-gpu-test to /workspaces/envbuilder-gpu-test...
#1: Enumerating objects: 4, done.
#1: Counting objects:  25% (1/4)
#1: Counting objects:  50% (2/4)
#1: Counting objects:  75% (3/4)
#1: Counting objects: 100% (4/4)
#1: Counting objects: 100% (4/4), done.
#1: Compressing objects:  50% (1/2)
#1: Compressing objects: 100% (2/2)
#1: Compressing objects: 100% (2/2), done.
#1: Total 4 (delta 0), reused 4 (delta 0), pack-reused 0
#1: 📦 Cloned repository! [193.807769ms]
#2: Deleting filesystem...
#2: 🏗️ Building image...
#2: Retrieving image manifest nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
#2: Retrieving image nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2 from registry nvcr.io
#2: Built cross stage deps: map[]
#2: Retrieving image manifest nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
#2: Returning cached image manifest
#2: Executing 0 build triggers
#2: Building stage 'nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2' [idx: '0', base-idx: '-1']
#2: 🏗️ Built image! [3.019338331s]
#3: no user specified, using root
#3: 🔄 Updating the ownership of the workspace...
#3: 👤 Updated the ownership of the workspace! [449.651µs]
=== Running the init command /bin/sh [-c /tmp/vectorAdd] as the "root" user...
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

@bpmct do you think we can get more details from the user?

[BrunoQuaresma]

I am closing this for now until we have more context from the user.

[marrotte]

Try using the NVIDIA k8s device plugin (DaemonSet) and not a NVIDIA container image, e.g.:

https://github.com/NVIDIA/k8s-device-plugin

https://learn.microsoft.com/en-us/azure/aks/gpu-cluster?tabs=add-ubuntu-gpu-node-pool#nvidia-device-plugin-installation

This is the recommended approach when using GPU-enabled node pools for Azure Linux.

[johnstcn]

Try using the NVIDIA k8s device plugin (DaemonSet)

@marrotte FYI while we tested using GKE, the cluster we tested on does use the device plugin. However, it appears to be a customized version for GKE COS, and I will freely admit that cluster is a bit old.

What Kubernetes version are you seeing issues with on AKS?

not a NVIDIA container image, e.g.:

This container image is the one NVIDIA recommends to test GPU support (c.f. https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#running-gpu-jobs)

What image would you recommend instead as a test? I note that the Azure docs you linked reference a separate MNIST test image.

[marrotte]

@johnstcn I'm seeing the issue on:

• K8s Rev: v1.27.7
• Node image: AKSUbuntu-2204gen2containerd-202401.09.0
• Plugin image: mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.11
• Pod image = "ghcr.io/coder/envbuilder:0.2.9"

[nikawang]

@BrunoQuaresma
Still have issues via using your test repo on AKS agains ghcr.io/coder/envbuilder:0.2.9

root@coder-daniel-kkk-copy-75f96d6c7b-dxxdn:/# nvidia-smi
NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure that the NVIDIA Display Driver is properly installed and present in your system.
Please also try adding directory that contains libnvidia-ml.so to your system PATH.
root@coder-daniel-kkk-copy-75f96d6c7b-dxxdn:/# cd /tmp
root@coder-daniel-kkk-copy-75f96d6c7b-dxxdn:/tmp# ls
coder.wTqTN7  vectorAdd
root@coder-daniel-kkk-copy-75f96d6c7b-dxxdn:/tmp# ./vectorAdd
[Vector addition of 50000 elements]
Failed to allocate device vector A (error code CUDA driver version is insufficient for CUDA runtime version)!

/usr/lib64# ll
total 176176
drwxr-xr-x  2 root root     4096 Jun 13 14:00 ./
drwxr-xr-x 14 root root     4096 Jun 13 13:49 ../
lrwxrwxrwx  1 root root       42 Jun 13 13:49 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2*
-rwxr-xr-x  1 root root 28392536 Jun 13 12:01 libcuda.so.550.54.15*
-rwxr-xr-x  1 root root 10524136 Jun 13 12:01 libcudadebugger.so.550.54.15*
-rwxr-xr-x  1 root root   168744 Jun 13 12:01 libnvidia-allocator.so.550.54.15*
-rwxr-xr-x  1 root root   398968 Jun 13 12:01 libnvidia-cfg.so.550.54.15*
lrwxrwxrwx  1 root root       36 Jun 13 14:00 libnvidia-ml.so -> /usr/lib64/libnvidia-ml.so.550.54.15*
-rwxr-xr-x  1 root root  2078360 Jun 13 12:01 libnvidia-ml.so.550.54.15*
-rwxr-xr-x  1 root root 86842616 Jun 13 12:01 libnvidia-nvvm.so.550.54.15*
-rwxr-xr-x  1 root root 23293568 Jun 13 12:01 libnvidia-opencl.so.550.54.15*
-rwxr-xr-x  1 root root    10168 Jun 13 12:01 libnvidia-pkcs11.so.550.54.15*
-rwxr-xr-x  1 root root 28670368 Jun 13 12:01 libnvidia-ptxjitcompiler.so.550.54.15*

[nikawang]

Oh, I fixed it by

echo "/usr/lib64" > /etc/ld.so.conf.d/customized.conf 
ldconfig