{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"NOTE: This documentation is under heavy development and is subject to change daily.","type":"text","marks":[{"type":"strong"}]}]},{"type":"panel","attrs":{"panelIconId":"atlassian-warning","panelIcon":":warning:","panelColor":"#FFF0B3","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Please note that there is no internet access from any nodes inside the DGX cluster for end users. As such, all work needs to be manually copied by the user onto and off of the cluster using ","type":"text"},{"text":"scp","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"rsync","type":"text","marks":[{"type":"code"}]},{"text":", or ","type":"text"},{"text":"sftp","type":"text","marks":[{"type":"code"}]},{"text":" methods.","type":"text"}]},{"type":"paragraph","content":[{"text":"For containers this means that binding the appropriate directories for accessing datasets and saving output are crucial to avoid losing time and data. As such, please be sure to review the ","type":"text"},{"text":"Container Directory Mounts","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/PreReleaseUKYHPCDocs/pages/58621980/Containerized+Jobs#Container-Directory-Binds"}}]},{"text":" section before you start running large-scale jobs.","type":"text"}]},{"type":"paragraph","content":[{"text":"Additionally, the containers that run on the DGX are unprivileged, unlike normal Docker containers. For this reason, it is important to understand the file permissions inside of containers and where files can be written. Please refer to ","type":"text"},{"text":"File Permissions in DGX Containers","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/PreReleaseUKYHPCDocs/pages/edit-v2/58621980#File-Permissions-in-DGX-Containers"}}]},{"text":" to better understand how to work within your permissions.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"PREFACE: If you are not familiar with Slurm job submissions, please review a brief primer at ","type":"text","marks":[{"type":"strong"}]},{"text":"Submitting jobs on the DGX cluster (for first-time users)","type":"text","marks":[{"type":"strong"},{"type":"link","attrs":{"href":"https://uk/wiki/spaces/PreReleaseUKYHPCDocs/pages/59703305"}}]},{"text":". ","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph"},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"6"},"outline":{"value":"false"},"style":{"value":"none"},"type":{"value":"list"},"printable":{"value":"false"}},"macroMetadata":{"macroId":{"value":"18b596ec-15ad-42b1-b518-93ea1f73de66"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"3c81c362-29b7-4bd9-82b8-0f7e1fe93e19"}},{"type":"heading","attrs":{"level":1},"content":[{"text":"Containers","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Background","type":"text"}]},{"type":"paragraph","content":[{"text":"In this guide, we’ll review how containerized jobs are run on the DGX HPC. Users are able to run unprivileged Docker containers on the DGX HPC by leveraging Nvidia’s ","type":"text"},{"text":"ENROOT","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/NVIDIA/enroot"}}]},{"text":" sandboxing tool in conjunction with their ","type":"text"},{"text":"Pyxis","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/NVIDIA/pyxis"}}]},{"text":" Slurm SPANK plugin. These two tools convert ","type":"text"},{"text":"srun","type":"text","marks":[{"type":"code"}]},{"text":" parameters and ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":" script directives into Docker settings (i.e. image and registry import, directory binding, etc.) that are run according to the privileges of the submitting user.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Simple Example - Running an Ubuntu Container","type":"text"}]},{"type":"paragraph","content":[{"text":"A very simple ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":" script to run an Ubuntu container:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n#SBATCH -p defq\n#SBATCH --time=00:15:00\n#SBATCH -A \n#SBATCH --container-image ubuntu\n\ngrep PRETTY /etc/os-release","type":"text"}]},{"type":"panel","attrs":{"panelIconId":"atlassian-info","panelIcon":":info:","panelColor":"#B3D4FF","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Note that ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code"}]},{"text":" is never called directly on the DGX, but rather the entire ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":" script is executed inside the container as if ","type":"text"},{"text":"docker","type":"text","marks":[{"type":"code"}]},{"text":" has been called.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"The important directive is the line ","type":"text"},{"text":"#SBATCH --container-image ubuntu","type":"text","marks":[{"type":"code"}]},{"text":". This tells Pyxis and ENROOT to build an unprivileged execution environment from the ","type":"text"},{"text":"ubuntu","type":"text","marks":[{"type":"code"}]},{"text":" Docker image (imported from DockerHub in this instance) and execute the ","type":"text"},{"text":"grep PRETTY /etc/os-release","type":"text","marks":[{"type":"code"}]},{"text":" inside of the container once running. ","type":"text"}]},{"type":"paragraph","content":[{"text":"After submitting this script using ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":", the ","type":"text"},{"text":"slurm-####.out","type":"text","marks":[{"type":"code"}]},{"text":" file will provide feedback from Pyxis regarding the successful loading of the Docker image as well as the results of the commands run:","type":"text"}]},{"type":"codeBlock","content":[{"text":"pyxis: imported docker image: ubuntu\nPRETTY_NAME=\"Ubuntu 22.04.4 LTS\"","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"File Permissions in DGX Containers","type":"text"}]},{"type":"paragraph","content":[{"text":"For those familiar with running Docker containers regularly, you’ll likely be used to containers being run as privileged, meaning inside the container your are the ","type":"text"},{"text":"root","type":"text","marks":[{"type":"code"}]},{"text":" user and can modify the files as needed. Additionally in privileged containers you can mount most directories on the host system as the ","type":"text"},{"text":"root","type":"text","marks":[{"type":"code"}]},{"text":" user as well. Through ","type":"text"},{"text":"ENROOT","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/NVIDIA/enroot"}}]},{"text":" and ","type":"text"},{"text":"Pyxis","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/NVIDIA/pyxis"}}]},{"text":", containers on the DGX run as unprivileged sandboxes. This means that the container is run as the submitting user and the rights with regards to mounting directories and reading or writing files match those of the submitting user. Additionally, inside of the container, the account is that of the submitting user rather than root, which means if that user doesn’t have write access to a particular directory they will receive an error trying to create or write a file in that directory inside the container. Similarly, if a directory is mounted that the user does not have the correct permissions for, they will receive an access error when attempting to interact with that directory inside the container.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using Nvidia GPUs in Containers","type":"text"}]},{"type":"paragraph","content":[{"text":"There are two steps to enabling Nvidia GPUs in containers on the DGX: supplying GPUs via Slurm, and activating the GPUs in the container itself.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Supplying GPUs via Slurm","type":"text"}]},{"type":"paragraph","content":[{"text":"GPUs for a Slurm task are enabled by supplying one or more of the GPU-related #SBATCH directives in your Slurm script:","type":"text"}]},{"type":"codeBlock","content":[{"text":"GPU scheduling options:\n --cpus-per-gpu=n number of CPUs required per allocated GPU\n -G, --gpus=n count of GPUs required for the job\n --gpu-bind=... task to gpu binding options\n --gpu-freq=... frequency and voltage of GPUs\n --gpus-per-node=n number of GPUs required per allocated node\n --gpus-per-socket=n number of GPUs required per allocated socket\n --gpus-per-task=n number of GPUs required per spawned task\n --mem-per-gpu=n real memory required per allocated GPU","type":"text"}]},{"type":"paragraph","content":[{"text":"We’ll use ","type":"text"},{"text":"--gpus-per-task","type":"text","marks":[{"type":"code"}]},{"text":" as an example:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n#SBATCH -p defq\n#SBATCH --ntasks=1\n#SBATCH --gpus-per-task=1 # <---- Attaches a GPU to the Slurm task\n#SBATCH --cpus-per-gpu=16 # <---- Associated option to add a number of CPUs for each GPU\n#SBATCH --container-image nvcr.io\\#nvidia/clara/clara-parabricks:4.2.1-1\n\npbrun -h","type":"text"}]},{"type":"paragraph","content":[{"text":"When you launch a Slurm job using this script, the Slurm scheduler will schedule one GPU and attach it to the container. It is then up to the image to make use of the GPU, which brings us to the next “step”: activating the GPUs in the container itself.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Activating the GPUs in the container itself","type":"text"}]},{"type":"paragraph","content":[{"text":"This is a slightly complicated topic, but the good news is that for most Nvidia and Nvidia-derived containers this is done for you so you won’t have to do anything. In a nutshell, in order for containers to see the Nvidia GPUs that have been allocated to them, they require some environmental variables to exist before the container is created. The two environmental variables are:","type":"text"}]},{"type":"codeBlock","content":[{"text":"NVIDIA_VISIBLE_DEVICES=all\nNVIDIA_DRIVER_CAPABILITIES=compute,utility","type":"text"}]},{"type":"paragraph","content":[{"text":"There are two ways to achieve this: using ","type":"text"},{"text":"ENV","type":"text","marks":[{"type":"code"}]},{"text":" commands in your ","type":"text"},{"text":"Dockerfile","type":"text","marks":[{"type":"code"}]},{"text":" if you’re building your container or using ","type":"text"},{"text":"export","type":"text","marks":[{"type":"code"}]},{"text":" commands from your login node shell before submitting the job to Slurm.","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Adding Nvidia GPU ENV lines to your Dockerfile","type":"text"}]},{"type":"paragraph","content":[{"text":"This is usually already done for Nvidia images (such as CUDA or Parabricks), but if the image you want to use doesn’t have this environmental variables already, you can do so by adding the following two lines to your ","type":"text"},{"text":"Dockerfile","type":"text","marks":[{"type":"code"}]},{"text":" and rebuilding your image:","type":"text"}]},{"type":"codeBlock","content":[{"text":"ENV NVIDIA_VISIBLE_DEVICES=all\nENV NVIDIA_DRIVER_CAPABILITIES=compute,utility","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Using export to Set Nvidia GPU Environmental Variables","type":"text"}]},{"type":"paragraph","content":[{"text":"If you don’t have an easy way to rebuild or augment an existing image, you can still set this environmental variables before creating the container. This can be done by running ","type":"text"},{"text":"export","type":"text","marks":[{"type":"code"}]},{"text":" commands to set environmental variables in your session:","type":"text"}]},{"type":"codeBlock","content":[{"text":"export NVIDIA_VISIBLE_DEVICES=all\nexport NVIDIA_DRIVER_CAPABILITIES=compute,utility","type":"text"}]},{"type":"paragraph","content":[{"text":"The lines add the required environmental variables and you can then run ","type":"text"},{"text":"sbatch