Singularity Container Composer(User Guide)
- UKY CCS RCD
Analytics
The Singularity Container Composer hosted at https://containercomposer.ccs.uky.edu is a tool designed to simplify and streamline the management of Singularity containers, particularly in High-Performance Computing environments. It allows users to efficiently build, configure, and orchestrate their Singularity-based workflows. The GUI makes it easy for users to search the software and add them to the Singularity definition files and build custom Singularity containers with clicks.
How to use the Singularity Container Composer
Part1: Sign Up and Login
At the moment, the tool uses an independent account management system(not related to linkblue id or cilogon id). Users need to sign up their accounts to continue. In the future, we may integrate this part with UK LDAP or CILogon.
Users need accounts to use this tool. First time users should click the “Sign Up“ button on the top right of the main page which will take them to the sign up page. Then they need to fill in the form with the username, email and password. Please read and agree to the privacy policy and check the checkbox. Once all the fields are filled in correctly, click the green “Sign Up“ button at the bottom of the page.
Once the admin approves the account, the user will receive an email.
Note you should expect emails from no-reply@containercomposer.ccs.uky.edu
Then users can click the link in the email to go to the home page and then click the “Login“ button the top right corner.
After users click the “Login“ button, they will be redirected to the GUI and till now the preparations/prerequisites are all met.
Part2: Use the GUI to Create Custom Singularity Containers
Users will see the page above after they have logged in. In general, the page can be divided into 3 parts.
(1)On the left side is the workflow(Step 1 to Step 4) to build a custom Singularity container.
(2)On the right side is the real-time Singularity definition file. The displayed file is updated when the user clicks the GUI elements on the left side.
(3)On the top right is the user profile dropdown. User can click the icon on the right side of “Welcome“ and see it, which allows the user to see their Singularity Container build requests (the View My Requests button) or log out(the Log Out button).
Workflow:
Users are expected to follow the workflow to build custom Singularity Containers via the GUI(Step1 to Step4) on the left side. It is a step-by-step process and the steps cannot be jumped. For example, if you have not finished Step1 but go to Step2, an error message will be popped up.
Step1: Select Base Image
We are using docker images to build Singularity containers and this is only one of the approaches. If you are adept in creating Singularity definition files, you can jump to Step4 and submit your custom Singularity definition files to build containers.
We provide different base images for different Linux distributions (Debian based os such as Ubuntu and RHEL based os such as RockyLinux) and architectures(Linux x86_64 and Linux arm64). Users can select the base image based on what Linux distribution they like and which architecture will the container be running on.
Note: For LCC, MCC, and ECC, the clusters all have Linux x86_64 architecture so users can pick either one of the first two options. Currently, building Singularity containers using ARM64 base images(the last two options) is under development since they have to be built on a ARM64 VM.
After user selects any option in the dropdown menu, the Singularity definition file on the right side will be updated accordingly. The changes are highlighted in the same color.
Step2: Add System-level Software or Environment Variables
After the base image is selected, users can go to Step2 to add system level software or add environment variables.
After users click the “Add system-level software“ button, they will see a new GUI popped up as above. By default, there are several commonly used utility software added in the box. Users can insert more in the box(separated by space) and click the “Confirm“ button. After the button is clicked, the GUI will become non-editable and the definition file on the right will be updated.
Note: Please at least keep “wget“ and “git“ since they will be used later.
Sometimes, you may also want to add environment variables. The button “Add environment variables“ is designed for this purpose.
The definition file will also be updated with the input environment variables after the “Confirmed” button is clicked.
Step3: Add Your Software
Users have different options here (1) Add software from Conda (2) Add software from other sources(URLs). Note: you can only pick one type per request. If you click “Add software from Conda“ and then click “Add software from other sources“ or vice versa, the GUI will be override.
Add Software from Conda
We encourage users to first try the Conda option, search and find out whether the software they want is available in the Conda repo.
After clicking the button “Add software from Conda“, users will be asked to input an arbitrary Conda Environment name.
If we input a name such as “myenvironment” and click “Confirm“ we will see:
There are something to note here. First on the left side, the GUI has changed. Currently we just have a Conda environment without any software installed. So that’s why the “Added Software“ section is None for the Conda environment “myenvironment“.
At the same time, on the right side the definition file is updated. The %environment section is added to configure miniconda3. In the %post section, Miniconda3 is installed and the Conda environment “myenvironment“(the name we input) is created. The %apprun section is used to activate this environment when we run the container.
Then we can start to type and search the software in the box. The results will be displayed when at least two letters are typed in the box. For example if we want to search “python“, we just type “python“. A dropdown of matching software will be displayed.
Users can click the item in the dropdown and they will see the info page of that software:
Users can then click the “Add Software to Env“ button to add the software to the current Conda environment. Both the GUI on the left and the definition file on the right will be updated. Users can also delete the software or the Conda Environment using the “Delete“ buttons. The software search box can be used to add multiple software as users want. Users can also create a second Conda environment by clicking the “Add software from Conda“ again and search/add software again if they need multiple conda environments.
Add Software from Other Sources
Even though the Conda repo has a lot of software, there still may exist cases where the software has to be installed from URLs.
Users can select Github or other Sources
These two options provide similar GUIs, and the only difference is that we run git clone for GitHub and wget url for other sources. Users are expected to input the dir where they want to install the software, the github url or the package url, and any instructions as if they have cloned the repo or downloaded the tarballs. Once the “Confirm“ button is clicked, the definition file on the right will be updated as well
Step4: Submit the request
If users have added all the software they need, they can go to step 4 to submit their requests. In this step, users can add their custom definition file by clicking “input/hide here“, copying and pasting the definition file in the box.(Note, anything in the box will override the existing definition file on the right side of the page)
The last step is to fill in the container name and hit the “Submit Request“ button.
View Status
After the requests have been submitted, users can view the status of the requests under their profile by clicking the “View My Requests“ button:
Then users will be redirected to a page where they can see all the requests they have submitted. At the top is the explanation of the status of a request followed by a table showing all the requests
In the screenshot above, the status of the request id 3 is in-progress which means it is being processed and the user just need to wait for it to finish. Users can also view their definition file or check the build log by clicking the “view file/log“ buttons.
After certain time, the build will finish and the table will be updated.
Users can click the link to download the container file or they can run “wget {link}” on LCC, MCC, ECC to download the file to the cluster. At the same time, the user will receive an email
Example1: Install python and numpy using Conda
# This definition file is generated by the UKY CCS Singularity Container Composer
# If you have questions, please submit a ticket in https://ukyrcd.atlassian.net/servicedesk/customer/portals
Bootstrap: docker
From: ubuntu:24.10
%environment
# Set environment for Miniconda
export PATH="/usr/local/Miniconda3/bin:/usr/bin:/usr/local/bin:/usr/bin:/bin:"
unset CONDA_DEFAULT_ENV
export ANACONDA_HOME=/usr/local/Miniconda3
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
export SHELL=/bin/bash
%post
apt-get -y update
apt-get install -y git wget bash vim nano make cmake unzip
apt-get clean
# Install Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -P /tmp
sh /tmp/Miniconda3-latest-Linux-x86_64.sh -b -p /usr/local/Miniconda3
export PATH=/usr/local/Miniconda3/bin:/usr/bin:$PATH
conda update -y conda
# Install Conda environment test
conda create -n test -y conda-forge::python conda-forge::numpy
%apprun test
exec /bin/bash -c 'source /usr/local/Miniconda3/bin/activate test && echo "Conda activated!" && "$@"' -- "$@"
Once the container is built, users can run the container using the code below:
#singularity run --app {conda_env_name} mycontainer.sif {Your_Command}
singularity run --app myenvironment mycontainer.sif python -c "import numpy as np; arr = np.array([1, 2, 3, 4, 5]); print('NumPy Array:', arr)"
Till here we have gone through the features of the Singularity Container Composer hosted at https://containercomposer.ccs.uky.edu/. Feel free to submit a ticket at https://ukyrcd.atlassian.net/servicedesk/customer/portals if you have any questions!
Related content
Center for Computational Sciences