Creating a Custom Conda Environment on HPC

Owned by UKY CCS RCD
Nov 09, 2024
2 min read

There are thousands of different applications installed onto our clusters at any given time – the trick is to know where they are! We maintain lists of applications installed into containers for MCC and LCC, and also have many available via modules – however there may be some that you require for your work that are not installed globally. The purpose of this document is to provide steps to install software into a personal conda environment to support your work.

Step 1: Load the MiniForge module

MiniForge is a minimal installer for conda that allows you to manage environments with packages from conda-forge. To get started, load the Miniforge3 module:

LCC & MCC:

module load ccs/Miniforge3 source activate

You are now in the (base) environment.

Step 2: Create a Custom Conda Environment

Now that MiniForge is loaded, you can create a custom environment using a YAML file or install packages manually.

In order to create a custom environment, you must direct the environment to be installed into a location where you have write/read/execute permissions. This is one of HOME, SCRATCH, PROJECT, or PSCRATCH – we recommend that you create a sub-directory in the PROJECT folder with your linkblue-id and install there.

On LCC:

echo $PROJECT

to determine where you project space is located, then:

mkdir /mnt/gpfs2_4m/project/PIsLinkblue_uksr/YourLinkblue

or:

On MCC:

For future steps, we would like to save an environment variable called $PERSONAL_PROJECT

2.1 Using a YAML File to Create an Environment

If you have a pre-defined YAML file, you can create the environment directly from it.

  1. Download the YAML file (here we use an example file for the github package xlstm, please change the URL to your specific file):

  2. Create the environment from the YAML file:

  3. Activate the newly created environment:

This will activate the environment named xlstm, and it will be ready for use.

2.2 Installing Packages Manually

If you do not have a YAML file, you can create an environment and install packages manually.

  1. Create a new environment. The --prefix flag allows you to provide the path where the conda environment should be stored:

  2. Activate the environment:

  3. Install packages as needed, for example:

Step 3: Verifying the Environment

After the environment is created and activated, you can verify that it works by running Python and checking the installed packages

  1. Run Python:

  2. Check installed packages:

If there are no errors, your environment is successfully set up.

Step 4: Installing Additional Packages (Optional)

You can always add more packages to your environment later. To install additional packages:

  1. Activate the environment:

At a later session, $PERSONAL_PROJECT may not be defined – the above export command set it for a single session. To set this variable for all future sessions, add it to your .bashrc file.

  1. Install packages via conda or pip:

    or using pip:

Step 5: Deactivating and Managing Environments

  1. To deactivate your environment:

Center for Computational Sciences