Creating a Custom Conda Environment on HPC

There are thousands of different applications installed onto our clusters at any given time – the trick is to know where they are! We maintain lists of applications installed into containers for MCC and LCC, and also have many available via modules – however there may be some that you require for your work that are not installed globally. The purpose of this document is to provide steps to install software into a personal conda environment to support your work.

Step 1: Load the MiniForge module

MiniForge is a minimal installer for conda that allows you to manage environments with packages from conda-forge. To get started, load the Miniforge3 module:

LCC & MCC:

Load MiniForge module and activate base environment

You are now in the (base) environment.

Step 2: Create a Custom Conda Environment

Now that MiniForge is loaded, you can create a custom environment using a YAML file or install packages manually.

In order to create a custom environment, you must direct the environment to be installed into a location where you have write/read/execute permissions. This is one of HOME, SCRATCH, PROJECT, or PSCRATCH – we recommend that you create a sub-directory in the PROJECT folder with your linkblue-id and install there.

On LCC:

Check PROJECT directory location

to determine where you project space is located, then:

Create personal directory in PROJECT space (gpfs2_4m)

or:

Create personal directory in PROJECT space (gpfs2_16m)

On MCC:

Create personal directory in PROJECT space on MCC

2.1 (Option 1) Create an environment with YAML file.

If you have a pre-defined YAML file, you can create the environment directly from it.

1. Download the YAML file (here we use an example file for the github package xlstm, please change the URL to your specific file):

   Download YAML environment file

   wget https://github.com/NX-AI/xlstm/raw/refs/heads/main/environment_pt220cu121.yaml

2. Create the environment from the YAML file:

   Create conda environment from YAML file

   conda env create --prefix $PERSONAL_PROJECT/xlstm -f environment_pt220cu121.yaml

3. Activate the newly created environment:

   Activate the xlstm conda environment

   conda activate $PERSONAL_PROJECT/xlstm

This will activate the environment named xlstm, and it will be ready for use.

2.2 (Option 2) Create an environment manually.

If you do not have a YAML file, you can create an environment and install packages manually.

1. Create a new environment. The --prefix flag allows you to provide the path where the conda environment should be stored:

   Create new conda environment with custom prefix

   conda create --prefix $PERSONAL_PROJECT/MyEnvName

2. Activate the environment:

   Activate custom conda environment

   conda activate $PERSONAL_PROJECT/MyEnvName

3. Install packages as needed, for example:

   Install example packages using conda

   conda install numpy pandas scikit-learn

Step 3: Verifying the Environment

After the environment is created and activated, you can verify that it works by running Python and checking the installed packages

1. Run Python:

   Start Python interpreter

   python

2. Check installed packages:

   Verify package installation and versions

   import numpy as np import pandas as pd print(np.__version__) print(pd.__version__)

If there are no errors, your environment is successfully set up.

Step 4: Installing Additional Packages (Optional)

You can always add more packages to your environment later. To install additional packages:

1. Activate the environment:

   Load module and activate environment in new session

   module load ccs/Miniforge3 source activate $PERSONAL_PROJECT/MyEnvName

2. Install packages via conda or pip:

   Install additional packages with conda

   conda install matplotlib seaborn

   or using pip:

   Install packages using pip

   pip install some_package

Step 5: Deactivating and Managing Environments

1. To deactivate your environment:

   Deactivate conda environment and unload module

   conda deactivate module unload ccs/Miniforge3