Creating a Custom Conda Environment on HPC

There are thousands of different applications installed onto our clusters at any given time – the trick is to know where they are! We maintain lists of applications installed into containers for MCC and LCC, and also have many available via modules – however there may be some that you require for your work that are not installed globally. The purpose of this document is to provide steps to install software into a personal conda environment to support your work.

Step 1: Load the MiniForge module

MiniForge is a minimal installer for conda that allows you to manage environments with packages from conda-forge. To get started, load the Miniforge3 module:

LCC & MCC:

You are now in the (base) environment.

Step 2: Create a Custom Conda Environment

Now that MiniForge is loaded, you can create a custom environment using a YAML file or install packages manually.

In order to create a custom environment, you must direct the environment to be installed into a location where you have write/read/execute permissions. This is one of HOME, SCRATCH, PROJECT, or PSCRATCH – we recommend that you create a sub-directory in the PROJECT folder with your linkblue-id and install there.

On LCC:

to determine where you project space is located, then:

or:

On MCC:

2.1 (Option 1) Create an environment with YAML file.

If you have a pre-defined YAML file, you can create the environment directly from it.

1. Download the YAML file (here we use an example file for the github package xlstm, please change the URL to your specific file):

   wget https://github.com/NX-AI/xlstm/raw/refs/heads/main/environment_pt220cu121.yaml

2. Create the environment from the YAML file:

   conda env create --prefix $PERSONAL_PROJECT/xlstm -f environment_pt220cu121.yaml

3. Activate the newly created environment:

   conda activate $PERSONAL_PROJECT/xlstm

This will activate the environment named xlstm, and it will be ready for use.

2.2 (Option 2) Create an environment manually.

If you do not have a YAML file, you can create an environment and install packages manually.

1. Create a new environment. The --prefix flag allows you to provide the path where the conda environment should be stored:

   conda create --prefix $PERSONAL_PROJECT/MyEnvName

2. Activate the environment:

   conda activate $PERSONAL_PROJECT/MyEnvName

3. Install packages as needed, for example:

   conda install numpy pandas scikit-learn

Step 3: Verifying the Environment

After the environment is created and activated, you can verify that it works by running Python and checking the installed packages

1. Run Python:

   python

2. Check installed packages:

   import numpy as np import pandas as pd print(np.__version__) print(pd.__version__)

If there are no errors, your environment is successfully set up.

Step 4: Installing Additional Packages (Optional)

You can always add more packages to your environment later. To install additional packages:

1. Activate the environment:

   module load ccs/Miniforge3 source activate $PERSONAL_PROJECT/MyEnvName

2. Install packages via conda or pip:

   conda install matplotlib seaborn

   or using pip:

   pip install some_package

Step 5: Deactivating and Managing Environments

1. To deactivate your environment:

   conda deactivate module unload ccs/Miniforge3