Nextflow with SLURM and Singularity

1. Introduction

NextFlow is a popular workflow manager designed to handle large-scale bioinformatics data processing. It provides a simple way to run and scale workflows, from a local machine to HPC clusters like LCC and MCC, which use the SLURM scheduler.

This guide explains how to configure and run NextFlow on the LCC and MCC clusters, with a focus on using SLURM, setting up the environment, creating a NextFlow configuration file, and submitting jobs.

2. Prerequisites

2.1. Software Requirements

Before running NextFlow on LCC or MCC, ensure the following software is loaded:

NextFlow: Load NextFlow using the module system:
- LCC:
  Loading NextFlow module on LCC
  module load ccs/nextflow
- MCC:
  Loading NextFlow module on MCC
  module load ccs/conda/nextflow/24.10.4
Singularity: (If using containers) Load the Singularity module:
- LCC:
  Loading Singularity module on LCC
  module load ccs/singularity
- MCC:
  (Loaded by default.)

2.2. SLURM on LCC/MCC

SLURM is used as the job scheduler on LCC and MCC. It helps manage and allocate computational resources to different tasks. You'll need basic SLURM commands such as sbatch, squeue, and scancel – these commands are available on all nodes without the need to load a module.

3. Configuring NextFlow on LCC and MCC

3.1. Creating a NextFlow Configuration File

Below is an example configuration file (nextflow.config) for SLURM, explicitly configured for LCC and MCC:

LCC:
NextFlow configuration for LCC with SLURM
process { executor = 'slurm' queue = 'SKY32M192_L' // One of the queues: CAL48M192_D, CAL48M192_L, CAC48M192_L, SKY32M192_D, SKY32M192_L, A2V80_ICE56M256_L, P4V12_SKY32M192_D, P4V12_SKY32M192_L, P4V16_HAS16M128_L, V4V16_SKY32M192_L, V4V32_CAS40M192_L, V4V32_SKY32M192_L cpus = 4 memory = '16 GB' // Use 4GB/CPU. time = '24h' // Account information (compute queues start with 'col', GPU queues start with 'gol') clusterOptions = '-A col_PILinkBlue_uksr' // Default account for compute (use gol for GPU jobs) }
MCC:
NextFlow configuration for MCC with SLURM
process { executor = 'slurm' queue = 'normal' // Use 'normal' for standard nodes, 'jumbo' for high-memory nodes. cpus = 4 memory = '16 GB' // Adjust memory requirements as needed. time = '24h' // Account information (compute queues start with 'coa') clusterOptions = '-A coa_PILinkBlue_uksr' }

Modify the above fields, such as queue, cpus, memory, and account, depending on your job type and available resources. For MCC, you might want to use the jumbo partition for high-memory jobs (500 GB+).

3.2. NextFlow Profiles

Nextflow profiles allow you to specify different job configurations for various types of jobs (e.g., small, medium, large) while keeping global parameters consistent. The global process block defines the default executor, queue, and account, while the profiles specify resource parameters like CPU, memory, and time. Each profile overrides the relevant parameters for the defined job type, making it easier to manage different workloads without changing the global configuration.

LCC:
NextFlow profiles configuration for LCC
process { executor = 'slurm' clusterOptions = '-A col_PILinkBlue_uksr' // Default account for compute (use gol for GPU jobs) } profiles { // Small CPU job profile small { process.time = '2h' process.memory = '4 GB' process.cpus = 1 process.queue = 'SKY32M192_L' // One of the CPU queues: CAL48M192_D, CAL48M192_L, CAC48M192_L, SKY32M192_D, SKY32M192_L } // Medium CPU job profile medium { process.time = '8h' process.memory = '16 GB' process.cpus = 4 process.queue = 'SKY32M192_L' // One of the CPU queues: CAL48M192_D, CAL48M192_L, CAC48M192_L, SKY32M192_D, SKY32M192_L } // Large CPU job profile large { process.time = '72h' process.memory = '128 GB' process.cpus = 32 process.queue = 'SKY32M192_L' // One of the CPU queues: CAL48M192_D, CAL48M192_L, CAC48M192_L, SKY32M192_D, SKY32M192_L } // GPU job profile gpu { process.time = '24h' process.memory = '16 GB' process.cpus = 4 process.queue = 'P4V12_SKY32M192_D' // One of the GPU queues: A2V80_ICE56M256_L, P4V12_SKY32M192_D, P4V12_SKY32M192_L, P4V16_HAS16M128_L, V4V16_SKY32M192_L, V4V32_CAS40M192_L, V4V32_SKY32M192_L process.clusterOptions = '-A gol_PILinkBlue_uksr --gres=gpu:1' // Request 1 GPU using SLURM's --gres flag and -A for account } }
MCC:
NextFlow profiles configuration for MCC
process { executor = 'slurm' queue = 'normal' // Default queue for standard nodes. clusterOptions = '-A coa_PILinkBlue_uksr' // Account information } profiles { // Small job profile small { process.time = '2h' process.memory = '4 GB' process.cpus = 1 } // Medium job profile medium { process.time = '8h' process.memory = '16 GB' process.cpus = 4 } // Large job profile large { process.time = '72h' process.memory = '256 GB' process.cpus = 128 } }

To run a pipeline with a specific profile, use the -profile flag:

Running NextFlow with a profile

nextflow run <pipeline_name> -profile small

3.3. Using Singularity with NextFlow

Singularity is a containerization platform that allows you to package applications and their dependencies into portable containers. It is especially useful in bioinformatics workflows, where reproducibility and environment consistency are critical.

3.3.a Configuring Singularity for NextFlow

If your pipeline requires containers, you can configure NextFlow to use Singularity. This section explains how to set it up on LCC and MCC.

LCC: To use Singularity on LCC, you need to load the Singularity module:

module load ccs/singularity

MCC: On MCC, Singularity is already loaded by default, so you don't need to load it manually.

3.3.b Modifying the NextFlow Configuration for Singularity

You can specify the container engine in the nextflow.config file under the process.container directive. Below is an example for configuring NextFlow to use Singularity containers:

LCC:

NextFlow configuration with Singularity for LCC

process {
    executor = 'slurm'
    clusterOptions = '-A col_PILinkBlue_uksr' // Default account for compute (use 'gol' for GPU jobs)
    container = 'singularity'  // Use Singularity as the container engine
    containerOptions = '-B /path/to/bind:/mnt'  // Mount paths into the container, as needed
}

profiles {
    // Small CPU job profile
    small {
        process.time = '2h'
        process.memory = '4 GB'
        process.cpus = 1
        process.queue = 'SKY32M192_L'  // One of the CPU queues: CAL48M192_D, CAL48M192_L, CAC48M192_L, SKY32M192_D, SKY32M192_L
    }
    // Add other profiles here
}
containerEngine = 'singularity'

singularity {
    enabled = true
    autoMounts = true
    runOptions = '-B /path/to/bind:/mnt'  // Optional bind path, change as needed
}

MCC:

NextFlow configuration with Singularity for MCC

process {
    executor = 'slurm'
    queue = 'normal'  // Default queue for standard nodes.
    clusterOptions = '-A coa_PILinkBlue_uksr'  // Account information
}

profiles {
    // Small job profile
    small {
        process.time = '2h'
        process.memory = '4 GB'
        process.cpus = 1
    }
    // Add other profiles here
}
containerEngine = 'singularity'

singularity {
    enabled = true
    autoMounts = true
    runOptions = '-B /path/to/bind:/mnt'  // Optional: only if needed
}

In this example:

The containerEngine directive tells NextFlow to use Singularity for the job.
The singularitydirective allows you to pass additional options to Singularity, such as mounting directories (-B option) inside the container.

4. Running a NextFlow Pipeline on LCC and MCC

4.1. Using Custom Configurations

To provide additional configuration options like reports:

Running NextFlow with reports and timeline

nextflow run <pipeline_name> -profile small -with-report report.html -with-timeline timeline.html

5. Monitoring and Managing NextFlow Jobs

Use SLURM commands to monitor jobs on LCC and MCC:

Check job status:
Checking SLURM job status
squeue -u <your_username>
Cancel a job:
Canceling a SLURM job
scancel <job_id>

NextFlow also has its own logging features:

View logs:
Viewing NextFlow logs
nextflow log

6. Troubleshooting

6.1. Memory/CPU Issues

If jobs fail due to resource limits, adjust your nextflow.config file. For instance, request more memory or CPUs if needed:

Adjusting memory and CPU resources

process.memory = '128 GB'
process.cpus = 32

6.2. SLURM-Specific Errors

SLURM errors like timeouts can often be resolved by modifying job directives in your config file. Check the SLURM logs for details.

7. Conclusion

NextFlow provides efficient and scalable ways to run bioinformatics workflows on LCC and MCC with SLURM. Customize the configuration to suit your needs and consult official documentation for further details.