Info: Warning: Please note that there is no internet access from any nodes inside the DGX cluster for end users. As such, all work needs to be manually copied by the user onto and off of the cluster using scp, rsync, or sftp methods.

Say that you have some scientific application that you want to run in a terminal. In your desktop/local machine, you can just run it directly like this (if the application is named "my_app"):

Running an application directly on a local machine

Example command running an application on a local machine

$ my_app -x -y -z # Running my_app directly in local machine

In DGX, you are not allowed to directly run like this. Instead, you have to write a job script that you will submit to the system. So, you would create a job script and then use the program sbatch to submit it into a queue. The job script will include the line that will specify your application, and it will also include lines on top that will tell the job scheduler about your job. These lines will begin with "#SBATCH ...". So, instead of running your program like in the previous example, you would run it through sbatch like below:

Submitting a job script using sbatch

Submitting a job script to SLURM using sbatch

$ sbatch my_job_script.sh

Example script files are located in /cm/shared/examples/uky and /cm/share/examples/workload/slurm/jobscripts. Below is a short example. First, use a text editor (e.g. vim) to create the script file. Or, you can create a script file on your local machine and copy it to DGX:

Creating a job script file with vim

Creating a job script file using vim editor

$ vim ./first_job.sh

Example file content:

Example SLURM job script with SBATCH directives

Example SLURM job script with resource requirements and directives

#!/bin/bash
#SBATCH --time=00:15:00     		# Time limit for the job 
#SBATCH --job-name=my_test_job    	# Job name
#SBATCH --ntasks=1       			# Number of cores for the job. Same as sbatch -n 1
#SBATCH --partition=defq     		# Partition/queue to run the job in
#SBATCH -e slurm-%j.err  			# Error file for this job. 
#SBATCH -o slurm-%j.out  			# Output file for this job.
#SBATCH -A <your project account>  	# Project allocation account name

echo "Hello world. This is my first job"   # This is the program that will be executed. You will substitute this with your scientific program.

Then run this job using sbatch. Remember, you are not allowed to run computations on the login nodes (i.e. you can't just execute your program directly. You must use sbatch because running the program directly in a login node can bog down the login node and slow down other users. What happens after you so sbatch myjob.sh is that the system will run your job in a special set of machines called "compute nodes". After submitting the job, you will get an output saying so, including the job's id. Once submitted, you can safely log off the login node and the job will still be in the system:

Submitting a job and receiving job ID confirmation

Submitting a job and receiving the assigned job ID

$ sbatch ./test_job.sh
Submitted batch job 123027

When you do the above command, your job may have to be put on queue (remember that many other users are using the system) before it is actually executed. That is, it may take some time before your scientific program actually runs (remember that many users are using the system simultaneously).

Once your job is done, you can see the slurm job output files (slurm is DGX's automated job scheduler):

Listing SLURM output files after job completion

Listing files showing SLURM output and error files

$ ls .
slurm-123027.err  slurm-123027.out  test_job.sh

The job script we have should have printed out our Hello World message to the slurm output file:

Viewing job output from SLURM output file

Viewing the contents of the SLURM output file

$ cat slurm-123027.out 
Hello world. This is my first job.

After you submit your job, you can see some useful details about it by calling scontrol:

Checking job details with scontrol

Using scontrol to view detailed job information

$ scontrol show job 123027

Above, "JobState" tells us the job is currently running

To see more examples, look at the script files under /cm/shared/examples/uky and /cm/share/examples/workload/slurm/jobscripts.

Queues/partitions

Each job needs to run in a queue or partition, or a set of nodes that have a specific set of resource limits (e.g. the number of Partition/queue information can be found by doing sinfo and scontrol show <partition name>:

sinfo will show all partitions, the time limit for each partition, and the state for a set of nodes in the partition. A queue/partition may have some nodes that are allocated (jobs are being run on them) while other nodes may be idle and ready to have jobs run on them. This is why, for a given queue/partition, you may see a row with a set of nodes that have the state of "alloc". or allocated nodes, while another row will show the same partition name with an "idle" state.

Example sinfo output showing partition states

Example sinfo output showing partition availability and node states

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite      1    mix dgx-01
defq*        up   infinite      4   idle dgx-[02-05]

A shortened output for DGX is shown above. Above, there is the current defq queue listed.

To see detailed information on a specific partition, do this:

Detailed partition information using scontrol

Using scontrol to view detailed partition configuration

$ scontrol show partition defq
PartitionName=defq
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=UNLIMITED DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   Nodes=dgx-[01-05]
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=1120 TotalNodes=5 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
   TRES=cpu=1120,mem=10319685M,node=5,billing=1120,gres/gpu=40

Setting time limits

It's important to set a time limit for a job. The time limit tells SLURM that your job will be killed after that specified time (the idea is that you would have an estimate on when the job would finish). The reason why you want to specify a time limit is that if you don't specify it, SLURM will put a default value which equals to the maximum time for that partition. This is bad because SLURM will assume your job will take the longest time possible for a given partition and SLURM will have to wait until enough resources are available to run your job. It's possible, then, that jobs by other users will be put ahead of yours in the queue if their time limit is much shorter than your job's. Thus, if you know that your program will finish in 3 hours, you can set the time limit to, say, 3.5 hours. If you have no idea how long a program runs, then you may omit the time limit the first time you run a job, or you can judiciously choose a long time.

Submitting a job and checking status

If you've named your job script as submit.sh, submit a job by doing this:

Submitting a job script and receiving job ID

Submitting a job script and receiving the job ID

$ sbatch run_fq2bam.sh
Submitted batch job 2465

You can remember the job number above to see the status of the job while it's waiting on queue or while it's running. After the job is finished, the information below will no longer be available:

Checking running job status with scontrol

Using scontrol to check running job status and details

$ scontrol show job 2465
JobId=2465 JobName=run_fq2bam.sh
   UserId=cdhick2(1003) GroupId=users(1003) MCS_label=N/A
   Priority=4294900011 Nice=0 Account=(null) QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:23 TimeLimit=UNLIMITED TimeMin=N/A
   SubmitTime=2024-03-14T14:46:19 EligibleTime=2024-03-14T14:46:19
   AccrueTime=2024-03-14T14:46:19
   StartTime=2024-03-14T14:46:19 EndTime=Unknown Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-03-14T14:46:19 Scheduler=Backfill
   Partition=defq AllocNode:Sid=slogin-01:3987877
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=dgx-01
   BatchHost=dgx-01
   NumNodes=1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=1,mem=2063937M,node=1,billing=1,gres/gpu=2
   AllocTRES=cpu=32,node=1,billing=32,gres/gpu=2
   Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=*
   MinCPUsNode=32 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/home/cdhick2/parabricks-test/run_fq2bam.sh
   WorkDir=/home/cdhick2/parabricks-test
   StdErr=/home/cdhick2/parabricks-test/slurm-2465.out
   StdIn=/dev/null
   StdOut=/home/cdhick2/parabricks-test/slurm-2465.out
   Power=
   CpusPerTres=gres:gpu:16
   TresPerTask=gres:gpu:2

If you have specified the mail options (see the first box), then SLURM will email you when the job is put on queue and when your job is done.

Common Slurm Commands

Command	Description

Command	Description
sbatch script_file	Submit SLURM job script
scancel job_id	Cancel job that has job_id
squeue -u user_id	Show jobs that are on queue for user_id
sinfo	Show partitions/queues, their time limits, number of nodes, and which compute nodes are running jobs or idle.

Slurm Job Script Options

Option	Short Version	Long Version	Example(s)	Explanation

Option	Short Version	Long Version	Example(s)	Explanation
Job name	#SBATCH –J jobname	#SBATCH --job-name=jobname	#SBATCH --job-name=my_first_job	The job will be custom-labeled with jobname (in addition to an integer id for the job automatically given by the program)
Partition/queue	#SBATCH -p partition_id	#SBATCH --partition=partition_id	#SBATCH -partition=normal # normal partition #SBATCH -partition=jumbo # jumbo partition	The job will be ran in compute node(s) that is/are in partition_id
Time limit	#SBATCH -t time_limit	#SBATCH --time=time_limit	#SBATCH --time=01:00:00 # one hour limit #SBATCH --time=2-00:00:00 # 2 day limit	The job will be killed if it reaches time_limit specified.
Memory (RAM)		#SBATCH --mem=memory_amount	#SBATCH --mem=32g # 32 GB ram asked	The job will use up to the specified memory_amount.
Project account	#SBATCH -A account	#SBATCH --account=account	#SBATCH --account=col_pi123_uksr	Run the job under this project account.
Standard error filename	#SBATCH -e filename	#SBATCH --error=filename	#SBATCH --error=slurm%A_@a.err # special variables used; will be substituted with job array number and job id number #SBATCH --error=prog_error.log # You can use any file name (no whitespaces)	Standard error of the job will be stored under filename
Standard output filename	#SBATCH -o filename	#SBATCH --output=filename	#SBATCH --output=slurm%A_@a.out #SBATCH --output=prog_output.log	Standard output of the job will be stored under filename

To view a more complete list of Slurm options, please view the sbatch input environmental variables page.

Submitting jobs on the DGX HPC cluster (for first-time users)