Complex submissions

In Submitting your job using Slurm we looked at the following script;

   
#!/bin/bash --login

# Specify the queue (also known as a partition)
#SBATCH --partition=amd

# run a single task, using a single CPU core
#SBATCH --ntasks=1

#SBATCH --output=myScript.o%J

#run a program command to print hostname and uptime
/bin/hostname && /bin/uptime
    

As mentioned in Submitting your job using Slurm, unless the HPC is quiet, you may find it difficult to get this job to run or that you need more resources. This section discusses how to request resources using limits and gives some examples of some scripts.

Slurm and Limits

Slurm does not know what you're attempting to do until you tell it. There are three pieces of information that the scheduler needs to best load balance your job in the queue. It needs to know how much memory you need, how many CPU cores (also known as slots) and how long the job is expected to run for. This information is passed to the scheduler from your job script. If you fail to specify the number of cpu cores, memory or time required, it will use the scheduler defaults, which may not be the best for your task.

Requesting CPU Cores

In the example above we only requested one task, which Slurm will map to using one core. We can request more cores by increasing this number, however there is no guarantee that these will be on the same node (and for many tasks we don't care if the cores are on the same node as each other). Just increasing the number of tasks Slurm allocates doesn't make multiple copies of our job run. To run multiple copies we have to use the srun command to launch our program and tell that how many copies we'd like.

   
#!/bin/bash --login

# Specify the queue (also known as a partition)
#SBATCH --partition=amd

# run a single task, using a single CPU core
#SBATCH --ntasks=8

#SBATCH --output=myScript.o%J

#run 8 copies of the program to print hostname and uptime
srun --ntasks=8 "/bin/hostname && /bin/uptime"
    

This will submit your job and require 8 cores on any number of nodes (in the same partition) to run. At this point, if you submit this job, it will wait until 8 cores become available.

Requesting all jobs run on a single node

Adding the --nodes=1 parameter to your script will force all tasks to run on the same node. Note that the number of tasks requested must be less than or equal to the number of CPU cores available on a node.

Requesting Memory

Requesting memory is achieved by using the --mem option in your script.

You can specify the memory limits like this:

   
#SBATCH --mem=40G
    

This will request 40G of RAM is allocated on each node the job runs on. If the job exceeds this memory usage then it will be killed by Slurm. For information about the specification of the nodes available, please see Bert and Ernie - An Overview.

If you submit a job requesting more memory (mem_free) than is available on a single node in the queue then your job will fail to run. e.g. If you submit a job requiring 512GB RAM to the intel queue, it will fail to run as the maximum amount of memory a node has in the intel queue is 192GB.

Requesting Time

You can limit how long a job is allowed to take. When this time limit is exceeded Slurm will stop the job.

Requesting time limits is achieved by using the --time option in your script. The format options are either minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes or days-hours:minutes:seconds.

for example to request a 24 hour time limit you could either use:

   
#SBATCH --time=24:00:00
    

or

   
#SBATCH --time=1-00
    

An example script

Here is a more realistic script that you may wish to run.

 
#!/bin/bash --login

#specify which queue you wish to use
#SBATCH --partition=amd

#the job name as it appears in the queue
#SBATCH --job-name=blast

#output and error files
#SBATCH --output=blast.o%J
#SBATCH --error=blast.e%J


#specify the number of tasks you require
#SBATCH --ntasks=32

#request all tasks run on the same node
#SBATCH --nodes=1

#specify how long the job will take
#SBATCH --time=720:00:00

#SBATCH --mem=100G

#Load the blast module and then run the blast
module load BLAST/blast-2.2.26
blastall -p blastn -d /ibers/repository/public/blast_db/blast_june/nt -i myfile.fasta -o myfile.blast -a $SLURM_NTASKS -m 7
  

This script uses the amd queue, which we know has four nodes with 64 CPU cores and 256GB RAM and three nodes with 32 CPU cores and 98GB RAM. The run time of this, since it's a large sequence that is to be run, is set to 720 hours, or 30 days. I have requested 100GB for this job and 32 cores. This means that of the 7 AMD nodes available, only four of them are actually able to run it.

To run this script:

 
sbatch myscript.slurm
  

Finally, you may notice that the blast command uses the -m flag to specify the number of CPU cores required. The $SLURM_TASKS variable is set by specifying the number of tasks you require. Using this notation you only need to specify the number of CPU cores needed in one place.