Difference between revisions of "Using the PBS / Torque queueing environment"

From Centre for Bioinformatics and Computational Biology
Jump to: navigation, search
 
(66 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
PBS Pro commands
+
The main commands for interacting with the Torque environment are:
−
qstat View queued jobs.
+
−
qsub Submit a job to the scheduler.
+
−
qdel Delete one of your jobs from queue.
+
−
Job script parameters
+
  
−
Parameters for any job submission are specified as #PBS comments in the job script file or as options to the qsub command. The essential options for the CHPC cluster include:
+
<source lang="c">
 +
> qstat
 +
</source>
 +
View queued jobs.
  
−
-l select=10:ncpus=24:mpiprocs=24
 
  
−
sets the size of the job in number of processors:
+
<source lang="c">
−
select=N number of nodes needed.
+
> qsub
−
ncpus=N number of cores per node
+
</source>
−
mpiprocs=N number of MPI ranks (processes) per node
+
Submit a job to the scheduler.
  
−
-l walltime=4:00:00
 
  
−
sets the total expected wall clock time in hours:minutes:seconds. Note the wall clock limits for each queue.
+
<source lang="c">
 +
> qdel
 +
</source>
 +
Delete one of your jobs from queue.
  
−
The job size and wall clock time must be within the limits imposed on the queue used:
 
  
−
-q normal
+
=== Job script parameters ===
  
−
to specify the queue.
+
Parameters for any job submission are specified as #PBS comments in the job script file or as options to the qsub command. The essential options for the cluster include:
  
−
Each job will draw from the allocation of cpu-hours granted to your Research Programme:
+
<source lang="c">
 +
#PBS -l nodes=1:ppn=14
 +
</source>
  
−
-P PRJT1234
+
sets the size of the job in number of processors:
  
−
specifies the project identifier short name, which is needed to identify the Research Programme allocation you will draw from for this job. Ask your PI for the project short name and replace PRJT1234 with it.
+
nodes=N sets the number of nodes needed.
−
Restricted queues
+
  
−
The large and bigmem queues are restricted to users who have need for them. If you are granted access to these queues then you should specify that you are a member of the largeq or bigmemq groups. For example:
+
ppn=N sets the number of cores per node.
  
−
#PBS -q large
+
<source lang="c">
−
#PBS -W group_list=largeq
+
#PBS -l walltime=8:00:00
 +
</source>
  
−
or
+
sets the total expected wall clock time in hours:minutes:seconds. Note the wall clock limits for each queue.
  
−
#PBS -q bigmem
 
−
#PBS -W group_list=bigmemq
 
  
−
Example job scripts
+
=== Example job scripts ===
−
An MPI program using 240 cores
+
  
−
Using the normal queue to run WRF:
+
A program using 14 cores on a single node:
  
 +
<source lang="c">
 
#!/bin/bash
 
#!/bin/bash
−
#PBS -l select=10:ncpus=24:mpiprocs=24:nodetype=haswell_reg
+
#PBS -l nodes=1:ppn=14
−
#PBS -P PRJT1234
+
#PBS -l walltime=8:00:00
 
#PBS -q normal
 
#PBS -q normal
−
#PBS -l walltime=4:00:00
+
#PBS -o /path/to/stdout (do not specify a filename here, just the path)
−
#PBS -o /mnt/lustre/users/USERNAME/WRF_Tests/WRFV3/run2km_100/wrf.out
+
#PBS -e /path/to/stderr (ditto)
−
#PBS -e /mnt/lustre/users/USERNAME/WRF_Tests/WRFV3/run2km_100/wrf.err
+
#PBS -k oe
−
#PBS -m abe
+
#PBS -m ae
 
#PBS -M your.email@address
 
#PBS -M your.email@address
−
ulimit -s unlimited
 
−
. /apps/chpc/earth/WRF-3.7-impi/setWRF
 
−
cd /mnt/lustre/users/USERNAME/WRF_Tests/WRFV3/run2km_100
 
−
rm wrfout* rsl*
 
−
nproc=`cat $PBS_NODEFILE | wc -l`
 
−
echo nproc is $nproc
 
−
cat $PBS_NODEFILE
 
−
time mpirun -np $nproc wrf.exe > runWRF.out
 
−
 
−
Assuming the above job script is saved as the text file example.job, the command to submit it to the PBSPro scheduler is:
 
−
 
−
qsub example.job
 
−
 
−
No additional parameters are needed for the qsub command since all the PBS parameters are specified within the job script file.
 
−
IMPORTANT
 
−
 
−
Note that in the above job script example the working directory is on the Lustre file system. Do not use your home directory for the working directory of your job. Use the directory allocated to you on the fast Lustre parallel file system:
 
  
−
/mnt/lustre/users/USERNAME/
+
module load bowtie2-2.3.4.1
 +
bowtie2 -x /path/to/genome -p 14 -1 /path/to/forwardreads.fastq -2 /path/to/reversereads.fastq -S /path/to/outputfile.sam
 +
</source>
  
−
where USERNAME is replace by your user name on the CHPC cluster.
 
  
−
Always provide the full absolute path to your Lustre sub-directories. Do not rely on a symbolic link from your home directory.
+
Assuming the above job script is saved as the text file run_bowtie.sh, the command to submit it to the Torque scheduler is:
−
Hybrid MPI/OpenMP
+
  
−
For example, to request an MPI job on one node with 12 cores per MPI rank, so that each MPI process can launch 12 OpenMP threads, change the -l parameters:
+
<source lang="c">
 +
> qsub run_bowtie.sh
 +
</source>
  
−
#PBS -l select=1:ncpus=24:mpiprocs=2:nodetype=haswell_reg
+
If you receive an email with exit status "0", that would usually indicate that the job completed successfully.
  
−
There are two MPI ranks, so mpirun -n 2 … .
+
=== Interactive jobs ===
−
Example interactive job request
+
  
−
To request an interactive session on a single node, the full command for qsub is:
+
* If you need an interactive terminal session on one of the cluster nodes (e.g. to compile code, setup jobs, test jobs), you can do this by using the qsub interactive mode, for example:
  
−
qsub -I -P PROJ0101 -q smp -l select=1:ncpus=24:mpiprocs=24:nodetype=haswell_reg
+
<source lang="c">
 +
> qsub -I -q interactive
 +
> This will give you a session on one of the cluster nodes (not bigmem), with a default walltime of 1 hour, 1 CPU,
 +
> and 16GB of memory. You can specify walltime up to 8 hours, memory up to 32 GB.
 +
> Please note the interactive mode is only for editing and testing scripts and code, not for running jobs. This should be done
 +
> with regular job submissions to the queue.
 +
</source>
  
−
Note:
+
=== The different queues available ===
  
−
    -I selects an interactive job
 
−
    you still must specify your project
 
−
    the queue must be smp, serial or test
 
−
    interactive jobs only get one node: select=1
 
−
    for the smp queue you can request several cores: ncpus=24
 
−
    you can run MPI code: indicate how many ranks you want with mpiprocs=
 
  
−
If you find your interactive session timing out too soon then add -l walltime=4:0:0 to the above command line to request the maximum 4 hours.
+
{| class="wikitable"
 +
!Queue Name
 +
!Max user jobs running
 +
!Max user cores <br /> running per job
 +
!Max memory
 +
!Max walltime
 +
!Description
 +
|-
 +
|short
 +
|112
 +
|28
 +
|128 GB
 +
|00:30:00
 +
|Short queue with 30 minute time limit
 +
|-
 +
|normal
 +
|8
 +
|28
 +
|128 GB
 +
|30:00:00
 +
|Medium queue with 30 hour time limit
 +
|-
 +
|long
 +
|6
 +
|28
 +
|128 GB
 +
|900:00:00
 +
|Long queue with 37.5 day time limit
 +
|-
 +
|bigmem
 +
|3
 +
|24
 +
|750 GB
 +
|720:00:00
 +
|High memory queue with 30 day time limit
 +
|-
 +
|mpi
 +
|1
 +
|112
 +
|128 GB
 +
|72:00:00
 +
|Queue for mpi parallel jobs with 3 day time limit
 +
|-
 +
|interactive
 +
|4
 +
|2 (default 1)
 +
|32 (default 16 GB)
 +
|8:00:00 (default 1 hr)
 +
|Interactive queue
 +
|}
  
 +
* PLEASE NOTE: The number of cores is sometimes referred to as number of threads in your software application. You should always set max number of threads/cores to the limits specified here (per queue).
 +
* If you need to run MPI jobs, please advise the system administrator so that the necessary security access can be enabled for your login.
 +
** Both MPICH and openMPI are installed. Please select the relevant environment using the "module load" functionality.
 +
** The node list for MPI can be accessed as $PBS_NODEFILE.
  
 +
=== Additional information ===
  
−
Much of this content has been adapted from the [http://wiki.chpc.ac.za/quick:start Quick Start Guide] at CHPC.
+
* A useful Torque tutorial can be found at [https://kb.iu.edu/d/avmy this link].
 +
* Detailed reference documentation for queue submission is available [http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/commands/qsub.htm here].

Latest revision as of 00:07, 14 June 2021

The main commands for interacting with the Torque environment are: 

> qstat

View queued jobs. 


> qsub

Submit a job to the scheduler. 


> qdel

Delete one of your jobs from queue.


Job script parameters

Parameters for any job submission are specified as #PBS comments in the job script file or as options to the qsub command. The essential options for the cluster include: 

#PBS -l nodes=1:ppn=14

sets the size of the job in number of processors:

nodes=N sets the number of nodes needed.

ppn=N sets the number of cores per node. 

#PBS -l walltime=8:00:00

sets the total expected wall clock time in hours:minutes:seconds. Note the wall clock limits for each queue.


Example job scripts

A program using 14 cores on a single node: 

#!/bin/bash
#PBS -l nodes=1:ppn=14
#PBS -l walltime=8:00:00
#PBS -q normal
#PBS -o /path/to/stdout (do not specify a filename here, just the path)
#PBS -e /path/to/stderr (ditto)
#PBS -k oe
#PBS -m ae
#PBS -M your.email@address
 
module load bowtie2-2.3.4.1
bowtie2 -x /path/to/genome -p 14 -1 /path/to/forwardreads.fastq -2 /path/to/reversereads.fastq -S /path/to/outputfile.sam


Assuming the above job script is saved as the text file run_bowtie.sh, the command to submit it to the Torque scheduler is: 

> qsub run_bowtie.sh

If you receive an email with exit status "0", that would usually indicate that the job completed successfully.

Interactive jobs

  • If you need an interactive terminal session on one of the cluster nodes (e.g. to compile code, setup jobs, test jobs), you can do this by using the qsub interactive mode, for example: 
> qsub -I -q interactive
> This will give you a session on one of the cluster nodes (not bigmem), with a default walltime of 1 hour, 1 CPU,
> and 16GB of memory. You can specify walltime up to 8 hours, memory up to 32 GB.
> Please note the interactive mode is only for editing and testing scripts and code, not for running jobs. This should be done
> with regular job submissions to the queue.

The different queues available

Queue Name Max user jobs running Max user cores
running per job 		Max memory Max walltime Description
short 112 28 128 GB 00:30:00 Short queue with 30 minute time limit
normal 8 28 128 GB 30:00:00 Medium queue with 30 hour time limit
long 6 28 128 GB 900:00:00 Long queue with 37.5 day time limit
bigmem 3 24 750 GB 720:00:00 High memory queue with 30 day time limit
mpi 1 112 128 GB 72:00:00 Queue for mpi parallel jobs with 3 day time limit
interactive 4 2 (default 1) 32 (default 16 GB) 8:00:00 (default 1 hr) Interactive queue
  • PLEASE NOTE: The number of cores is sometimes referred to as number of threads in your software application. You should always set max number of threads/cores to the limits specified here (per queue).
  • If you need to run MPI jobs, please advise the system administrator so that the necessary security access can be enabled for your login.
    • Both MPICH and openMPI are installed. Please select the relevant environment using the "module load" functionality.
    • The node list for MPI can be accessed as $PBS_NODEFILE.

Additional information

  • A useful Torque tutorial can be found at this link.
  • Detailed reference documentation for queue submission is available here.
Retrieved from "http://wiki.bi.up.ac.za/wiki/index.php?title=Using_the_PBS_/_Torque_queueing_environment&oldid=337"