Difference between revisions of "Using the PBS / Torque queueing environment"
(64 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
The main commands for interacting with the Torque environment are: | The main commands for interacting with the Torque environment are: | ||
− | |||
− | |||
− | |||
+ | <source lang="c"> | ||
+ | > qstat | ||
+ | </source> | ||
+ | View queued jobs. | ||
− | |||
− | |||
− | |||
<source lang="c"> | <source lang="c"> | ||
− | + | > qsub | |
</source> | </source> | ||
+ | Submit a job to the scheduler. | ||
− | + | <source lang="c"> | |
− | + | > qdel | |
− | + | </source> | |
− | + | Delete one of your jobs from queue. | |
− | |||
− | + | === Job script parameters === | |
− | + | Parameters for any job submission are specified as #PBS comments in the job script file or as options to the qsub command. The essential options for the cluster include: | |
− | + | <source lang="c"> | |
+ | #PBS -l nodes=1:ppn=14 | ||
+ | </source> | ||
− | + | sets the size of the job in number of processors: | |
− | + | nodes=N sets the number of nodes needed. | |
− | + | ppn=N sets the number of cores per node. | |
− | + | <source lang="c"> | |
− | + | #PBS -l walltime=8:00:00 | |
+ | </source> | ||
− | + | sets the total expected wall clock time in hours:minutes:seconds. Note the wall clock limits for each queue. | |
− | |||
− | |||
− | + | === Example job scripts === | |
− | + | A program using 14 cores on a single node: | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
+ | <source lang="c"> | ||
#!/bin/bash | #!/bin/bash | ||
− | #PBS -l | + | #PBS -l nodes=1:ppn=14 |
− | #PBS - | + | #PBS -l walltime=8:00:00 |
#PBS -q normal | #PBS -q normal | ||
− | + | #PBS -o /path/to/stdout (do not specify a filename here, just the path) | |
− | #PBS -o / | + | #PBS -e /path/to/stderr (ditto) |
− | #PBS -e / | + | #PBS -k oe |
− | #PBS -m | + | #PBS -m ae |
#PBS -M your.email@address | #PBS -M your.email@address | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | module load bowtie2-2.3.4.1 | |
+ | bowtie2 -x /path/to/genome -p 14 -1 /path/to/forwardreads.fastq -2 /path/to/reversereads.fastq -S /path/to/outputfile.sam | ||
+ | </source> | ||
− | |||
− | + | Assuming the above job script is saved as the text file run_bowtie.sh, the command to submit it to the Torque scheduler is: | |
− | + | ||
− | + | <source lang="c"> | |
+ | > qsub run_bowtie.sh | ||
+ | </source> | ||
− | + | If you receive an email with exit status "0", that would usually indicate that the job completed successfully. | |
− | + | === Interactive jobs === | |
− | + | * If you need an interactive terminal session on one of the cluster nodes (e.g. to compile code, setup jobs, test jobs), you can do this by using the qsub interactive mode, for example: | |
− | + | ||
− | + | <source lang="c"> | |
− | + | > qsub -I -q interactive | |
− | + | > This will give you a session on one of the cluster nodes (not bigmem), with a default walltime of 1 hour, 1 CPU, | |
− | + | > and 16GB of memory. You can specify walltime up to 8 hours, memory up to 32 GB. | |
− | + | > Please note the interactive mode is only for editing and testing scripts and code, not for running jobs. This should be done | |
− | + | > with regular job submissions to the queue. | |
− | + | </source> | |
− | + | ||
− | + | ||
− | + | ||
− | + | === The different queues available === | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | {| class="wikitable" | |
+ | !Queue Name | ||
+ | !Max user jobs running | ||
+ | !Max user cores <br /> running per job | ||
+ | !Max memory | ||
+ | !Max walltime | ||
+ | !Description | ||
+ | |- | ||
+ | |short | ||
+ | |112 | ||
+ | |28 | ||
+ | |128 GB | ||
+ | |00:30:00 | ||
+ | |Short queue with 30 minute time limit | ||
+ | |- | ||
+ | |normal | ||
+ | |8 | ||
+ | |28 | ||
+ | |128 GB | ||
+ | |30:00:00 | ||
+ | |Medium queue with 30 hour time limit | ||
+ | |- | ||
+ | |long | ||
+ | |6 | ||
+ | |28 | ||
+ | |128 GB | ||
+ | |900:00:00 | ||
+ | |Long queue with 37.5 day time limit | ||
+ | |- | ||
+ | |bigmem | ||
+ | |3 | ||
+ | |24 | ||
+ | |750 GB | ||
+ | |720:00:00 | ||
+ | |High memory queue with 30 day time limit | ||
+ | |- | ||
+ | |mpi | ||
+ | |1 | ||
+ | |112 | ||
+ | |128 GB | ||
+ | |72:00:00 | ||
+ | |Queue for mpi parallel jobs with 3 day time limit | ||
+ | |- | ||
+ | |interactive | ||
+ | |4 | ||
+ | |2 (default 1) | ||
+ | |32 (default 16 GB) | ||
+ | |8:00:00 (default 1 hr) | ||
+ | |Interactive queue | ||
+ | |} | ||
+ | * PLEASE NOTE: The number of cores is sometimes referred to as number of threads in your software application. You should always set max number of threads/cores to the limits specified here (per queue). | ||
+ | * If you need to run MPI jobs, please advise the system administrator so that the necessary security access can be enabled for your login. | ||
+ | ** Both MPICH and openMPI are installed. Please select the relevant environment using the "module load" functionality. | ||
+ | ** The node list for MPI can be accessed as $PBS_NODEFILE. | ||
+ | === Additional information === | ||
− | + | * A useful Torque tutorial can be found at [https://kb.iu.edu/d/avmy this link]. | |
+ | * Detailed reference documentation for queue submission is available [http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/commands/qsub.htm here]. |
Latest revision as of 00:07, 14 June 2021
The main commands for interacting with the Torque environment are:
> qstat
View queued jobs.
> qsub
Submit a job to the scheduler.
> qdel
Delete one of your jobs from queue.
Contents
Job script parameters
Parameters for any job submission are specified as #PBS comments in the job script file or as options to the qsub command. The essential options for the cluster include:
#PBS -l nodes=1:ppn=14
sets the size of the job in number of processors:
nodes=N sets the number of nodes needed.
ppn=N sets the number of cores per node.
#PBS -l walltime=8:00:00
sets the total expected wall clock time in hours:minutes:seconds. Note the wall clock limits for each queue.
Example job scripts
A program using 14 cores on a single node:
#!/bin/bash #PBS -l nodes=1:ppn=14 #PBS -l walltime=8:00:00 #PBS -q normal #PBS -o /path/to/stdout (do not specify a filename here, just the path) #PBS -e /path/to/stderr (ditto) #PBS -k oe #PBS -m ae #PBS -M your.email@address module load bowtie2-2.3.4.1 bowtie2 -x /path/to/genome -p 14 -1 /path/to/forwardreads.fastq -2 /path/to/reversereads.fastq -S /path/to/outputfile.sam
Assuming the above job script is saved as the text file run_bowtie.sh, the command to submit it to the Torque scheduler is:
> qsub run_bowtie.sh
If you receive an email with exit status "0", that would usually indicate that the job completed successfully.
Interactive jobs
- If you need an interactive terminal session on one of the cluster nodes (e.g. to compile code, setup jobs, test jobs), you can do this by using the qsub interactive mode, for example:
> qsub -I -q interactive > This will give you a session on one of the cluster nodes (not bigmem), with a default walltime of 1 hour, 1 CPU, > and 16GB of memory. You can specify walltime up to 8 hours, memory up to 32 GB. > Please note the interactive mode is only for editing and testing scripts and code, not for running jobs. This should be done > with regular job submissions to the queue.
The different queues available
Queue Name | Max user jobs running | Max user cores running per job |
Max memory | Max walltime | Description |
---|---|---|---|---|---|
short | 112 | 28 | 128 GB | 00:30:00 | Short queue with 30 minute time limit |
normal | 8 | 28 | 128 GB | 30:00:00 | Medium queue with 30 hour time limit |
long | 6 | 28 | 128 GB | 900:00:00 | Long queue with 37.5 day time limit |
bigmem | 3 | 24 | 750 GB | 720:00:00 | High memory queue with 30 day time limit |
mpi | 1 | 112 | 128 GB | 72:00:00 | Queue for mpi parallel jobs with 3 day time limit |
interactive | 4 | 2 (default 1) | 32 (default 16 GB) | 8:00:00 (default 1 hr) | Interactive queue |
- PLEASE NOTE: The number of cores is sometimes referred to as number of threads in your software application. You should always set max number of threads/cores to the limits specified here (per queue).
- If you need to run MPI jobs, please advise the system administrator so that the necessary security access can be enabled for your login.
- Both MPICH and openMPI are installed. Please select the relevant environment using the "module load" functionality.
- The node list for MPI can be accessed as $PBS_NODEFILE.