This work is licensed under a Creative Commons Attribution 4.0 International License.
Intro
Teaching material used during the High Performance Computing session of the EMBL Software Carpentry course.
Commands
Below are most of the commands used during the practical, so they can be copy/pasted, but I highly recommend typing along if you can.
Login to the frontend node
ssh <username>@login.cluster.embl.de
Clone this git repository
git clone https://git.embl.de/msmith/embl_hpc.git
Identifying our computer
hostname
Our first SLURM job
srun hostname
Exploring our example program (don't run!)
cd $HOME/embl_hpc/exercises
./hpc_example -t 10 -m 100
Running example program on on the cluster
srun ./hpc_example -t 10 -m 100
Using our reserved training space
srun --reservation=training ./hpc_example -t 10 -m 100
Running in the background
sbatch --reservation=training ./batch_job.sh
Redirecting output
sbatch --output=output.txt --reservation=training ./batch_job.sh
Creating a larger list
You will need to edit batch_jobs.sh to take arguments
sbatch --output=output.txt --reservation=training ./batch_job.sh 20 ???
Displaying details of our cluster queue
scontrol show partition
Requesting more resources
sbatch --mem=8200 --reservation=training ./batch_job.sh 30 8000
Requesting a lot more resources
sbatch --mem=100G --reservation=training ./batch_job.sh 30 5000
Cancel jobs
scancel <jobID>
scancel -u <username>
Defining time limits
sbatch --time=0000:00:30 \
--reservation=training \
batch_job.sh 60 500
Job efficiency statistics
seff <jobID>
Emailing output
sbatch --mailuser=<first.last>@embl.de \
--reservation=training \
./batch_job.sh 20 500
Finding and using software
module avail
module spider samtools
module load BWA
BWA example
nano bwa/bwa_batch.sh
sbatch --reservation=training bwa/bwa_batch.sh
Running interactive jobs
srun --pty bash
Interactive job with more memory
srun --mem=250 --pty bash
sbatch
instead
Using sbatch batch_jobs.sh
Using job dependencies to build pipelines
jid=$(sbatch --parsable batch_job.sh)
sbatch --dependency=afterok:$jid batch_job.sh