Skip to content
Snippets Groups Projects
Select Git revision
  • master default
  • reorganisation
  • march2019
  • swc-hpc
  • mar2019
  • may2018
  • post-may2018-updates
  • june2018
  • oct2017 protected
9 results
Compare
Name Last commit Last update
cluster_setup
exercises
LICENSE.md
README.md
cluster_training2018.odp
cluster_training2018.pdf
hpc_slides.pdf
README.md

Creative Commons Licence
This work is licensed under a Creative Commons Attribution 4.0 International License.

Intro

Teaching material used during the High Performance Computing session of the EMBL Software Carpentry course.

Commands

Below are most of the commands used during the practical, so they can be copy/pasted, but I highly recommend typing along if you can.

Login to the frontend node

ssh <username>@login.cluster.embl.de

Clone this git repository

git clone https://git.embl.de/msmith/embl_hpc.git

Identifying our computer

hostname

Our first SLURM job

srun hostname

Exploring our example program (don't run!)

cd $HOME/embl_hpc/exercises
./hpc_example -t 10 -m 100

Running example program on on the cluster

srun ./hpc_example -t 10 -m 100

Using our reserved training space

srun --­­reservation=training ./hpc_example -­t 10 -­m 100

Running in the background

sbatch ­­--reservation=training ./batch_job.sh

Redirecting output

sbatch --output=output.txt --reservation=training ./batch_job.sh

Creating a larger list

You will need to edit batch_jobs.sh to take arguments

sbatch --output=output.txt --reservation=training ./batch_job.sh 20 ???

Displaying details of our cluster queue

scontrol show partition

Requesting more resources

sbatch --mem=8200 --reservation=training ./batch_job.sh 30 8000

Requesting a lot more resources

sbatch --mem=100G --reservation=training ./batch_job.sh 30 5000

Cancel jobs

scancel <jobID>
scancel -u <username>

Defining time limits

sbatch ­­--time=00­00:00:30  \
    ­­--reservation=training \
    batch_job.sh 60 500

Job efficiency statistics

seff <jobID>

Emailing output

sbatch ­­--mail­user=<first.last>@embl.de \
    ­­--reservation=training \
    ./batch_job.sh 20 500

Finding and using software

module avail
module spider samtools
module load BWA

BWA example

nano bwa/bwa_batch.sh
sbatch ­­­­--reservation=training bwa/bwa_batch.sh

Running interactive jobs

srun --pty bash

Interactive job with more memory

srun --mem=250 --pty bash

Using sbatch instead

sbatch batch_jobs.sh

Using job dependencies to build pipelines

jid=$(sbatch --parsable batch_job.sh)

sbatch --dependency=afterok:$jid batch_job.sh