Skip to content
Snippets Groups Projects
README.md 911 B
Newer Older
Thomas Schwarzl's avatar
Thomas Schwarzl committed
# Failed slurm jobs + Snakemake

Thomas Schwarzl's avatar
Thomas Schwarzl committed
This setup allows snakemake to detect failed jobs on the cluster. 
Snakemake will track the status of the job and then respond accordingly. 

Previously when cluster jobs failed (time limit exceeded, memory exceeded, failed 
otherwise), snakemake was waiting for the output file indefinitely. 


## Test run
Thomas Schwarzl's avatar
Thomas Schwarzl committed

# test if it is working
Thomas Schwarzl's avatar
Thomas Schwarzl committed

You can use this as a positive control. 
```
Thomas Schwarzl's avatar
Thomas Schwarzl committed
sh submit_slurm.sh all
Thomas Schwarzl's avatar
Thomas Schwarzl committed
```

This will create A.txt

Thomas Schwarzl's avatar
Thomas Schwarzl committed

# time outs
Thomas Schwarzl's avatar
Thomas Schwarzl committed
In this test run, the job will time out after a minute or two

```
sh submit_slurm.sh timeout
```

This will create B.txt

# clean up

```
snakemake clean
```

This will clean up A.txt and B.txt
Thomas Schwarzl's avatar
Thomas Schwarzl committed


## How does it work?

Snakemake will use the status.py script to check the cluster job instead of 
just waiting for the output file. 

https://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html#using-cluster-status