EMBL utilities
perms - a script to display permissions of a target file/folder up to the root of the filesystem (useful when granting permissions to someone else).
submitjob - a polyglot script that submits jobs to SGE, LSF and SLURM clusters.
q - shortcut to query state of own jobs across queue systems (SLURM, LSF and SGE)
docker_wrap.sh - a convenience script to start a container that shares the user's home and creates a user to ensure consistent permissions.
tabview and csvview - quick preview of tab and csv delimited files. Aligns columns based on content. Works fine with streams and large files. Uses xpipe. tabview converts tabs to ¬ (seldomly used character) to workaround https://github.com/karelzak/util-linux/issues/575
xpipe - like xargs but for splitting large or infinite streams into chunks and execute commands on them.
notify-monitor.py - monitor servers and notify when conditions are met. Monitors system load, memory usage and presence of processes. Notifies with a shell bell or email.
qs - collect queue status across different SGE clusters/queues. Requires key-based authentication with SSH.
mv_keep_parents - equivalent to cp --parents src/*.txt dest/ but moving instead of copying. Passing -k will keep empty directories in src/.
symlink_relative.py and symlink_absolute.py - convert symlinks to relative/absolute respectively.
Examples
submitjob
Save this into myscript.job and launch with submitjob myscript.job. If using SLURM you can use submit-slurm instead:
# Lines starting with % will be executed for every job in the array
# On SGE and using old modulefiles include the next line
% if [ -f /etc/profile.d/modules.sh ]; then source /etc/profile.d/modules.sh; fi
# On SLURM you can simply load the desired module
% module add my_anaconda
% source activate py3-pandas
# All the lines below will be jobs on the same job-array
# NOTE that if the cluster is big enough and enough slots were requested
# the entire array will be started roughly at the same time
# Run the same script 20 times (useful for simulations, ngless, jug...)
@20 python compute_scores.py
# Some more jobs to run as part of the same array
python compute_something_else.py
python validate.pyperms
Display permissions of path to reach the file, resolving any symlinks along the way
% perms /g/scb2/bork/mocat/software/R/3.4.1/bin/R
-r-xr-xr-x 3 mocat mocat 9468 Jan 1 1970 /g/bork3/home/mocat/nix/store/6549ws1c4y0ibj3qyfmd7d0kzyfzgzcd-R-3.4.1/bin/R
dr-xr-xr-x 2 mocat mocat 4096 Jan 1 1970 /g/bork3/home/mocat/nix/store/6549ws1c4y0ibj3qyfmd7d0kzyfzgzcd-R-3.4.1/bin
dr-xr-xr-x 6 mocat mocat 4096 Jan 1 1970 /g/bork3/home/mocat/nix/store/6549ws1c4y0ibj3qyfmd7d0kzyfzgzcd-R-3.4.1
drwxr-xr-x 6717 mocat mocat 3424256 Feb 14 19:22 /g/bork3/home/mocat/nix/store
drwxr-xr-x 7 mocat mocat 4096 Mar 25 2017 /g/bork3/home/mocat/nix
drwxr-xr-x 16 mocat mocat 4096 Feb 14 19:27 /g/bork3/home/mocat
drwxr-x--t 86 adm_bork bork 8192 Nov 3 10:51 /g/bork3/home
drwxr-xr-x 32 adm_bork bork 4096 Apr 4 2017 /g/bork3
drwxr-xr-x 10 root root 0 Feb 19 13:31 /g
drwxr-xr-x 36 root root 4096 Oct 12 15:40 /qs
Show status of different clusters:
% qs alpha beta sigma
Host CPU CPU-Free MEM MEM-Free Usage % Avail % Rank
alpha 64 57 940.4GiB 486.2GiB 28.62 50.70 2.99
beta 64 24 234.9GiB 0.0B 80.25 0.00 0.00
sigma 92 10 173.5GiB 90.8GiB 68.54 9.77 0.12
--------------- -------- -------- -------- -------- -------- -------- --------
GLOBAL 220 91 1.3TiB 577.0GiB 61.49 17.56mv_keep_parents
Moves (sub)directories keeping folder structure:
% mv_keep_parents CC/*/reads.filtered.263RefGeneCatalog.padded.solexaqa TO_REMOVEproduced:
% tree TO_REMOVE
TO_REMOVE
└── CC
├── CCHB10959853ST-11-0
│ └── reads.filtered.263RefGeneCatalog.padded.solexaqa
├── CCHB10961334ST-11-0
│ └── reads.filtered.263RefGeneCatalog.padded.solexaqa
...