script cleanup

- moved to scripts folder
- wrote README
- removed obsolete
- renamed properly

README.md

 # spongfold
 
-This repository documents the idea of Fabi and Niko to use predicted protein structures and structure similarity to annotate proteomes.
+This repository documents the idea of Fabi and Niko to use predicted protein structures and
+structure similarity to annotate proteomes.
 
 ## Background
-[AlphaFold](https://www.nature.com/articles/s41586-021-03819-2) changed how we think about protein structures. By leveraging deep learning, multiple sequence alignments, and the ever-expanding library of solved protein structures, AlphaFold is able to predict three-dimensional protein structures at resolutions that rival solved crystal structures, and has immediately found use in large parts of biological research.
+[AlphaFold](https://www.nature.com/articles/s41586-021-03819-2) changed how we think about protein
+structures. By leveraging deep learning, multiple sequence alignments, and the ever-expanding
+library of solved protein structures, AlphaFold is able to predict three-dimensional protein
+structures at resolutions that rival solved crystal structures, and has immediately found use in
+large parts of biological research.
 
 One of the older dogmata in molecular biology concerns proteins, and holds that
 
 > Sequence defines structure defines function.
 
-In the past this was taken to mean that sequence similarity, above a certain threshold, correlated with sequence homology or even orthology. Sufficient sequence similarity was then inferred to mean high structural similarity (this was mostly proven by the success of homology modelling in CASP) as well as functional equivalence. In fact, "bidirectional best BLAST hits" is the _de facto_ baseline for functional annotation.
+In the past this was taken to mean that sequence similarity, above a certain threshold, correlated
+with sequence homology or even orthology. Sufficient sequence similarity was then inferred to mean
+high structural similarity (this was mostly proven by the success of homology modelling in CASP) as
+well as functional equivalence. In fact, "bidirectional best BLAST hits" is the _de facto_ baseline
+for functional annotation.
 
-The question of the threshold turns out to be an important one. Burkhardt Rost famously outlined it as the ["twilight zone"](https://pubmed.ncbi.nlm.nih.gov/10195279/) of protein similarity; briefly, while we can confidently assert structural similarity (at least to the fold level) for sequences with >30% sequence identity, this deteriorates impressively fast at 25% sequence identity. In the years since this pronouncement, sequence search algorithms with increased sensitivity exploited the mountains of new sequencing data to dive into the twilight zone and detect remote homology; however, at low sequence identity, the sequence information alone is still not enough to guarantee homology.
+The question of the threshold turns out to be an important one. Burkhardt Rost famously outlined it
+as the ["twilight zone"](https://pubmed.ncbi.nlm.nih.gov/10195279/) of protein similarity; briefly,
+while we can confidently assert structural similarity (at least to the fold level) for sequences
+with >30% sequence identity, this deteriorates impressively fast at 25% sequence identity. In the
+years since this pronouncement, sequence search algorithms with increased sensitivity exploited the
+mountains of new sequencing data to dive into the twilight zone and detect remote homology; however,
+at low sequence identity, the sequence information alone is still not enough to guarantee homology.
 
-However, structure is more conserved than sequence. In theory, predicted structures can be compared against known structures that are otherwise annotated, allowing for the transfer of functional annotations (albeit less specific than sequence-based ones, since we will be detecting very remote homology at best). This is of particular interest for non-model organisms, especially ones outside the well-studied taxonomic groups (e.g. vertebrates or ecdysozoans).
+However, structure is more conserved than sequence. In theory, predicted structures can be compared
+against known structures that are otherwise annotated, allowing for the transfer of functional
+annotations (albeit less specific than sequence-based ones, since we will be detecting very remote
+homology at best). This is of particular interest for non-model organisms, especially ones outside
+the well-studied taxonomic groups (e.g. vertebrates or ecdysozoans).
 
 ## Follow-up ideas
-Having a phylome, scRNAseq/cell type annotation, functional proteomic data and the prediction of protein structures for a non-bilaterian Metazoan presents a unique combination that would allow to ask many fundamental questions. Potential follow-up analysis include:
+Having a phylome, scRNAseq/cell type annotation, functional proteomic data and the prediction of
+protein structures for a non-bilaterian Metazoan presents a unique combination that would allow to
+ask many fundamental questions. Potential follow-up analysis include:
 
-- Correltation between AF prediction accuracy (overall, domain specific, etc.) and sequence identity/similarity or bitscore of best FoldSeek hit. I.e.: "Does higher sequence identity mean better prediction accuracy?"
-- Relationship between identified homologs through sequence search (orthofinder, eggnog-mapper, blast, phylome) and best hits in FoldSeek for single sponge proteins. I.e.: "Do the best AF hits also include proteins identified as homologs in the phylome? Is there a sequence identity threshold to that?"
-- Is there biological meaning to best FoldSeek hits of un-annotated, highly expressed genes in the scRNAseq dataset or differentially regulated hits in the functional proteomic datasets?. I.e.: "Can we transfer function / functionally annotate previously un-annotated hits in scRNAseq and protomics data and most importantly, do these hits make sense when taking prior knowledge into account?"
+- Correlation between AF prediction accuracy (overall, domain specific, etc.) and sequence
+  identity/similarity or bitscore of best FoldSeek hit. I.e.: "Does higher sequence identity mean
+  better prediction accuracy?"
+- Relationship between identified homologs through sequence search (orthofinder, eggnog-mapper,
+  blast, phylome) and best hits in FoldSeek for single sponge proteins. I.e.: "Do the best AF hits
+  also include proteins identified as homologs in the phylome? Is there a sequence identity
+  threshold to that?"
+- Is there biological meaning to best FoldSeek hits of un-annotated, highly expressed genes in the
+  scRNAseq dataset or differentially regulated hits in the functional proteomic datasets?. I.e.:
+  "Can we transfer function / functionally annotate previously un-annotated hits in scRNAseq and
+  protomics data and most importantly, do these hits make sense when taking prior knowledge into
+  account?"
 
 ## Usage
 (eventually a tutorial on what order to use the scripts in, if we don't have a master script).
 
 ## Roadmap
-We will write this in as general a way as possible so that it can be reused by others/for other species. Platynereis is an obvious candidate after we proof _Spongilla_.
+We will write this in as general a way as possible so that it can be reused by others/for other
+species. Platynereis is an obvious candidate after we proof _Spongilla_.
 
 ## Authors and contributions
 

batch_predict.sh

-#!/bin/bash -ex
-#SBATCH -t 03-00
-#SBATCH -o /g/arendt/npapadop/cluster/alphafold-%j.out
-#SBATCH -e /g/arendt/npapadop/cluster/alphafold-%j.err
-#SBATCH -p gpu
-#SBATCH -C gpu=A100
-#SBATCH --gres=gpu:1
-#SBATCH -N 1
-#SBATCH --ntasks=32
-#SBATCH --mem=512000
-
-module load Anaconda3
-module load GCC
-module load bzip2
-module load CUDA
-source ~/.bash_profile
-conda activate /g/arendt/npapadop/repos/condas/maf
-
-BASE="/scratch/npapadop/"
-BATCH=$1
-
-cd "${BASE}"
-
-echo "$BATCH"
-
-colabfold_batch msas/"$BATCH"/ spongilla_structures/ --stop-at-score 85
-
-module unload
-

gpu_test.sh

-#!/bin/bash -ex
-#SBATCH -J predict
-#SBATCH -t 30:00
-#SBATCH -o /g/arendt/npapadop/cluster/2080alphafold.out
-#SBATCH -e /g/arendt/npapadop/cluster/2080alphafold.err
-#SBATCH -p gpu
-#SBATCH -C gpu=2080Ti
-#SBATCH --gres=gpu:1
-#SBATCH -N 1
-#SBATCH --ntasks=32
-#SBATCH --mem=256000
-
-module load Anaconda3
-module load GCC
-module load bzip2
-module load CUDA
-source ~/.bash_profile
-conda activate /g/arendt/npapadop/repos/condas/maf
-
-BASE="/scratch/npapadop/results/"
-
-cd "${BASE}"
-colabfold_batch test/ predictions/ --stop-at-score 85
-
-module unload
\ No newline at end of file

mkdb.sh

-#!/bin/bash -ex
-#SBATCH -J mkdb
-#SBATCH -t 10:00
-#SBATCH -c 1
-#SBATCH --mem=1G
-#SBATCH -o /g/arendt/npapadop/cluster/mkdb.out
-#SBATCH -e /g/arendt/npapadop/cluster/mkdb.err
-
-INPUT_PEP="$1"
-OUTPUT_DIR="$2"
-NAME="$3"
-
-mkdir ${OUTPUT_DIR}
-cd ${OUTPUT_DIR}
-
-module load MMseqs2
-
-mmseqs createdb ${INPUT_PEP} ${NAME}
-
-module unload MMseqs2
\ No newline at end of file

predict_protein.sh

-#!/bin/bash
-
-#SBATCH --time=2-00:00:00
-#SBATCH -e AF_%x_err.txt
-#SBATCH -o AF_%x_out.txt
-#SBATCH --qos=normal
-#SBATCH -p gpu
-#SBATCH -N 1
-#SBATCH --ntasks=32
-#SBATCH --mem=512000
-
-module load AlphaFold
-module load GCC/10.2.0
-module load tqdm
-module load matplotlib
-
-SOFTWARE_DIR=<your dir>
-export PYTHONPATH=$SOFTWARE_DIR/ColabFold:$PYTHONPATH
-
-# If you use --cpus-per-task=X and --ntasks=1 your script should contain:
-# export ALPHAFOLD_JACKHMMER_N_CPU=$SLURM_CPUS_PER_TASK
-# export ALPHAFOLD_HHBLITS_N_CPU=$SLURM_CPUS_PER_TASK
-
-# TF_FORCE_UNIFIED_MEMORY='1' XLA_PYTHON_CLIENT_MEM_FRACTION='4.0' are optional but may be necessary for bigger sequences.
-# If you read this after 2050-01-01, probably you want to adjust the date
-# Add "--model-type AlphaFold2-ptm" option to run the old ColabFold for complexes,
-# equivalent to the original https://colab.research.google.com/github/sokrypton/ColabFold/blob/main/beta/AlphaFold2_advanced.ipynb
-TF_FORCE_UNIFIED_MEMORY='1' XLA_PYTHON_CLIENT_MEM_FRACTION='4.0' time $SOFTWARE_DIR/ColabFold/bin/colabfold_batch dimer.fasta ./ --templates
\ No newline at end of file

predict_structure.sh

-#!/bin/bash -ex
-#SBATCH -J test_colab
-#SBATCH -t 30:00
-#SBATCH -o /g/arendt/npapadop/cluster/alphafold_test.out
-#SBATCH -e /g/arendt/npapadop/cluster/alphafold_test.err
-#SBATCH -p gpu
-#SBATCH -C gpu=A100
-#SBATCH --gres=gpu:1
-#SBATCH -N 1
-#SBATCH --ntasks=32
-#SBATCH --mem=512000
-
-module load Anaconda3
-module load GCC
-module load bzip2
-module load CUDA
-source ~/.bash_profile
-conda activate /g/arendt/npapadop/repos/condas/colabfold
-
-BASE="/scratch/npapadop/"
-
-colabfold_batch "$BASE"/msas/test/ "$BASE"/test_res/ --stop-at-score 85
-
-module unload

scripts/README.md

+# scripts usage
+
+Here we list the scripts in their logical use order:
+
+### Get sequence databases and obtain multiple sequence alignments for _Spongilla_.
+
+* `databases.sh`: download and build indices for the sequence databases UniRef30 and ColabFold
+  EnvDB. Modified from ColabFold.
+* `databases_pdb.sh`: download (sequence) PDB and build index
+* `align.sh`: build multiple sequence alignments for the _Spongilla_ proteome. A wrapper around
+  `colabfold_search`.
+
+### Predict structures for _Spongilla_
+
+* `fasta-splitter.pl`: written by Kirill Kryukov. A utility to partition FASTA files into pieces.
+  Used with `--n-parts 32` to split the _Spongilla_ proteome in batches.
+* `predict_structures.sh`: a wrapper around `colabfold_batch` to submit jobs to the EMBL cluster.
+* `submit_colab.sh`: a simple for loop to handle all 32 batches.
+
+### Use FoldSeek to search against available structures
+
+* `fs_query.sh`: build structural database for _Spongilla_ predicted structures.
+* `fs_afdb.sh, fs_pdb.sh, fs_sp.sh`: download the precomputed FoldSeek databases for AFDB, PDB, and
+  SwissProt, respectively. Split in three so we could run them in parallel.
+* `fs_search_afdb.sh, fs_search_pdb.sh, fs_search_swissprot.sh`: search with the _Spongilla_
+  structure database against the three target databases, AFDB, PDB, and SwissProt. 
\ No newline at end of file

create_index.sh → scripts/databases_pdb.sh

@@ -6,12 +6,13 @@
 #SBATCH -o /g/arendt/npapadop/cluster/create_index.out
 #SBATCH -e /g/arendt/npapadop/cluster/create_index.err
 
+# build the indices for the sequence databases uniref30, PDB. This is later used for the
+# MSAs of the Spongilla sequences.
+
 cd /scratch/npapadop/database
 
 module load MMseqs2
 
-mmseqs tsv2exprofiledb "uniref30_2103" "uniref30_2103_db"
-mmseqs createindex "uniref30_2103_db" tmp1 --remove-tmp-files 1 --threads 64
 mmseqs databases PDB PDB tmp
 mmseqs createindex PDB tmp3 --remove-tmp-files 1 --threads 64
 

spongilla_create_index.sh

-#!/bin/bash -ex
-
-SCRIPT="/g/arendt/npapadop/repos/spongfold/create_index.sh"
-OUTPUT_DIR="/scratch/npapadop/Spongilla_lacustris_70AAcutoffTransDecoder/"
-DB_NAME="spongilla"
-
-sbatch ${SCRIPT} ${OUTPUT_DIR} ${DB_NAME}
\ No newline at end of file