README.md 5.96 KB
Newer Older
1
# AI4IA
2

3
Training and inference scripts for deep learning tools for cell segmentation in microscopy images and their setup for the EMBL cluster.
4 5

Available tools:
Constantin Pape's avatar
Constantin Pape committed
6
- [stardist](https://github.com/mpicbg-csbd/stardist): stardist for convex object segmentation
7
- utils: functionality for visualisation and job submission on compute clusters
8 9 10 11 12 13 14 15 16 17 18 19 20


## Data Layout

The goal of this small package is to provide an easy way to train different tools via the command line from the cell data layout.
In order to use it, you will need training data images and labels in the following layout:
```
root-folder/
  images/
  labels/
```
The folder `images` contains the training image data and the labels the training label data.
The corresponding images and labels **must have exactly the same name**.
21
The data should be stored in tif format. For multi-channel images, we assume that they are stored channel-first, i.e. in cyx order.
Constantin Pape's avatar
Constantin Pape committed
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43


## Setting up conda

The software dependencies of this repository are installed via conda.
If you don't have conda installed yet, check out the [installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/).
Note that there are two different versions of conda:
Anaconda, which comes with the conda package manager and a complete environment with popular python packages as well as miniconda, which only contains the conda package manager.
For our purposes, it is sufficient to install miniconda (but using anaconda does not hurt if you need it for some other purpose).

### Setting up a multi-user conda installation

In order to set up a shared conda installation  for multiple users on linux infrastrcture, follow these steps:

1. Download the latest version of miniconda: https://docs.conda.io/en/latest/miniconda.html. 
2. Open a terminal and cd to the location where you have downloaded the installation script
3. Set executable permissions: `chmod +x Miniconda3-latest-Linux-x86_64.sh`
4. Exectute the installation script: `./Miniconda3-latest-Linux-x86_64.sh`
5. Agree to the license
6. Choose the installation directory and start the installation. Make sure that all users have read and execute permissions for the installation location.
7. After the installation is finished, choose `no` when asked `Do you wish to initialize Miniconda3 by running conda init?`.
8. This will end the installation and print out a few lines explaining how to activate the conda base environment.
Constantin Pape's avatar
Constantin Pape committed
44
9. Copy the line `eval "$(/path/to/miniconda3/bin/conda shell.YOUR_SHELL_NAME hook)"`and paste it into a file `init_conda.sh`.
Constantin Pape's avatar
Constantin Pape committed
45 46
10. Replace `YOUR_SHELL_NAME` with `bash` (assuming you and your users are using a bash shell; for zshell, replace with `zsh`, etc.).

Constantin Pape's avatar
Constantin Pape committed
47
Now, the conda base environment can be activated via running `source init_conda.sh`. 
Constantin Pape's avatar
Constantin Pape committed
48 49 50 51 52
Use it to set up the environments to make the applications available to your users.
Important: in order to install version conflicts use separate environments for different application and don't install them to the base environment!

In order for users to activate one of these environments, they will need to first activate the base environment and then the desired env:
```shell
Constantin Pape's avatar
Constantin Pape committed
53
source init_conda.sh
Constantin Pape's avatar
Constantin Pape committed
54 55 56 57 58 59
conda activate <ENV_NAME>
```

### Advanced: setting up a conda environment for multiple maintainers and users

TODO
Christian Tischer's avatar
Christian Tischer committed
60 61 62 63


### Launching an interactive jupyter notebook on a GPU cluster node

Christian Tischer's avatar
Christian Tischer committed
64 65 66
#### Prerequisites

- Make sure you are in the EMBL intranet
Christian Tischer's avatar
Christian Tischer committed
67
- You need a terminal with the `ssh` command. On Mac and Linux this is included. On Windows please install [git for windows](https://git-scm.com/download/win) which includes a ssh command client (choose *Run Git and included Unix tools form Windows Command Prompt* on installation).
Christian Tischer's avatar
Christian Tischer committed
68 69 70 71
- Your data and jupyter notebooks must be on a group-share, e.g. (Linux: `/g/kreshuk`; Mac: `/Volumes/kreshuk`; Windows: `\\kreshuk\kreshuk` )

#### Log into cluster and start an interactive jupyter notebook with stardist environment

Christian Tischer's avatar
Christian Tischer committed
72
![](./docs/images/interactive-jupyter-on-cluster.gif = 250x)
Christian Tischer's avatar
Christian Tischer committed
73

Christian Tischer's avatar
Christian Tischer committed
74
- Open a terminal window on your computer
Christian Tischer's avatar
Christian Tischer committed
75
- Log into the EMBL cluster: 
Christian Tischer's avatar
Christian Tischer committed
76
    - `ssh EMBLUSERNAME@login.cluster.embl.de`
Christian Tischer's avatar
Christian Tischer committed
77 78 79 80 81 82 83 84 85 86
- Enter you EMBL password.
- Ask for an interactive job on a GPU node: 
    - `srun -t 60:00 -N1 -n4 --mem 32G -p gpu -C "gpu=2080Ti|gpu=1080Ti" --gres=gpu:1 -W 0 --pty -E $SHELL`
    - The time and memory needs to be adjusted (HOW?)
    - If the cluster is busy, it may take some time until you get the job...
- Store the IP address of the cluster node into a variable: 
    - `IP=$(hostname -i)`
- Activate conda:
    - `source /g/almf/software/miniconda3/run_conda.sh`
- Activate the stardist environment
Christian Tischer's avatar
Christian Tischer committed
87
    - `conda activate stardist-gpu`
Christian Tischer's avatar
Christian Tischer committed
88
- Go to a folder containing your notebooks
Christian Tischer's avatar
Christian Tischer committed
89
    - `cd /g/GROUP/FOLDER/WITH/NOTEBOOKS`
Christian Tischer's avatar
Christian Tischer committed
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110
- Run jupyter notebook with the IP of the cluster node that you are currently on
    - `jupyter notebook --ip $IP`

The last command will result in an output like this:

```
[I 08:51:35.097 NotebookApp] Serving notebooks from local directory: /home/tischer
[I 08:51:35.097 NotebookApp] The Jupyter Notebook is running at:
[I 08:51:35.097 NotebookApp] http://10.11.12.249:8888/?token=235e5613589eb1d80435099afe395dad536bde40e3fde755
[I 08:51:35.097 NotebookApp]  or http://127.0.0.1:8888/?token=235e5613589eb1d80435099afe395dad536bde40e3fde755
[I 08:51:35.097 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 08:51:35.133 NotebookApp] No web browser found: could not locate runnable browser.
[C 08:51:35.133 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///home/tischer/.local/share/jupyter/runtime/nbserver-51995-open.html
    Or copy and paste one of these URLs:
        http://10.11.12.249:8888/?token=235e5613589eb1d80435099afe395dad536bde40e3fde755
     or http://127.0.0.1:8888/?token=235e5613589eb1d80435099afe395dad536bde40e3fde755
```

Christian Tischer's avatar
Christian Tischer committed
111 112 113
Finally, you need to copy and paste the second last line, here `http://10.11.12.249:8888/?token=235e5613589eb1d80435099afe395dad536bde40e3fde755` into a webbrowser on your computer.


Christian Tischer's avatar
Christian Tischer committed
114 115


Christian Tischer's avatar
Christian Tischer committed
116
     
Christian Tischer's avatar
Christian Tischer committed
117