Skip to content

Changes

Page history
Final AdaptiveFeedback.md authored by Antonio Politi's avatar Antonio Politi
Show whitespace changes
Inline Side-by-side
AdaptiveFeedback.md
View page @ e6468084
# Adaptive feedback microscopy/online image analysis # MyPiC adaptive feedback microscopys
<a name="back"></a> <a name="back"></a>
With the adaptive feedback microscopy capability of MyPiC the user can combine online image analysis and image acquisition to perform: With the adaptive feedback microscopy capability of MyPiC the user can combine online image analysis and image acquisition to perform:
* Detection of rare events and start a different imaging protocol * Detection of rare events and start of a different imaging protocol
* Automated FCS and imaging experiments * Automated FCS and imaging experiments
* Time-lapse FCS measurements * Time-lapse FCS measurements
* Cell tracking * Cell tracking
* Fluorescence recovery after photobleaching (FRAP) experiments * Fluorescence recovery after photobleaching (FRAP) experiments
Images are acquired by the microscope and then processed by an image analysis program that is monitoring the arrival of new images to be analyzed. After the image has been analyzed commands are sent to the microscope in order to start a different pipeline or update the XYZ position for tracking an object in space and time. To start a different pipeline upon event detection the settings for [Trigger1 and/or Trigger2](#trigger) must be defined. Images are acquired by the microscope and then processed by an image analysis program that is monitoring the arrival of new images to be analyzed. After the image has been analyzed, commands are sent to the microscope in order to start a different pipeline or update the XYZ position for tracking an object in space and time. To start a different pipeline upon event detection the settings for [Trigger1 and/or Trigger2](#trigger) must be defined.
The communication between the image analysis software is through the [windows registry](#windowsregistry), thus any program that can read and write to the windows registry can be used. An example of an application using ImageJ for online image analysis can be found in [https://git.embl.de/politi/adaptive\_feedback\_mic\_fiji](https://git.embl.de/politi/adaptive_feedback_mic_fiji). The communication between the image analysis software is through the [windows registry](#windowsregistry). Thus any program that can read and write to the windows registry can be used. An example of an application using ImageJ for online image analysis can be found in [https://git.embl.de/politi/adaptive\_feedback\_mic\_fiji](https://git.embl.de/politi/adaptive_feedback_mic_fiji).
**Contents of the manual** **Contents of the manual**
* [Trigger1 and Trigger2 pipelines](#trigger) * [Trigger1 and Trigger2 pipelines](#trigger)
* [Adaptive Feedback workflows](#adaptive) * [Adaptive feedback workflows](#adaptive)
* [Automated imaging and F(C)CS for FCS-calibrated imaging and more.](#fcscalibrated) * [Automated imaging and F(C)CS for FCS-calibrated imaging and more](#fcscalibrated)
* [Windows registry commands](#windowsregistry) * [Windows registry commands](#windowsregistry)
...@@ -28,13 +25,13 @@ The communication between the image analysis software is through the [windows re ...@@ -28,13 +25,13 @@ The communication between the image analysis software is through the [windows re
## [<img src="./images/up.PNG">](#back)<a name= trigger></a> Trigger1 and Trigger2 pipelines ## [<img src="./images/up.PNG">](#back)<a name= trigger></a> Trigger1 and Trigger2 pipelines
The pipelines **Trigger1** or **Trigger2** are executed when the command ```trigger1``` or ```trigger2```, respectively, is received through the windows registry [**CodeMic**](#codemic). A typical workflow is a multi-position low-resolution imaging using the **Default** pipeline. When objects of interest are detected using online image analysis then the **Trigger1/2** pipeline is started with position(s) specified using the registry keys [**X, Y, Z**](#xyz). If no stage coordinates are specified imaging is performed at the current stage and focus position of the **Default** pipeline. The pipelines **Trigger1** or **Trigger2** are executed when the command ```trigger1``` or ```trigger2```, respectively, is received through the windows registry [**CodeMic**](#codemic). A typical workflow is a multi-position low-resolution imaging using the **Default** pipeline. When objects of interest are detected using online image analysis, the **Trigger1/2** pipeline is started with position(s) specified using the registry keys [**X, Y, Z**](#xyz). If no stage coordinates are specified, imaging is performed at the current stage and focus position of the **Default** pipeline.
Multiple imaging/stage positions can be imaged sequentially without returning to the **Default** pipeline. This occurs when Multiple imaging/stage positions can be imaged sequentially without returning to the **Default** pipeline. This occurs when:
1. Several positions are specified at once in the registry [**X, Y, Z**](#xyz) from the analysis of a single image from the **Default** pipeline. 1. Several positions are specified at once in the registry [**X, Y, Z**](#xyz) from the analysis of a single image from the **Default** pipeline.
2. The start of the **Trigger1/2** pipeline is delayed. The pipeline starts when a certain number of stage positions are specified within a time interval. During this interval the **Default** pipeline continues. 2. When **max. #Pts** > 1 the start of the **Trigger1/2** pipeline is delayed up all points are found. There is a time limit for search set by **max. waiting (sec)**.
When all repetitions of all positions are finished the imaging continues with the **Default** pipeline. The setup of the pipelines is similar as for the default pipeline. Within a **Trigger1/2** pipeline additional online image analysis steps can be specified. When all repetitions of all positions are finished the imaging continues with the **Default** pipeline. The setup of the pipelines is similar as for the default pipeline. Within a **Trigger1/2** pipeline additional online image analysis steps can be specified.
...@@ -42,40 +39,39 @@ When all repetitions of all positions are finished the imaging continues with th ...@@ -42,40 +39,39 @@ When all repetitions of all positions are finished the imaging continues with th
Trigger1/2 pipelines | Explanations of the buttons Trigger1/2 pipelines | Explanations of the buttons
:------------|:--------------------- :------------|:---------------------
<img src="./images/PipCon_trigger1.PNG" width = "400px"> | 1. Set thd order of tasks and Z-offset <br/> 2. Specify whether further process task <br/> 3. Set repetitions <br/> 4. Starts pipeline after **max. waiting (sec)** interval has elapsed. Interval is <br/> 5. Start pipeline when at least **max # Pts** stage positions have been specified. <br/> 6. If **Keep parent position** is unchecked then the stage position <br/>of the **Default** pipeline from which the trigger has been released is deleted. <br/> This option is useful to avoid triggering repeated false positives from the same position. <img src="./images/PipCon_trigger1.PNG" width = "400px"> | 1. Set the order of tasks and Z offset <br/> 2. Specify further process task <br/> 3. Set repetitions <br/> 4. Start pipeline after **max. waiting (sec)** interval has elapsed <br/> 5. Start pipeline when at least **max. # Pts** stage positions have been specified <br/> 6. If **Keep parent position** is unchecked, the stage position <br/>of the **Default** pipeline from which the trigger has been released is deleted. <br/> This option is useful to avoid triggering repeated false positives from the same position.
## [<img src="./images/up.PNG">](#back) <a name=adaptiveFeedback></a> Adaptive feedback workflows ## [<img src="./images/up.PNG">](#back) <a name=adaptiveFeedback></a> Adaptive feedback workflows
With adaptive feedback microscopy complex experiments can be performed. We provide here an example for high-throughout F(C)CS and imaging using the ImageJ/FiJi macro [Automated FCS](https://git.embl.de/politi/adaptive_feedback_mic_fiji). With adaptive feedback microscopy complex experiments can be performed. We provide here an example for high-throughput FC(C)S and imaging using the ImageJ/FiJi macro [Automated FCS](https://git.embl.de/politi/adaptive_feedback_mic_fiji).
### <a name=fcscalibrated></a> Automated imaging and FC(C)S for FCS-calibrated imaging and more. ### <a name=fcscalibrated></a> Automated imaging and FC(C)S for FCS-calibrated imaging and more
The workflow can be used to obtain F(C)CS and imaging data in high-throuhput. The FCS measurements and images are linked so that the data can be used for FCS-Calibrated imaging. The workflow can be used to obtain FC(C)S and imaging data in high-throuhghput. The FCS measurements and images are linked so that the data can be used for FCS-calibrated imaging.
For an explanations of the parameters of the image analysis program refer to [https://git.embl.de/politi/adaptive\_feedback\_mic\_fiji.](https://git.embl.de/politi/adaptive_feedback_mic_fiji) For an explanation of the parameters of the image analysis program refer to [https://git.embl.de/politi/adaptive\_feedback\_mic\_fiji].(https://git.embl.de/politi/adaptive_feedback_mic_fiji)
For FCS analysis software and FCS calibrated imaging please refer to [https://git.embl.de/politi/FCSanalyze.](https://git.embl.de/politi/FCSAnalyze/) For the FCS analysis software and FCS-calibrated imaging please refer to [https://git.embl.de/politi/FCSanalyze](https://git.embl.de/politi/FCSAnalyze/).
In this example cells express a fluorescent protein (mEGFP) and their DNA has been stained with SiR-DNA (far red dye). Following imaging and FCS jobs are used in MyPiC In this example cells express a fluorescent protein (mEGFP) and their DNA has been stained with SiR-DNA (a far red dye). The following imaging and FCS jobs are used in MyPiC:
* AF: XZ scan with reflection mode on. * AF: XZ scan with reflection mode on
* LR: Low resolution image to detect cells expressing the protein at the correct level. * LR: Low resolution image to detect cells expressing the protein at the correct level. First channel is GFP second channel is DNA
* HR: High-resolution image is used to automatically place FCS points. The image can also be used to compute FCS-calibration parameters. * HR: High resolution image to automatically place FCS points. The image can also be used to compute FCS-calibration parameters. The image has 3 fluorescent channels. The first channel is GFP, second channel is an external marker, and third channel is DNA
* fcsPOI: FCS settings to measure the fluorescent protein. * fcsPOI: FCS settings to measure the fluorescent protein
A two step workflow is used to find cell of interest and start the FCS measurements A two step workflow is used to find a cell of interest and start the FCS measurements
* Default pipeline: * Default pipeline:
* task1: A XZ scan to find the reflection of the glass. The AF job with process **Center of Mass (thr)** and **TrackZ**. * task 1: A XZ scan to find the reflection of the glass. The AF job with process **Center of Mass (thr)** and **TrackZ**
* task2: A low resolution imaging of the fluorescent protein (Channel 1) and DNA (Channel 2). At the end of the image MyPiC waits for a command from **Automated FCS** . The LR job with proces **Online img. Analysis**. * task 2: A low resolution imaging of the fluorescent protein (Channel 1) and DNA (Channel 2). At the end of the image MyPiC waits for a command from **Automated FCS**. The LR job with proces **Online img. Analysis**
* Trigger 1: * Trigger 1:
* task1: A XZ scan to find the reflection of the glass. The AF job with process **Center of Mass (thr)** and **TrackZ**. * task 1: A XZ scan to find the reflection of the glass. The AF job with process **Center of Mass (thr)** and **TrackZ**
* task2: A high resolution imaging of the fluorescent protein (Channel 1) and DNA (Channel 3). At the end of the image MyPiC waits for a command from **Automated FCS**. The HR job image with proces **Online img. Analysis**. * task 2: A high resolution imaging of the fluorescent protein (Channel 1) and DNA (Channel 3). At the end of the image MyPiC waits for a command from **Automated FCS**. The HR job image with proces **Online img. Analysis**
* task 3: FCS measurements * task 3: FCS measurements
...@@ -87,35 +83,35 @@ A two step workflow is used to find cell of interest and start the FCS measureme ...@@ -87,35 +83,35 @@ A two step workflow is used to find cell of interest and start the FCS measureme
The FiJi plugin [**Automated FCS**](https://git.embl.de/politi/adaptive_feedback_mic_fiji) is used analyze the image Default pipeline task2 and Trigger1 pipeline task 2. Following parameters are used The FiJi plugin [**Automated FCS**](https://git.embl.de/politi/adaptive_feedback_mic_fiji) is used to analyze the image of Default pipeline task 2 and Trigger1 pipeline task 2. The following parameters are used:
<div align = "center" > <img src = './images/fcscalibrated/automatedfcs2.png' width = "400px" ></div> <div align = "center" > <img src = './images/fcscalibrated/automatedfcs2.png' width = "400px" ></div>
The **Automated FCS** plugins performs: The **Automated FCS** plugin performs:
* Job1: Cell detection using the DNA channel. Cells where the fluorescent protein is within a certain range are selected. The coordinates of cells that fulfill all criteria are passed to MyPiC to start the Trigger1 pipeline. * Job1: Cell detection using the DNA channel. Cells expressing the fluorescent protein within a certain range are selected. The coordinates of cells fulfilling all criteria are passed to MyPiC to start the Trigger1 pipeline.
* Job2: Cell detection using the DNA channel. Chooses the cell closest to the center where the fluorescent protein is within a certain range. Determines the coordinates of the FCS measurements and pass their values to MyPiC. In task 3 of Trigger1 MyPiC will start FCS measurements at these positions. * Job2: Cell detection using the DNA channel. Chooses the cell closest to the center in which the fluorescent protein is expressed within a certain range. Determines the coordinates of the FCS measurements and passes their values to MyPiC. In task 3 of Trigger1 MyPiC will start FCS measurements at these positions.
**JOB1: Output generated for every processed image** | **JOB1: Output generated when objects of interest are found** **JOB1: Output generated for every processed image** | **JOB1: Output generated when objects of interest are found**
:--- | :--- :--- | :---
<img src = './images/fcscalibrated/example_output.png' width = "400px" > | <img src = './images/fcscalibrated/example_output2.png' width = "400px" > <img src = './images/fcscalibrated/example_output.png' width = "400px" > | <img src = './images/fcscalibrated/example_output2.png' width = "400px" >
Segmentation of the DNA channel (red) returned 17 ROIs. <br/>Only for ROI 10 and 13 is the intensity of the fluorescent protein (green) within the specified range. | Outlines for the two cells picked are shown in green.<br/> The coordinates are passed to MyPiC to execute the Trigger1 pipeline for each position. Segmentation of the DNA channel (red) returned 17 ROIs. <br/>Only for ROI 10 and 13 the intensity of the fluorescent protein (green) is within the specified range. | Outlines for the two cells picked are shown in green.<br/> The coordinates are passed to MyPiC to execute the Trigger1 pipeline for each position.
**JOB2: Output generated for every processed image**| **JOB2: Output generated when objects of interest are found** **JOB2: Output generated for every processed image**| **JOB2: Output generated when objects of interest are found**
<img src = './images/fcscalibrated/example_output3.png' width = "400px" > | <img src = './images/fcscalibrated/example_output4.png' width = "400px" > <img src = './images/fcscalibrated/example_output3.png' width = "400px" > | <img src = './images/fcscalibrated/example_output4.png' width = "400px" >
Segmenation of the DNA channel (red) returned 2 ROIs. <br/> For ROI 1 and 2 the intensity of the fluorescent protein (green) is within the specified range. | Cell closest to center of image is picked (green outline). <br/>The red outline shows a cell that has not been picked because too far from the center of the image. <br/> Coordinates for the two FCS points are passed to MyPiC. Segmenation of the DNA channel (red) returned 2 ROIs. <br/> For ROI 1 and 2 the intensity of the fluorescent protein (green) is within the specified range. | The cell closest to the center of the image is picked (green outline).<br/> The red outline shows a cell that has not been picked because it is too far away from the center of the image. <br/> Coordinates for the two FCS points are passed to MyPiC.
## [<img src="./images/up.PNG">](#back) <a name=windowsregistry></a> Windows registry commands ## [<img src="./images/up.PNG">](#back) <a name=windowsregistry></a> Windows registry commands
MyPiC communicates with external programs by reading and writing in the Windows Registry MyPiC communicates with external programs by reading and writing in the Windows Registry:
``` ```
HKEY_CURRENT_USER\Software\VB and VBA Program Settings\OnlineImageAnalysis\macro HKEY_CURRENT_USER\Software\VB and VBA Program Settings\OnlineImageAnalysis\macro
``` ```
This occurs for each imaging job when **Online img. analysis** in the Method of [Processing and Tracking](#taskopt) is set. After the acquisition of the corresponding job the key [**CodeMic**](#codemic) will be set to `wait`, [**CodeOia**](#codeoia) to `newImage` and [**filePath**](#filepath) to the name of image to analyze. MyPiC will continuously monitor **CodeMic** until the value changes from `wait` to one of the other possible values. If **CodeMic** is `focus`, `trigger1`, `trigger2` coordinates from keys [**X, Y, Z**](#xyz) areb read. When the **CodeMic** is `setFcsPos` coordinates from the keys [**fcsX, fcsY, fcsZ**](#fcsxyz) are read. Via the registry the user can also dynamically set bleaching or acquisition [ROIs](#rois) for an automated FRAP experiment. The full list of online image analysis keys can be obtained by clicking on the **Show OIA Keys** button of the main Macro window. Here we provide a list of the most important commands. This occurs for each imaging job when **Online img. analysis** in the Method of [Processing and Tracking](#taskopt) is set. After the acquisition of the corresponding job the key [**CodeMic**](#codemic) will be set to `wait`, [**CodeOia**](#codeoia) to `newImage` and [**filePath**](#filepath) to the name of the image to analyze. MyPiC will continuously monitor **CodeMic** until the value changes from `wait` to one of the other possible values. If **CodeMic** is `focus`, `trigger1`, `trigger2` coordinates from keys [**X, Y, Z**](#xyz) are read. When the **CodeMic** is `setFcsPos`, coordinates from the keys [**fcsX, fcsY, fcsZ**](#fcsxyz) are read. Via the registry the user can also dynamically set bleaching or acquisition [ROIs](#rois) for an automated FRAP experiment. The full list of online image analysis keys can be obtained by clicking on the **Show OIA Keys** button of the main Macro window. Here we provide a list of the most important commands.
<div align='center'> <div align='center'>
<img src="./images/windowsRegistryCommands.jpg" width = "600px"> <img src="./images/windowsRegistryCommands.jpg" width = "600px">
...@@ -128,14 +124,14 @@ Contains the commands for the microscope ...@@ -128,14 +124,14 @@ Contains the commands for the microscope
Value | Meaning Value | Meaning
----------- | ------ ----------- | ------
`wait` | do not progress in imaging `wait` | Do not progress in imaging
`error` | the online image analsys pipeline has encountered an error `error` | The online image analysis pipeline has encountered an error
`nothing` | Nothing has been found `nothing` | Nothing has been found
`trigger1` | Start **Trigger1** pipeline using values given in **X**, **Y**, **Z** registry entries `trigger1` | Start **Trigger1** pipeline using values given in **X**, **Y**, **Z** registry entries
`trigger2` | Start **Trigger2** imaging using values given in **X**, **Y**, **Z** registry entries `trigger2` | Start **Trigger2** imaging using values given in **X**, **Y**, **Z** registry entries
`focus` | Update XYZ coordinate of current imaging position. This can be used to perform object based tracking when **TrackXY** and/or **TrackZ** are checked in MyPiC. `focus` | Update XYZ coordinate of current imaging position. This can be used to perform object based tracking when **TrackXY** and/or **TrackZ** are checked in MyPiC.
`setFcsPos` | Read position(s) for FCS measurement from **fcsX**, **fcsY**, **fcsZ**. <br/> Start an FCS experiment if next task in pipeline is a FCS job `setFcsPos` | Read position(s) for FCS measurement from **fcsX**, **fcsY**, **fcsZ**. <br/> Start an FCS experiment if the next task in the pipeline is a FCS job
`setRoi` | Read ROIs and update ROIs for subsequent task(s) in pipeline `setRoi` | Read ROIs and update ROIs for subsequent task(s) in the pipeline
<a name="codeoia"></a> <a name="codeoia"></a>
#### Key **CodeOia** #### Key **CodeOia**
...@@ -148,7 +144,7 @@ Contains the commands for the image analysis program. ...@@ -148,7 +144,7 @@ Contains the commands for the image analysis program.
<a name="filepath"></a> <a name="filepath"></a>
#### Key **filePath** #### Key **filePath**
Contains the full path of image that needs to be analysed Contains the full path of the image that needs to be analysed.
Value | Meaning Value | Meaning
------ | ------ ------ | ------
...@@ -158,25 +154,25 @@ Contains the full path of image that needs to be analysed ...@@ -158,25 +154,25 @@ Contains the full path of image that needs to be analysed
<a name="xyz"></a> <a name="xyz"></a>
#### Keys **X**, **Y**, **Z** #### Keys **X**, **Y**, **Z**
Specifies stage positions. Coordinates with respect to upper left corner of the image (in pixels) and first slice (0,0,0). Several positions are separated by `;` Specifies stage positions. Coordinates with respect to the upper left corner of the image (in pixels) and the first slice (0,0,0). Several positions are separated by `;`
Value | Meaning Value | Meaning
----------- | ------ ----------- | ------
`X1;X2;X3;...` | X positions with respect to left corner in pixels `X1;X2;X3;...` | X positions with respect to left corner in pixels
`Y1;Y2;Y3;...` | Y positions with respect to upper corner in pixels `Y1;Y2;Y3;...` | Y positions with respect to upper corner in pixels
`Z1;Z2;Z3;...` | Z positions with respect to bottom slice in pixels. If this key is left empty current focus position is used. `Z1;Z2;Z3;...` | Z positions with respect to bottom slice in pixels. If this key is empty the current focus position is used
<a name="fcsxyz"></a> <a name="fcsxyz"></a>
#### Keys **fcsX**, **fcsY**, **fcsZ** #### Keys **fcsX**, **fcsY**, **fcsZ**
Specifies FCS scanner positions. Coordinates with respect to upper left corner of the image (in pixels) and first slice (0,0,0). Several positions are separated by `;` Specifies FCS scanner positions. Coordinates with respect to the upper left corner of the image (in pixels) and first slice (0,0,0). Several positions are separated by `;`
Value | Meaning Value | Meaning
----------- | ------ ----------- | ------
`X1;X2;X3;...` | X positions with respect to left corner in pixels `X1;X2;X3;...` | X positions with respect to left corner in pixels
`Y1;Y2;Y3;...` | Y positions with respect to upper corner in pixels `Y1;Y2;Y3;...` | Y positions with respect to upper corner in pixels
`Z1;Z2;Z3;...` | Z positions with respect to bottom slice in pixels. If this key is left empty current focus position is used `Z1;Z2;Z3;...` | Z positions with respect to bottom slice in pixels. If this key is empty the current focus position is used
<a name="rois"></a> <a name="rois"></a>
...@@ -187,10 +183,10 @@ A ROI is defined by 4 different keys. ...@@ -187,10 +183,10 @@ A ROI is defined by 4 different keys.
----------- | ------ ----------- | ------
`roiAim` | This can take the values `nothing`, `bleach`, `acquisition`, `acquisitionBleach` `roiAim` | This can take the values `nothing`, `bleach`, `acquisition`, `acquisitionBleach`
`roiType` | This can take the values `circle`, `rectangle`, `polyline`, `ellipse` `roiType` | This can take the values `circle`, `rectangle`, `polyline`, `ellipse`
`roiX` | The X coordinates of the ROI (in pixels) separated by a comma. `roiX` | The X coordinates of the ROI (in pixels) separated by a comma
`roiY`| The Y coordinates of the ROI (in pixels) separated by a comma. `roiY`| The Y coordinates of the ROI (in pixels) separated by a comma
* For a `circle` specify 2 points: the center of the circle and one point on the circle. * For a `circle` specify 2 points: the center of the circle and one point on the circle
* For a `rectangle` specify 2 points: the upper left and lower right corners. * For a `rectangle` specify 2 points: the upper left and lower right corners
* For a `polyline` specify at least 3 points. * For a `polyline` specify at least 3 points
* For an `ellipse` specify 3 points: the center of the ellipse and a point on axis1 and axis2, respectively. * For an `ellipse` specify 3 points: the center of the ellipse and a point on axis1 and axis2, respectively
\ No newline at end of file \ No newline at end of file