Commit 3ed6a343 authored by Balint Balazs's avatar Balint Balazs
Browse files

initial commit with working Windows binaries

parents
The MIT License (MIT)
Copyright (c) 2017 Balint Balazs
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# B³D compression library
B³D is a fast (~1 GB/s), GPU based image compression method, developed for
fluorescence microscopy applications. Alongside lossless compression, it offers
a noise dependent lossy compression mode, where the loss can be tuned as a
proportion of the inherent image noise (accounting for photon shot noise and
camera read noise). It not only allows for fast compression during imaging, but
can achieve compression ratios up to 100 fold.
Corresponding publication:
[http://www.biorxiv.org/content/early/2017/07/21/164624](http://www.biorxiv.org/content/early/2017/07/21/164624)
## Requirements
* Microsoft Windows 7 or newer, 64-bit
* HDF5 1.8.15 64-bit or later installed. Obtain the
latest version here: [https://support.hdfgroup.org/HDF5/release/obtain518.html](https://support.hdfgroup.org/HDF5/release/obtain518.html)
* Microsoft Visual C++ 2013 Redistributable (x64) (installer included in
`vcredist` folder)
* CUDA capable graphics card, minimum Compute Capability 3.0
* CUDA 8 compatible graphics driver (for GeForce products 378.66 or later)
HDF5 1.10 should also be compatible, but was not tested.
## Installation
All necessary binary files can be found in the `win64` folder. In order for
HDF5 to recognize the filter, the path of this folder has to be stored in the
`%HDF5_PLUGIN_PATH%` environment variable. This can be done automatically by
running the `install_b3d.bat` batch file. It will set the environment variable
for the current user, or if it already exists (e.g. other filters are already
in use), it will copy the necessary files there.
## Reading compressed files
If all the requiriements are present, and installation was successfull, any
HDF5 application should be able to automatically use the filter. This
functionality can be tested by opening the provided example file
`B3D_demo_drosophila_compressed.h5` in HDFView for example.
### Reading files in Fiji
Fiji comes with its own distribution of HDF5 in the form of
`jhdf5-14.12.5.jar` in the `Fiji.app/jars` folder. On Windows machines this
is **not** compatible with the filter because a [different version of Visual
Studio](http://siomsystems.com/mixing-visual-studio-versions/) was used to
compile the native libraries. To circumvent this issue, we provide a recompiled
version of the same file, included in the `win64` folder. Copy this file to the
`Fiji.app/jars` folder, overwriting the original if necessary.
To open HDF5 files, you need to enable the HDF5 update site (Help > Update,
then click on "Manage update sites" and select "HDF5" from the list). During
the update process you might get a warning message: "There are locally modified
files". As this is a consequence of the previous step, it's important not to
let Fiji overwrite the locally modified files.
## Writing compressed files
To write compressed files, the compression has to be set by calling
`H5Pset_filter` on the dataset creation property list (`dcpl_id`).
```c
H5Pset_filter(dcpl_id, 32016, H5Z_FLAG_OPTIONAL, cd_nelmts, cd_values);
```
where 32016 is the filter id for B³D compression, and `cd_nelmts` is the number
of elements in `cd_values`. `cd_values` is an array comprising of the
following elements:
* cd_values[0] Quantization step*. Relative to sigma (see paper for details;
default = 0). Set to 0 for lossless compression.
* cd_values[1] Compression mode. 1: Mode1, 2: Mode2 (see paper for details;
default = 1)
* cd_values[2] Camera conversion parameter* in DN/e- (default=1000)
* cd_values[3] Camera background offset parameter in DN (default = 0)
* cd_values[4] Camera read noise parameter* in e- (default = 0)
* cd_values[5] Tile size, to optimize parallel execution (default=24)
\* Since `cd_values` can only be integers, for floating point parameters the
value*1000 rounded to integers has to be stored in `cd_values`. For lossless
compression all elements can be omitted.
For more information on how to use HDF5 with compression,
[see the official HDF5 website.](https://support.hdfgroup.org/HDF5/faq/compression.html)
### Performance considerations
Filters can only be applied to datasets if chunking is also enabled. For
optimal performance, we recommend choosing a chunk size ~2 MB (1M elements
with 16 bit data).
### Supported data types
* Datasets of 2 or 3 dimensions
* signed or unsigned 8 bit or 16 bit integers
## Usage examples
Example scripts are located in the `sample_scripts` folder. Currently this
includes a Python example (tested with Python 2.7.10 and 3.5.3)
\ No newline at end of file
# -*- coding: utf-8 -*-
"""
Created on Fri Jan 27 16:37:34 2017
@author: Balint Balazs
"""
import os
import h5py
#%% load uncompressed dataset
baseFolder = "../"
inFile = "B3D_demo_drosophila_uncompressed.h5"
f = h5py.File(baseFolder+inFile, "r+")
dset = f['Data']
# set chunk size to the same as input
chunks = dset.chunks
#%% This will remove outifle if it already exists
outFile = "B3D_demo_drosophila_compressed.h5"
if (os.access(baseFolder+outFile, os.F_OK)):
os.remove(baseFolder+outFile)
#%% lossless compression
f2 = h5py.File(baseFolder+outFile)
dset2= f2.create_dataset("B3D_Lossless", data=dset.value, chunks=chunks,
compression=32016)
f2.close()
#%% within noise level compression, Mode 1
compressionMode = 1
quantStep = 1 # q=1*sigma
conversion = 2.1845 # in DN/e-, here 65535/30000
bgLevel = 0 # set it to camera average background level
readNoise = 1.6 # in e-
outPath = baseFolder+outFile
f2 = h5py.File(outPath)
dset2= f2.create_dataset("B3D_Mode1_1.00", data=dset.value, chunks=chunks,
compression=32016,
compression_opts=(round(quantStep*1000), compressionMode, round(conversion*1000), bgLevel, round(readNoise*1000)))
f2.close()
#%% within noise level compression, Mode 2
compressionMode = 2
quantStep = 1 # q=1*sigma
conversion = 2.1845 # 65535/30000
bgLevel = 0 # set it to camera average background level
readNoise = 1.6 # in e-
outPath = baseFolder+outFile
f2 = h5py.File(outPath)
dset2= f2.create_dataset("B3D_Mode2_1.00", data=dset.value, chunks=chunks,
compression=32016,
compression_opts=(round(quantStep*1000), compressionMode, round(conversion*1000), bgLevel, round(readNoise*1000)))
f2.close()
#%%
f.close()
\ No newline at end of file
@echo off
set folder=%~dp0
IF %folder:~-1%==\ SET folder=%folder:~0,-1%
rem echo %folder%
if defined HDF5_PLUGIN_PATH (
echo HDF5_PLUGIN_PATH defined
if "%folder%"=="%HDF5_PLUGIN_PATH%" (
echo Everything is already set up.
) else (
echo Copying dlls to %HDF5_PLUGIN_PATH%
copy /-Y "%folder%\*.dll" "%HDF5_PLUGIN_PATH%"
)
) else (
echo HDF5_PLUGIN_PATH not defined. Setting it to current directory.
setx HDF5_PLUGIN_PATH "%folder%"
)
pause
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment