This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
downloads:cryolo_1 [2019/02/12 13:28] twagner [Installation] |
downloads:cryolo_1 [2019/07/11 11:25] twagner [General PhosaurusNet models] |
||
---|---|---|---|
Line 7: | Line 7: | ||
If you are interested in using crYOLO in a commercial context please contact [[stefan.raunser@mpi-dortmund.mpg.de|stefan.raunser@mpi-dortmund.mpg.de]] | If you are interested in using crYOLO in a commercial context please contact [[stefan.raunser@mpi-dortmund.mpg.de|stefan.raunser@mpi-dortmund.mpg.de]] | ||
+ | |||
+ | ====== Paper ====== | ||
+ | |||
+ | You can find more technical details in our paper: | ||
+ | |||
+ | Nature Communications Biology: | ||
+ | [[https:// | ||
+ | |||
+ | Preprint: | ||
+ | [[https:// | ||
+ | |||
+ | < | ||
+ | <a href=" | ||
+ | </ | ||
+ | |||
====== Download ====== | ====== Download ====== | ||
Line 19: | Line 34: | ||
====crYOLO==== | ====crYOLO==== | ||
- | Version: 1.2.3 | + | Version: 1.4.0 |
- | Uploaded: | + | Uploaded: |
- | [[ftp:// | + | [[ftp:// |
+ | |||
+ | Please see [[downloads: | ||
====crYOLO boxmanager==== | ====crYOLO boxmanager==== | ||
- | Version: 1.1.1 | + | Version: 1.2.3 |
- | Uploaded: | + | Uploaded: |
- | [[ftp:// | + | [[ftp:// |
[{{ : | [{{ : | ||
- | ==== General PhosaurusNet | + | ==== General PhosaurusNet |
- | | + | |
- | Uploaded: 21. December 2018 | ||
- | [[ftp:// | + | === For cryo images (low-pass filtered) === |
+ | Number of datasets: 38 real, 10 simulated, 10 particle free datasets on various grids with contamination | ||
+ | |||
+ | Uploaded: 17. May 2019 | ||
+ | |||
+ | [[ftp:// | ||
|DOWNLOAD]] | |DOWNLOAD]] | ||
- | [[pipeline: | + | [[pipeline: |
- | Info: Trained on cryo images, | + | === For cryo images |
+ | Number of datasets: 38 real, 10 simulated, 10 particle free datasets on various grids with contamination | ||
+ | Uploaded: 11. July 2019 | ||
+ | [[ftp:// | ||
+ | |DOWNLOAD]] | ||
+ | |||
+ | [[pipeline: | ||
+ | |||
+ | < | ||
+ | Plase plas pdlasdas dasd | ||
+ | </ | ||
+ | |||
+ | |||
+ | |||
+ | === For negative stain images === | ||
+ | Number of datasets: 10 real datasets | ||
+ | |||
+ | Uploaded: 26. February 2019 | ||
+ | |||
+ | [[ftp:// | ||
+ | |DOWNLOAD]] | ||
+ | [[pipeline: | ||
====ARCHIVE==== | ====ARCHIVE==== | ||
Line 55: | Line 96: | ||
====== Known issues ===== | ====== Known issues ===== | ||
* Issue 0: Training on multiple GPUs sometimes lead to worse performance (higher loss). We currently recommend to train on single gpus. | * Issue 0: Training on multiple GPUs sometimes lead to worse performance (higher loss). We currently recommend to train on single gpus. | ||
+ | * Issue 17: On the fly filtering (%%--%%otf) is slower than using it not, as the filtering is not parallelized in this case. | ||
+ | <hidden Closed issues> | ||
* < | * < | ||
* < | * < | ||
Line 66: | Line 109: | ||
* < | * < | ||
* < | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | * < | ||
+ | </ | ||
- | ====== Change log ===== | + | |
+ | ====== Installation ====== | ||
+ | |||
+ | **System requirements: | ||
+ | |||
+ | crYOLO was tested on **Ubuntu 16.04.4 LTS** and **Ubuntu 18.04** with an NVIDIA Geforce 1080 / Geforce 1080Ti. | ||
+ | |||
+ | However, it should run on **Windows** as well. | ||
+ | |||
+ | As the GPU accelerated version of tensorflow does not support **MacOS**, crYOLO does not support it either. | ||
+ | |||
+ | |||
+ | crYOLO depends on CUDA Toolkit 9.0 and the cuDNN 7.1.2 library. | ||
+ | It will be automatically installed during crYOLO installation. | ||
+ | |||
+ | **Install crYOLO!** | ||
+ | |||
+ | The following instructions assume that pip and [[https:// | ||
+ | In case you have a old cryolo environment installed, first remove the old one with: | ||
+ | < | ||
+ | conda env remove --name cryolo | ||
+ | </ | ||
+ | After that, create a new virtual environment: | ||
+ | < | ||
+ | conda create -n cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 cython | ||
+ | </ | ||
+ | |||
+ | Activate the environment: | ||
+ | < | ||
+ | source activate cryolo | ||
+ | </ | ||
+ | |||
+ | Install fast numpy from conda: | ||
+ | < | ||
+ | conda install numpy==1.15.4 | ||
+ | </ | ||
+ | |||
+ | In case you run **crYOLO on a GPU** run: | ||
+ | < | ||
+ | pip install cryolo-X.Y.Z.tar.gz[gpu] | ||
+ | </ | ||
+ | |||
+ | But if you want to run **crYOLO on a CPU** run: | ||
+ | < | ||
+ | pip install cryolo-X.Y.Z.tar.gz[cpu] | ||
+ | </ | ||
+ | |||
+ | Finally you install the cryolo boxmanager: | ||
+ | < | ||
+ | pip install cryoloBM-X.Y.Z.tar.gz | ||
+ | </ | ||
+ | |||
+ | **That' | ||
+ | |||
+ | You might want to check if everything is running as expected. Here is a reference example: | ||
+ | |||
+ | [[http:// | ||
+ | |||
+ | ===== Run it on the CPU ==== | ||
+ | |||
+ | There is also a way to run crYOLO on CPU. To use it, just follow the instruction in the [[downloads: | ||
+ | |||
+ | Picking with crYOLO is also quite fast on the CPU. On my local machine (Intel i9) it takes roughly 1 second per micrograph and on our low-performance notebooks (Intel i3) 4 seconds. | ||
+ | |||
+ | Training crYOLO is much more computational expensive. Training a model with 14 micrographs from scratch on my local machine take 34 minutes per epoch on the CPU. Given that you often need 25 epochs until convergence it is a task to do overnight (~ 12 hours). However, you might want to try [[pipeline: | ||
+ | |||
+ | ====== Start picking! ====== | ||
+ | |||
+ | Use the **__'' | ||
+ | |||
+ | ====== Change log ====== | ||
====crYOLO==== | ====crYOLO==== | ||
+ | |||
+ | **crYOLO 1.4.0:** | ||
+ | * Support Just Another Noise 2 Noise Implemnentation ([[: | ||
+ | * Add --mask_width as optional parameter for the filament mode | ||
+ | * Update tensorflow from 1.10.1 to 1.12.3 to make crYOLO compatible with JANNI | ||
+ | * Update numpy from 1.14.5 to 1.15.4 to make crYOLO compatible with JANNI | ||
+ | |||
+ | <hidden **Old crYOLO change logs**> | ||
+ | **crYOLO 1.3.6:** | ||
+ | * Changed filament search radius factor from 0.8 to 1.41 (this fixed issue 21) | ||
+ | * Add search radius factor as [[pipeline: | ||
+ | * Improved error message in case of corrupted config file | ||
+ | * Fixed issue 22: If absolute paths are used in the field “train_image” in your configuration file, filtering is skipped. | ||
+ | |||
+ | **crYOLO 1.3.5:** | ||
+ | * Fixed issue 20: During training the images are normalized separately, but during prediction is done batch wise. The lead to confusing results: some micrographs were perfectly picked, some totally unreasonable, | ||
+ | * Remove unnecessary dependencies | ||
+ | * Add %%__%%version%%__%% to %%__%%init%%__%%.py for easy access to package version. | ||
+ | |||
+ | **crYOLO 1.3.4:** | ||
+ | * Support for SPHIRE 1.2 | ||
+ | * Changed the minimum threshold for cbox files from 0.01 to 0.1. Much faster in many cases but still low enough. If -t is lower than 0.1, the new threshold is used as minimum. | ||
+ | * Installation now checks if python 3 is used. | ||
+ | * Fix issue 19: Filtering does not work if target image directory is absolute path. | ||
+ | * Fix crash when %%--%%otf was specified but filtering was not specified in the config file | ||
+ | |||
+ | **crYOLO 1.3.3:** | ||
+ | * Fix issue 18: Prediction is broken in 1.3.2. It removes all particles as it claim they are not fully immersed in the image. | ||
+ | |||
+ | **crYOLO 1.3.2:** | ||
+ | * Speedup prediction: Vectorized some parts of the code and optimized the creation of the cbox files. 30% speed up picking / 15% faster training compared to 1.3.1/ | ||
+ | * Bug fix in merging of filaments that sometimes throw " | ||
+ | * Fix in cryolo_evaluation: | ||
+ | * Change library requirement to PILLOW version 6.0.0 | ||
+ | * Fix issue 16: %%--%%gpu_fraction only works for prediction, not for training. | ||
+ | |||
+ | **crYOLO 1.3.1:** | ||
+ | * Fix Issue 15: -g was ignored when --gpu_fraction was used. | ||
+ | |||
+ | **crYOLO 1.3.0:** | ||
+ | * Fine tune the general network to your data using the new fine tune option with %%--%%fine_tune (https:// | ||
+ | * One-the-fly micrograph filtering during particle picking with %%--%%otf (don't double your dataset during picking)(https:// | ||
+ | * Interactive threshold adjustment after prediction using the new cbox-files and the crYOLO boxmanager 1.2 (https:// | ||
+ | * Pick only fully immersed particles (Issue 13) | ||
+ | * Improved filament mode | ||
+ | * Rewrote tracing | ||
+ | * Rewrote and speed up merging of filaments | ||
+ | * Fixed parallelisation of the filament mode (Issue 14) | ||
+ | * Add tifffile as dependency, as imageio throws a lot of warning for some tif files. | ||
+ | * Add conversion for uint16 images, as pillow cannot work with them. | ||
+ | * Add option %%--%%skip_augmentation to deactivate augmentation during training (Thanks to Tijmen de Wolf). (https:// | ||
+ | * Add option %%--%%num_cpu to specify the number of CPUs used during training and during prediction. (Thanks to Nikolaus Dietz) (https:// | ||
+ | * Add option to limit the amount of GPU memory reserved by crYOLO with %%--%%gpu_fraction (Thanks to Nikolaus Dietz) (https:// | ||
+ | * Save anchor size in model every time you write a new model during training (not only at the end) | ||
+ | * In case of using %%--%%min_distance, | ||
**crYOLO Version 1.2.3:** | **crYOLO Version 1.2.3:** | ||
- | * crYOLO now saves the anchors which were used to train a model inside the model and takes care that it is used during prediction. | + | * crYOLO now saves the anchors which were used during training |
- | * LineEnhancer dependency is now installed via PyPi, as --follow-dependency-links is removed in pip 19. | + | * [[https:// |
* Fix Issue 9: Removed zignor dependency as it leads to problems for some users (Thanks to Jason Kaelber) | * Fix Issue 9: Removed zignor dependency as it leads to problems for some users (Thanks to Jason Kaelber) | ||
* Attempt to fix Issue 10: Removed opencv dependency which was connected to this problem (Thanks to Shaun Rawson) | * Attempt to fix Issue 10: Removed opencv dependency which was connected to this problem (Thanks to Shaun Rawson) | ||
Line 159: | Line 338: | ||
* Unify image augmentation | * Unify image augmentation | ||
+ | </ | ||
====crYOLO Boxmanager==== | ====crYOLO Boxmanager==== | ||
+ | **crYOLO Boxmanager Version 1.2.3:** | ||
+ | * Make it compatible with current new environment | ||
+ | |||
+ | <hidden **Old crYOLO Boxmanager change logs**> | ||
+ | **crYOLO Boxmanager Version 1.2.2:** | ||
+ | * Makes sure that the correct version of MatplotLib is used. | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2.1:** | ||
+ | * Press " | ||
+ | * Fix for loading different box sets with different colors for the case that on of the box sets are cbox files. | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2: | ||
+ | * Add interactive threshold selection using cbox files | ||
+ | |||
**crYOLO Boxmanager Version 1.1.1: | **crYOLO Boxmanager Version 1.1.1: | ||
* Fix Issue 3 | * Fix Issue 3 | ||
Line 184: | Line 378: | ||
* Fix crash when cancel import boxfiles | * Fix crash when cancel import boxfiles | ||
* Fix crash with qt4 | * Fix crash with qt4 | ||
+ | </ | ||
==== General PhosaurusNet model ==== | ==== General PhosaurusNet model ==== | ||
+ | **Version 20190516: | ||
+ | * Added four more inhouse datasets | ||
+ | * Added SNRNP (Thanks to Clement Charenton) | ||
- | Same datasets as the general YOLO network model version 20181120 but with trained with PhosaurusNet. | + | <hidden Old General |
+ | **Version 20190315: | ||
+ | * Added KLH | ||
+ | * Added one inhouse dataset | ||
+ | **Version 20190218:** | ||
+ | * Added K3 apoferritin (Thanks to Shaun Rawson) | ||
+ | * Added two more inhouse datasets | ||
+ | |||
+ | **Version 20181221:** | ||
+ | * Same datasets as the general YOLO network model version 20181120 but with trained with PhosaurusNet. | ||
+ | </ | ||
==== General YOLO network model in patch mode ==== | ==== General YOLO network model in patch mode ==== | ||
+ | <hidden Old general YOLO network model in patch mode> | ||
**Version 20181120:** | **Version 20181120:** | ||
Line 240: | Line 447: | ||
* picornavirus (EMPIAR 10033) and | * picornavirus (EMPIAR 10033) and | ||
* an internal dataset. | * an internal dataset. | ||
- | + | </hidden> | |
- | + | ||
- | ====== System requirements ===== | + | |
- | crYOLO was tested on **Ubuntu 16.04.4 LTS** and **Ubuntu 18.04** with an NVIDIA Geforce 1080 / Geforce 1080Ti. | + | |
- | + | ||
- | However, it should run on **Windows** as well. | + | |
- | + | ||
- | As the GPU accelerated version of tensorflow does not support **MacOS**, crYOLO does not support it either. | + | |
- | + | ||
- | ====== Installation ====== | + | |
- | + | ||
- | crYOLO depends on CUDA Toolkit 9.0 and the cuDNN 7.1.2 library. | + | |
- | It will be automatically installed during crYOLO installation. | + | |
- | + | ||
- | **Install crYOLO!** | + | |
- | + | ||
- | The following instructions assume that pip and [[https:// | + | |
- | In case you have a old cryolo environment installed, first remove the old one with: | + | |
- | < | + | |
- | conda env remove --name cryolo | + | |
- | </ | + | |
- | After that, create a new virtual environment: | + | |
- | < | + | |
- | conda create -n cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 | + | |
- | </ | + | |
- | + | ||
- | Activate the environment: | + | |
- | < | + | |
- | source activate cryolo | + | |
- | </ | + | |
- | + | ||
- | Install crYOLO: | + | |
- | < | + | |
- | pip install numpy | + | |
- | pip install cryolo-X.Y.Z.tar.gz | + | |
- | pip install cryoloBM-X.Y.Z.tar.gz | + | |
- | </ | + | |
- | + | ||
- | That's it! | + | |
- | + | ||
- | ===== Run it on the CPU ==== | + | |
- | + | ||
- | There is also a way to run crYOLO on CPU. This is especially usefull when you would like to apply the generalized model and don't have a NVIDIA GPU. Picking with crYOLO is also quite fast on the CPU. On my local machine (Intel i9) it takes roughly 1 second per micrograph and on our low-performance notebooks (Intel i3) 4 seconds. | + | |
- | + | ||
- | After you followed the crYOLO installation instructions just replace tensorflow-gpu by tensorflow: | + | |
- | < | + | |
- | pip uninstall tensorflow-gpu | + | |
- | pip install tensorflow==1.10.1 | + | |
- | </code> | + | |
- | + | ||
- | Now crYOLO should work on the CPU as well! | + | |
- | + | ||
- | ====== Start picking! ====== | + | |
- | + | ||
- | Use the **__'' | + | |