This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
downloads:cryolo_1 [2019/02/12 13:28] twagner [crYOLO] |
downloads:cryolo_1 [2020/03/17 13:01] twagner [crYOLO Boxmanager] |
||
---|---|---|---|
Line 7: | Line 7: | ||
If you are interested in using crYOLO in a commercial context please contact [[stefan.raunser@mpi-dortmund.mpg.de|stefan.raunser@mpi-dortmund.mpg.de]] | If you are interested in using crYOLO in a commercial context please contact [[stefan.raunser@mpi-dortmund.mpg.de|stefan.raunser@mpi-dortmund.mpg.de]] | ||
- | ====== Download ====== | ||
+ | ====== Paper ====== | ||
+ | You can find more technical details in our paper: | ||
- | ---- | + | Nature Communications Biology: |
+ | [[https:// | ||
+ | Preprint: | ||
+ | [[https:// | ||
+ | |||
+ | < | ||
+ | <a href=" | ||
+ | </ | ||
+ | |||
+ | ====== Download ====== | ||
+ | |||
+ | <note important> | ||
Before downloading or using this product, make sure you **understand and accept the [[: | Before downloading or using this product, make sure you **understand and accept the [[: | ||
+ | </ | ||
- | ---- | ||
====crYOLO==== | ====crYOLO==== | ||
- | Version: 1.2.3 | ||
- | Uploaded: 12. February 2019 | + | Version: 1.5.6 |
- | [[ftp://ftp.gwdg.de/pub/ | + | Uploaded: |
+ | |||
+ | [[https://pypi.org/project/cryolo/#files|DOWNLOAD]] | ||
+ | |||
+ | Please see [[downloads: | ||
====crYOLO boxmanager==== | ====crYOLO boxmanager==== | ||
- | Version: 1.1.1 | + | Version: 1.2.9 |
- | Uploaded: | + | Uploaded: |
- | [[ftp://ftp.gwdg.de/pub/misc/sphire/ | + | [[https://pypi.org/project/cryoloBM/#files|DOWNLOAD]] |
[{{ : | [{{ : | ||
- | ==== General PhosaurusNet | + | ==== General PhosaurusNet |
- | | + | |
- | Uploaded: 21. December 2018 | + | [[pipeline:window: |
- | [[ftp:// | + | === For cryo images (low-pass filtered) === |
+ | Number of datasets: 43 real, 10 simulated, 10 particle free datasets on various grids with contamination | ||
+ | |||
+ | Uploaded: 16. March 2020 | ||
+ | |||
+ | [[ftp:// | ||
|DOWNLOAD]] | |DOWNLOAD]] | ||
- | [[pipeline:window: | + | === For cryo images (neural network denoised with JANNI) === |
+ | Number of datasets: 43 real, 10 simulated, 10 particle free datasets on various grids with contamination | ||
- | Info: Trained on cryo images, therefore negative stain will not work. | + | Uploaded: 15. Oktober 2019 |
+ | [[ftp:// | ||
+ | |DOWNLOAD]] | ||
+ | < | ||
+ | The performance of the general model based on JANNI denoised data compared to low-pass filtered data did not improve. The average AUC on the validation data was in both cases the same (0.85). But this might be because of the data selected for the general model. I assume that especially on very noisy micrographs JANNI will improve the results. | ||
+ | </ | ||
+ | |||
+ | |||
+ | === For negative stain images === | ||
+ | Number of datasets: 10 real datasets | ||
+ | |||
+ | Uploaded: 26. February 2019 | ||
+ | |||
+ | [[ftp:// | ||
+ | |DOWNLOAD]] | ||
====ARCHIVE==== | ====ARCHIVE==== | ||
Previous versions of crYOLO, the boxmanager and the general models can be found here: [[http:// | Previous versions of crYOLO, the boxmanager and the general models can be found here: [[http:// | ||
- | ====== Known issues ===== | ||
- | * Issue 0: Training on multiple GPUs sometimes lead to worse performance (higher loss). We currently recommend to train on single gpus. | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | * < | ||
- | ====== | + | ====== |
- | ====crYOLO==== | + | **System requirements: |
- | **crYOLO Version 1.2.3:** | + | crYOLO was tested on **Ubuntu 16.04.4 LTS** and **Ubuntu 18.04** with an NVIDIA Geforce 1080 / Geforce 1080Ti. |
- | * crYOLO now saves the anchors which were used to train a model inside the model and takes care that it is used during prediction. | + | However, |
- | | + | |
- | | + | |
- | | + | |
- | | + | |
- | **crYOLO Version 1.2.2:** | + | As the GPU accelerated version of tensorflow does not support |
- | * Added the PhosaurusNet to the crYOLO backend, which makes the patch mode needless for picking single particles. | + | |
- | | + | |
- | | + | |
- | | + | |
- | | + | |
- | * Fix Issue 6: crYOLO | + | |
- | * Fix Issue 7: crYOLO introduces a shift for non square images proportional to the aspect ratio. (Thanks to Shaun Rawson) | + | |
- | * Fix Issue 8: crYOLO sometimes connects two parallel filaments. The filament tracing was optimized and seems now working properly. | + | |
- | * Fix a severe bug in filament tracing. Curved filaments splitted by crYOLO in more straight sub pieces. However, during the division, one half of the splitted filament was lost. (Thanks to Sabrina Pospich) | + | |
- | * Added a wiki entry about the [[: | + | |
- | * Added a wiki entry about the [[: | + | |
- | * Added a wiki entry [[: | + | |
- | **crYOLO Version 1.2.1:** | ||
- | * Fix Issue 2: Tiff files are now written as 32 bit when internal filtering is used. | ||
- | * cryolo_evaluation now additionally estimates the optimal threshold based on the F2 score, which puts more weight on recall than on precision | ||
- | * File ending of filament box files is now .box instead of .txt (Thanks to Jesse M. Hansen) | ||
- | **crYOLO | + | crYOLO |
- | * Switch to Python3 (**Please use a fresh environment!**) | + | It will be automatically installed during |
- | * (Hopefully) fixed that crYOLO | + | |
- | * Fix that training with multiple GPUs did not speed up small datasets | + | |
- | * Low-pass filtering is now [[http:// | + | |
- | * Fix two bugs in cryolo_evaluation that lead to an underestimation the performance parameters | + | |
- | * cryolo_evaluation is now multithreaded if your training data is organised in subfolders | + | |
- | * cryolo_evaluation now contains a better method for optimal picking threshold estimation | + | |
- | * Refactoring | + | |
- | * Minor bug fixes | + | |
- | **crYOLO | + | **Install |
- | * Hot fix for filament mode when applied to non square images. | + | |
- | **crYOLO Version 1.1.3:** | + | The following instructions assume that pip and [[https:// |
- | * Improved non-maximum-suppression brings 60% speedup during picking! | + | In case you have a old cryolo environment installed, first remove the old one with: |
- | * Multi GPU support for training and prediction (e.g by adding -g 0 1 for GPU 0 and GPU 1 to the training/prediction command) | + | < |
- | * Bug fixed which leads to a crash if no particles are picked on the first micrograph (Thanks to Björn Klink). | + | conda env remove |
+ | </code> | ||
+ | After that, create | ||
+ | < | ||
+ | conda create -n cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 numpy==1.14.5 cython wxPython==4.0.4 intel-openmp==2019.4 | ||
+ | </ | ||
- | **crYOLO Version 1.1.2:** | + | Activate the environment: |
- | * STAR files could now used for training. However, as they don't contain size information the size specified in the anchors in the config.json is used. | + | < |
- | * Slightly improved speed of the filament-mode | + | source activate cryolo |
- | * Fixed another bug running filament mode on non-square images (Thanks to Gregory Alushin) | + | </ |
- | **crYOLO | + | In case you run **crYOLO |
- | * More efficient MRC reading and batch prediction leads to ~50% faster training and ~70% faster picking when crYOLO is used in patch-mode (compared with the patch-mode in 1.1.0). | + | < |
- | * 6x faster filament picking | + | pip install ' |
- | * Reading of annotation data is now super-fast | + | </ |
- | * Optimized filament picking parameters | + | |
- | * Fixed bug which made training fail for some 16 bit images | + | |
- | * Fixed bug which could lead to double picked filaments | + | |
- | * Fixed bug running filament mode on non-square images (Thanks to Gregory Alushin) | + | |
- | * Supports EMAN1 helix coordinates | + | |
- | * Support for star file format. During prediction, both box and star files are written. | + | |
- | **crYOLO | + | But if you want to run **crYOLO |
- | * crYOLO now supports filaments | + | < |
- | * New evaluation tool | + | pip install ' |
- | * Supports empty box files for training on particle-free images | + | </ |
- | * Extended data augmentation: | + | |
- | * Experimental support of periodic restarts during training (with --warm_restarts) | + | |
- | **crYOLO | + | < |
- | * Fix a problem reading backend weights from read-only filesystem (Thanks to Michael Cianfrocco and Jason Key) | + | During the installation of crYOLO |
- | * Make sure that tensorflow version is >= 1.5.0 and < 1.9.0 | + | '' |
- | * Add support for subfolders in training and validation directories | + | However, you can ignore it. It is actually also working with numpy==1.14.5 |
- | * More clear error message when the trained model does not fit to the architecture specified in the config file. | + | </ |
- | **crYOLO | + | <note tip> |
- | * Ignore non-image files during training and predction (Thanks to Kellie Woll) | + | In case you want to integrate |
- | * Fixed misleading error when non existing folder | + | </ |
- | * Add distance threshold during prediction by adding -d distanceInPixel parameter to prediction command (Thanks to Lifei Fu) | + | |
- | * Add " | + | |
- | **crYOLO Version 1.0.2:** | + | {{page> |
- | | + | **That's it!** |
- | **crYOLO Version 1.0.1:** | + | You might want to check if everything is running as expected. Here is a reference example: |
- | * Normalization technique is now the same for 8-bit and 32 bit images. | + | |
- | * Unify image augmentation | + | |
+ | [[http:// | ||
+ | |||
+ | ===== Run it on the CPU ==== | ||
+ | |||
+ | There is also a way to run crYOLO on CPU. To use it, just follow the instruction in the [[downloads: | ||
+ | |||
+ | Picking with crYOLO is also quite fast on the CPU. On my local machine (Intel i9) it takes roughly 1 second per micrograph and on our low-performance notebooks (Intel i3) 4 seconds. | ||
+ | |||
+ | Training crYOLO is much more computational expensive. Training a model with 14 micrographs from scratch on my local machine take 34 minutes per epoch on the CPU. Given that you often need 25 epochs until convergence it is a task to do overnight (~ 12 hours). However, you might want to try [[pipeline: | ||
+ | |||
+ | ====== Tutorials ====== | ||
+ | |||
+ | Use our **__'' | ||
+ | |||
+ | ====== Change log ====== | ||
+ | |||
+ | ====crYOLO==== | ||
+ | {{page> | ||
====crYOLO Boxmanager==== | ====crYOLO Boxmanager==== | ||
+ | |||
+ | **crYOLO Boxmanager 1.3.0** | ||
+ | * Add option to plot size- and confidence distribution for cbox files. | ||
+ | * Add slider to filter particles according their estimated size. | ||
+ | * Add addition field for the number of boxes with live update. | ||
+ | * Add wildcard commandline option | ||
+ | * Show progress-bar for reading and writing box-files | ||
+ | |||
+ | <hidden **Old crYOLO Boxmanager change logs**> | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2.9:** | ||
+ | * Fixed a problem that only one filament is shown | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2.8:** | ||
+ | * Add a low pass filter | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2.6:** | ||
+ | * Make it compatible with current new environment | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2.3:** | ||
+ | * Make it compatible with current new environment | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2.2:** | ||
+ | * Makes sure that the correct version of MatplotLib is used. | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2.1:** | ||
+ | * Press " | ||
+ | * Fix for loading different box sets with different colors for the case that on of the box sets are cbox files. | ||
+ | |||
+ | **crYOLO Boxmanager Version 1.2: | ||
+ | * Add interactive threshold selection using cbox files | ||
+ | |||
**crYOLO Boxmanager Version 1.1.1: | **crYOLO Boxmanager Version 1.1.1: | ||
* Fix Issue 3 | * Fix Issue 3 | ||
Line 184: | Line 219: | ||
* Fix crash when cancel import boxfiles | * Fix crash when cancel import boxfiles | ||
* Fix crash with qt4 | * Fix crash with qt4 | ||
+ | </ | ||
==== General PhosaurusNet model ==== | ==== General PhosaurusNet model ==== | ||
+ | **Version 201912:**: | ||
+ | * Added two more datasets. | ||
- | Same datasets | + | **Version 20190516: |
+ | * Added four more inhouse | ||
+ | * Added SNRNP (Thanks to Clement Charenton) | ||
- | ==== General | + | <hidden Old General |
+ | **Version 20190315: | ||
+ | * Added KLH | ||
+ | * Added one inhouse dataset | ||
+ | **Version 20190218:** | ||
+ | * Added K3 apoferritin (Thanks to Shaun Rawson) | ||
+ | * Added two more inhouse datasets | ||
+ | |||
+ | **Version 20181221:** | ||
+ | * Same datasets as the general YOLO network model version 20181120 but with trained with PhosaurusNet. | ||
+ | </ | ||
+ | ==== General YOLO network model in patch mode ==== | ||
+ | <hidden Old general YOLO network model in patch mode> | ||
**Version 20181120:** | **Version 20181120:** | ||
Line 240: | Line 291: | ||
* picornavirus (EMPIAR 10033) and | * picornavirus (EMPIAR 10033) and | ||
* an internal dataset. | * an internal dataset. | ||
- | + | </hidden> | |
- | + | ||
- | ====== System requirements ===== | + | |
- | crYOLO was tested on **Ubuntu 16.04.4 LTS** and **Ubuntu 18.04** with an NVIDIA Geforce 1080 / Geforce 1080Ti. | + | |
- | + | ||
- | However, it should run on **Windows** as well. | + | |
- | + | ||
- | As the GPU accelerated version of tensorflow does not support **MacOS**, crYOLO does not support it either. | + | |
- | + | ||
- | ====== Installation ====== | + | |
- | + | ||
- | crYOLO depends on CUDA Toolkit 9.0 and the cuDNN 7.1.2 library. | + | |
- | It will be automatically installed during crYOLO installation. | + | |
- | + | ||
- | **Install crYOLO!** | + | |
- | + | ||
- | The following instructions assume that pip and [[https:// | + | |
- | In case you have a old cryolo environment installed, first remove the old one with: | + | |
- | < | + | |
- | conda env remove --name cryolo | + | |
- | </ | + | |
- | After that, create a new virtual environment: | + | |
- | < | + | |
- | conda create -n cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 | + | |
- | </ | + | |
- | + | ||
- | Activate the environment: | + | |
- | < | + | |
- | source activate cryolo | + | |
- | </ | + | |
- | + | ||
- | Install crYOLO: | + | |
- | < | + | |
- | pip install numpy | + | |
- | pip install cryolo-X.Y.Z.tar.gz --process-dependency-links | + | |
- | pip install cryoloBM-X.Y.Z.tar.gz | + | |
- | </ | + | |
- | + | ||
- | That's it! | + | |
- | + | ||
- | ===== Run it on the CPU ==== | + | |
- | + | ||
- | There is also a way to run crYOLO on CPU. This is especially usefull when you would like to apply the generalized model and don't have a NVIDIA GPU. Picking with crYOLO is also quite fast on the CPU. On my local machine (Intel i9) it takes roughly 1 second per micrograph and on our low-performance notebooks (Intel i3) 4 seconds. | + | |
- | + | ||
- | After you followed the crYOLO installation instructions just replace tensorflow-gpu by tensorflow: | + | |
- | < | + | |
- | pip uninstall tensorflow-gpu | + | |
- | pip install tensorflow==1.10.1 | + | |
- | </code> | + | |
- | + | ||
- | Now crYOLO should work on the CPU as well! | + | |
- | + | ||
- | ====== Start picking! ====== | + | |
- | + | ||
- | Use the **__'' | + | |