User Tools

Site Tools


downloads:cryolo_1

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
downloads:cryolo_1 [2019/02/13 16:23]
twagner [Known issues]
downloads:cryolo_1 [2019/12/09 15:39]
twagner [General PhosaurusNet model]
Line 7: Line 7:
  
 If you are interested in using crYOLO in a commercial context please contact [[stefan.raunser@mpi-dortmund.mpg.de|stefan.raunser@mpi-dortmund.mpg.de]] If you are interested in using crYOLO in a commercial context please contact [[stefan.raunser@mpi-dortmund.mpg.de|stefan.raunser@mpi-dortmund.mpg.de]]
-====== Download ====== 
  
 +====== Paper ======
  
 +You can find more technical details in our paper:
  
-----+Nature Communications Biology: 
 +[[https://doi.org/10.1038/s42003-019-0437-z|SPHIRE-crYOLO is a fast and accurate fully automated particle picker for cryo-EM]]
  
 +Preprint:
 +[[https://www.biorxiv.org/content/10.1101/356584v2|SPHIRE-crYOLO: A fast and accurate fully automated particle picker for cryo-EM]]
 +
 +<html>
 +<a href="https://f1000.com/prime/733517098?bd=1" target="_blank"><img src="https://s3.amazonaws.com/cdn.f1000.com/images/badges/badgef1000.gif" alt="Access the recommendation on F1000Prime" id="bg" /></a>
 +</html>
 +
 +====== Download ======
 +
 +<note important>
 Before downloading or using this product, make sure you **understand and accept the [[:cryolo_license|terms of the license]]**.  Before downloading or using this product, make sure you **understand and accept the [[:cryolo_license|terms of the license]]**. 
 +</note>
  
----- 
  
  
 ====crYOLO==== ====crYOLO====
-Version: 1.2.3 
  
-Uploaded 12February 2019+Version1.5.5
  
-[[ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_V1_2_3/cryolo-1.2.3.tar.gz|DOWNLOAD]]+Uploaded:  03. December 2019 
 + 
 +[[https://pypi.org/project/cryolo/#files|DOWNLOAD]] 
 + 
 +Please see [[downloads:cryolo_1#installation|install instructions]] how to get it running on the CPU.
  
 ====crYOLO boxmanager==== ====crYOLO boxmanager====
  
-Version: 1.1.1+Version: 1.2.9
  
-Uploaded: 21. December 2018+Uploaded: 09. December 2019
  
-[[ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO_BM_V1_1_1/cryoloBM-1.1.1.tar.gz|DOWNLOAD]]+[[https://pypi.org/project/cryoloBM/#files|DOWNLOAD]]
  
 [{{ :downloads:cryolophosaurusdb.jpg?150|**crYOLO Phosauraus**Net's eponym}}] [{{ :downloads:cryolophosaurusdb.jpg?150|**crYOLO Phosauraus**Net's eponym}}]
  
-==== General PhosaurusNet model ==== +==== General PhosaurusNet models ====
- Number of datasets: 27 (real), 10 simulated, 10 particle free datasets on various grids with contaminations+
  
-Uploaded21. December 2018+[[pipeline:window:cryolo:picking_general|Please see the tutorial how to use the general models]]
  
-[[ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO-GENERAL-MODELS/gmodel_phosnet_20181221_loss0037.h5+=== For cryo images (low-pass filtered) === 
 +Number of datasets: 43 real, 10 simulated, 10 particle free datasets on various grids with contamination 
 + 
 +Uploaded: 15. Oktober 2019 
 + 
 +[[ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO-GENERAL-MODELS/gmodel_phosnet_201912_N63.h5
 |DOWNLOAD]]  |DOWNLOAD]] 
  
-[[pipeline:window:cryolo#configuration1 |Valid configuration file]]+=== For cryo images (neural network denoised with JANNI) === 
 +Number of datasets43 real, 10 simulated, 10 particle free datasets on various grids with contamination
  
-InfoTrained on cryo images, therefore negative stain will not work.+Uploaded15Oktober 2019
  
 +[[ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO-GENERAL-MODELS/gmodel_phosnet_201912_nn_N63.h5
 +|DOWNLOAD]] 
  
 +<note>
 +The performance of the general model based on JANNI denoised data compared to low-pass filtered data did not improve. The average AUC on the validation data was in both cases the same (0.85). But this might be because of the data selected for the general model. I assume that especially on very noisy micrographs JANNI will improve the results.
 +</note>
  
 +
 +
 +=== For negative stain images ===
 +Number of datasets: 10 real datasets
 +
 +Uploaded: 26. February 2019
 +
 +[[ftp://ftp.gwdg.de/pub/misc/sphire/crYOLO-GENERAL-MODELS/gmodel_phosnet_negstain_20190226.h5
 +|DOWNLOAD]] 
 ====ARCHIVE==== ====ARCHIVE====
  
 Previous versions of crYOLO, the boxmanager and the general models can be found here: [[http://sphire.mpg.de/wiki/doku.php?id=cryolo_archive#cryolo|Archive]]. Previous versions of crYOLO, the boxmanager and the general models can be found here: [[http://sphire.mpg.de/wiki/doku.php?id=cryolo_archive#cryolo|Archive]].
  
-====== Known issues =====+ 
 +====== Installation =====
 + 
 +**System requirements:** 
 + 
 +crYOLO was tested on **Ubuntu 16.04.4 LTS** and **Ubuntu 18.04** with an NVIDIA Geforce 1080 / Geforce 1080Ti. 
 + 
 +However, it should run on **Windows** as well. 
 + 
 +As the GPU accelerated version of tensorflow does not support **MacOS**, crYOLO does not support it either. 
 + 
 + 
 +crYOLO depends on CUDA Toolkit 9.0 and the cuDNN 7.1.2 library.  
 +It will be automatically installed during crYOLO installation. 
 + 
 +**Install crYOLO!** 
 + 
 +The following instructions assume that pip and [[https://conda.io/projects/conda/en/latest/user-guide/install/index.html|anaconda]] or [[https://docs.conda.io/en/latest/miniconda.html|miniconda]] are available. 
 +In case you have a old cryolo environment installed, first remove the old one with: 
 +<code> 
 +conda env remove --name cryolo 
 +</code> 
 +After that, create a new virtual environment: 
 +<code> 
 +conda create -n cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 numpy==1.14.5 cython wxPython==4.0.4 intel-openmp==2019.4 
 +</code> 
 + 
 +Activate the environment: 
 +<code> 
 +source activate cryolo 
 +</code> 
 + 
 +In case you run **crYOLO on a GPU** run: 
 +<code> 
 +pip install 'cryolo[gpu]'  
 +</code> 
 + 
 +But if you want to run **crYOLO on a CPU** run: 
 +<code> 
 +pip install 'cryolo[cpu]'  
 +</code> 
 + 
 +<note> 
 +During the installation of crYOLO you will see the following error message: 
 +''ERROR: imagecodecs-lite 2019.2.22 has requirement numpy>=1.15.4, but you'll have numpy 1.14.5 which is incompatible.'' 
 +However, you can ignore it. It is actually also working with numpy==1.14.5 
 +</note> 
 + 
 +<note tip> 
 +In case you want to integrate crYOLO in your module system, [[pipeline:window:cryolo:integration_into_module_system|here is a brief explanation]]. 
 +</note> 
 + 
 + 
 +<note important> 
 +**Known issues** 
   * Issue 0: Training on multiple GPUs sometimes lead to worse performance (higher loss). We currently recommend to train on single gpus.   * Issue 0: Training on multiple GPUs sometimes lead to worse performance (higher loss). We currently recommend to train on single gpus.
 +  * Issue 17: On the fly filtering (%%--%%otf) is slower than using it not, as the filtering is not parallelized in this case. 
 +
 +
 +
 +<hidden Closed issues>
   * <del>  Issue 1: crYOLO sometimes not exit properly after training finished. Has to be killed manually.</del>   * <del>  Issue 1: crYOLO sometimes not exit properly after training finished. Has to be killed manually.</del>
   * <del>Issue 2: If you use automatic filtering with .tif files, you get an error like "OSError: cannot identify image file 'filtered_folder/another_folder/my_image.tif'". It will be fixed in the next release.</del>   * <del>Issue 2: If you use automatic filtering with .tif files, you get an error like "OSError: cannot identify image file 'filtered_folder/another_folder/my_image.tif'". It will be fixed in the next release.</del>
Line 66: Line 160:
   * <del>Issue 10: On machines with many cores (e.g 64) an error during filtering might pop up: "[ERROR:0] 53: Can't spawn new thread"</del>   * <del>Issue 10: On machines with many cores (e.g 64) an error during filtering might pop up: "[ERROR:0] 53: Can't spawn new thread"</del>
   * <del>Issue 11: If the -g parameter is not provided, crYOLO will use the memory of all GPUs. Will be fixed in 1.2.3.</del>   * <del>Issue 11: If the -g parameter is not provided, crYOLO will use the memory of all GPUs. Will be fixed in 1.2.3.</del>
-  * Issue 12: The LineEnhancer depdenceny of crYOLO is still dependent from opencv. Workaround: In the crYOLO environment: conda install opencv+  * <del>Issue 12: The LineEnhancer depdenceny of crYOLO is still dependent from opencv. Workaround: In the crYOLO environment: conda install opencv</del> 
 +  * <del>Issue 13: After picking it can happen that some of the boxes are not fully immersed in the image. Will be fixed in 1.2.4.</del> 
 +  * <del>Issue 14: Parallelization in filament mode is broken. Will be fixed in 1.2.4.</del> 
 +  * <del>Issue 15: If the %%--%%gpu_fraction is used, crYOLO always uses GPU 0. Will be fixed in 1.3.1.</del>  
 +  * <del>Issue 16: %%--%%gpu_fraction only works for prediction, not for training. Will be fixed in 1.3.2.</del> 
 +  * <del>Issue 18: Prediction is broken in 1.3.2. It removes all particles as it claim they are not fully immersed in the image.</del> 
 +  * <del>Issue 19: Filtering does not work if target image directory is absolute path.</del> 
 +  * <del>Issue 20: crYOLO 1.3.4 has a normalization bug. During training the images are normalized seperately, but during prediction is done batch wise. Workaround: Use -pbs 1 during prediction. It will be fixed in 1.3.5.</del> 
 +  * <del>Issue 21: The search range for filament tracing is too low for many datasets. To check if you are affected: Use your trained model and pick without the filament options. Check if your filaments a nicely picked (many consecutive boxes on a filament). In the next version, the search range will be increased and added as an optional parameter.</del> 
 +  * <del>Issue 22: If absolute paths are used in the field "train_image" in your configuration file, filtering is skipped.</del> 
 +  * <del>Issue 23: Since crYOLO 1.4.0 it sometimes take long until it starts picking. The reason seems to be the tensorflow update.<del> 
 +  * <del>Issue 24: Fine-tune mode does not start (cannot find layer model_3). Will be fixed in 1.4.1.<del> 
 +  * <del>Issue 25: When using GUI, prediction behaves differently than using command line. The reason is, that it uses a different multiprocessing start method. Will be fixed with 1.5.1</del> 
 +  * <del>Issue 26: If you select filtering "None" crYOLO does not train properly.</del> 
 +</hidden> 
 +</note>
  
-====== Change log =====+**That's it!**
  
-====crYOLO====+You might want to check if everything is running as expected. Here is a reference example:
  
-**crYOLO Version 1.2.3:**+[[http://sphire.mpg.de/wiki/doku.php?id=cryolo_reference_example#reference_setup|Reference example with TcdA1]]
  
-  * crYOLO now saves the anchors which were used during training inside the .h5 file and takes care that the correct anchors are used during prediction. +===== Run it on the CPU ====
-  * [[https://pypi.org/project/lineenhancer|LineEnhancer]] dependency is now installed via PyPi, as --follow-dependency-links is removed in pip 19. +
-  * Fix Issue 9: Removed zignor dependency as it leads to problems for some users (Thanks to Jason Kaelber) +
-  * Attempt to fix Issue 10: Removed opencv dependency which was connected to this problem (Thanks to Shaun Rawson) +
-  * Fix issue 11: crYOLO uses now GPU 0 by default if not specified otherwise (e.g. by CUDA_VISIBLE_DEVICES)+
  
-**crYOLO Version 1.2.2:** +There is also a way to run crYOLO on CPUTo use it, just follow the instruction in the [[downloads:cryolo_1#installation|install section]]. This is especially useful when you would like to apply the generalized model and don't have a NVIDIA GPU
-  * Added the PhosaurusNet to the crYOLO backend, which makes the patch mode needless for picking single particles. +
-  * crYOLO now outputs separate folders for EMAN box files and STAR files +
-  * When picking filaments it will now additionally output EMAN Start-End and STAR Start-End coordinates (Thanks to Jesse M. Hansen). +
-  * Fix Issue 4: The filament mode will crash if crYOLO cannot identify a single particle in the image. +
-  * Fix Issue 5: If movies were aligned with cisTEM and picked with crYOLO, the box positions were vertically flipped. (Thanks to Wei-Chun Kao) +
-  * Fix Issue 6: crYOLO overwrote the CUDA_VISIBLE_DEVICES variable if the -g parameter is not passed. (Thanks to Shaun Rawson) +
-  * Fix Issue 7: crYOLO introduces a shift for non square images proportional to the aspect ratio. (Thanks to Shaun Rawson) +
-  * Fix Issue 8: crYOLO sometimes connects two parallel filaments. The filament tracing was optimized and seems now  working properly.  +
-  * Fix a severe bug in filament tracing. Curved filaments splitted by crYOLO in more straight sub pieces. However, during the division, one half of the splitted filament was lost. (Thanks to Sabrina Pospich) +
-  * Added a wiki entry about the [[:cryolo_nets|networks which are supported by crYOLO]] +
-  * Added a wiki entry about the [[:cryolo_config|crYOLO configuration file]] +
-  * Added a wiki entry [[:cryolo_filament_import_relion|how to import crYOLO filament coordinates (from the ''EMAN_HELIX_SEGMENTED'' folder) into Relion]].+
  
-**crYOLO Version 1.2.1:** +Picking with crYOLO is also quite fast on the CPUOn my local machine (Intel i9) it takes roughly 1 second per micrograph and on our low-performance notebooks (Intel i3) 4 seconds
-  * Fix Issue 2: Tiff files are now written as 32 bit when internal filtering is used. +
-  * cryolo_evaluation now additionally estimates the optimal threshold based on the F2 score, which puts more weight on recall than on precision +
-  * File ending of filament box files is now .box instead of .txt (Thanks to Jesse MHansen)+
  
-**crYOLO Version 1.2.0:** +Training crYOLO is much more computational expensiveTraining a model with 14 micrographs from scratch on my local machine take 34 minutes per epoch on the CPUGiven that you often need 25 epochs until convergence it is a task to do overnight (~ 12 hours). However, you might want to try [[pipeline:window:cryolo##picking_particles_-_using_the_general_model_refined_for_your_data|refining the general model]], which takes 12 minutes per epoch (~ 5 hours).
-  * Switch to Python3 (**Please use a fresh environment!**) +
-  * (Hopefully) fixed that crYOLO sometimes freezes during/after training (hard to reproduce, so I'm not 100% sure if it is fixed.) +
-  * Fix that training with multiple GPUs did not speed up small datasets +
-  * Low-pass filtering is now [[http://sphire.mpg.de/wiki/doku.php?id=pipeline:window:cryolo#picking_-_using_a_model_trained_for_your_data|integrated]] into crYOLO +
-  * Fix two bugs in cryolo_evaluation that lead to an underestimation the performance parameters +
-  * cryolo_evaluation is now multithreaded if your training data is organised in subfolders +
-  * cryolo_evaluation now contains a better method for optimal picking threshold estimation +
-  * Refactoring +
-  * Minor bug fixes+
  
-**crYOLO Version 1.1.4:** +====== Tutorials ======
-  * Hot fix for filament mode when applied to non square images. +
  
-**crYOLO Version 1.1.3:** +Use our **__''[[pipeline:window:cryolo|step-by-step tutorials]]''__** to get started!
-  * Improved non-maximum-suppression brings 60% speedup during picking! +
-  * Multi GPU support for training and prediction (e.g by adding -g 0 1 for GPU 0 and GPU 1 to the training/prediction command) +
-  Bug fixed which leads to a crash if no particles are picked on the first micrograph (Thanks to Björn Klink).+
  
-**crYOLO Version 1.1.2:** +====== Change log ======
-  * STAR files could now used for training. However, as they don't contain size information the size specified in the anchors in the config.json is used. +
-  * Slightly improved speed of the filament-mode +
-  * Fixed another bug running filament mode on non-square images (Thanks to Gregory Alushin)+
  
-**crYOLO Version 1.1.1:** +====crYOLO==== 
-  * More efficient MRC reading and batch prediction leads to ~50% faster training and ~70% faster picking when crYOLO is used in patch-mode (compared with the patch-mode in 1.1.0). +{{page>pipeline:window:cryolo:changelog}} 
-  * 6x faster filament picking +====crYOLO Boxmanager====
-  * Reading of annotation data is now super-fast :-) (Box filename has to be contained into image filename) +
-  * Optimized filament picking parameters +
-  * Fixed bug which made training fail for some 16 bit images +
-  * Fixed bug which could lead to double picked filaments +
-  * Fixed bug running filament mode on non-square images (Thanks to Gregory Alushin) +
-  * Supports EMAN1 helix coordinates +
-  * Support for star file format. During prediction, both box and star files are written.+
  
-**crYOLO Version 1.1.0:** +**crYOLO Boxmanager Version 1.2.9:** 
-  * crYOLO now supports filaments +  * Fixed a problem that only one filament is shown
-  * New evaluation tool +
-  * Supports empty box files for training on particle-free images +
-  * Extended data augmentation: Horizontal flip and flip along both axes +
-  * Experimental support of periodic restarts during training (with --warm_restarts)+
  
-**crYOLO Version 1.0.4:** +**crYOLO Boxmanager Version 1.2.8:** 
-  * Fix problem reading backend weights from read-only filesystem (Thanks to Michael Cianfrocco and Jason Key)  +  * Add low pass filter
-  * Make sure that tensorflow version is >= 1.5.0 and < 1.9.0 +
-  * Add support for subfolders in training and validation directories +
-  * More clear error message when the trained model does not fit to the architecture specified in the config file.+
  
-**crYOLO Version 1.0.3:** +<hidden **Old crYOLO Boxmanager change logs**>
-  * Ignore non-image files during training and predction (Thanks to Kellie Woll)  +
-  * Fixed misleading error when non existing folder is used as input for prediction (Thanks to Kellie Woll) +
-  * Add distance threshold during prediction by adding -d distanceInPixel parameter to prediction command (Thanks to Lifei Fu) +
-  * Add "--write_empty" parameter to prediction command if an empty box file should be written if no particle is picked.+
  
-**crYOLO Version 1.0.2:** +**crYOLO Boxmanager Version 1.2.6:** 
-  * Fix problem when mrc image has dimensions (1,width,height) (Thanks to Reza Behrouzi) +  * Make it compatible with current new environment
  
-**crYOLO Version 1.0.1:** +**crYOLO Boxmanager Version 1.2.3:** 
-  * Normalization technique is now the same for 8-bit and 32 bit images+  * Make it compatible with current new environment 
-  * Unify image augmentation+ 
 +**crYOLO Boxmanager Version 1.2.2:** 
 +  * Makes sure that the correct version of MatplotLib is used. 
 + 
 +**crYOLO Boxmanager Version 1.2.1:** 
 +   * Press "h" for hiding the boxes 
 +   * Fix for loading different box sets with different colors for the case that on of the box sets are cbox files. 
 + 
 +**crYOLO Boxmanager Version 1.2:**  
 +  * Add interactive threshold selection using cbox files
  
-====crYOLO Boxmanager==== 
 **crYOLO Boxmanager Version 1.1.1:**  **crYOLO Boxmanager Version 1.1.1:** 
   * Fix Issue 3   * Fix Issue 3
Line 185: Line 249:
   * Fix crash when cancel import boxfiles   * Fix crash when cancel import boxfiles
   * Fix crash with qt4   * Fix crash with qt4
 +</hidden>
 ==== General PhosaurusNet model ==== ==== General PhosaurusNet model ====
 +**Version 201912:**:
 +  * Added two more datasets.
  
-Same datasets as the general YOLO network model version 20181120 but with trained with PhosaurusNet.+**Version 20190516:**: 
 +  * Added four more inhouse datasets 
 +  * Added SNRNP (Thanks to Clement Charenton) 
  
-==== General YOLO network model in patch mode ====+<hidden Old General PhosaurusNet model change logs> 
 +**Version 20190315:**: 
 +  * Added KLH 
 +  * Added one inhouse dataset
  
 +**Version 20190218:**
 +  * Added K3 apoferritin (Thanks to Shaun Rawson)
 +  * Added two more inhouse datasets 
 +
 +**Version 20181221:**
 +  * Same datasets as the general YOLO network model version 20181120 but with trained with PhosaurusNet.
 +</hidden>
 +==== General YOLO network model in patch mode ====
 +<hidden Old general YOLO network model in patch mode>
 **Version 20181120:** **Version 20181120:**
  
Line 241: Line 321:
   * picornavirus (EMPIAR 10033) and    * picornavirus (EMPIAR 10033) and 
   * an internal dataset.    * an internal dataset. 
- +</hidden>
- +
-====== System requirements ===== +
-crYOLO was tested on **Ubuntu 16.04.4 LTS** and **Ubuntu 18.04** with an NVIDIA Geforce 1080 / Geforce 1080Ti. +
- +
-However, it should run on **Windows** as well. +
- +
-As the GPU accelerated version of tensorflow does not support **MacOS**, crYOLO does not support it either. +
- +
-====== Installation ====== +
- +
-crYOLO depends on CUDA Toolkit 9.0 and the cuDNN 7.1.2 library.  +
-It will be automatically installed during crYOLO installation. +
- +
-**Install crYOLO!** +
- +
-The following instructions assume that pip and [[https://conda.io/docs/user-guide/install/index.html|anaconda]] or [[https://conda.io/miniconda.html|miniconda]] are available. +
-In case you have a old cryolo environment installed, first remove the old one with: +
-<code> +
-conda env remove --name cryolo +
-</code> +
-After that, create a new virtual environment: +
-<code> +
-conda create -n cryolo -c anaconda python=3.6 pyqt=5 cudnn=7.1.2 +
-</code> +
- +
-Activate the environment: +
-<code> +
-source activate cryolo +
-</code> +
- +
-Install crYOLO: +
-<code> +
-pip install numpy +
-pip install cryolo-X.Y.Z.tar.gz  +
-pip install cryoloBM-X.Y.Z.tar.gz +
-</code> +
- +
-That's it! +
- +
-===== Run it on the CPU ==== +
- +
-There is also a way to run crYOLO on CPU. This is especially usefull when you would like to apply the generalized model and don't have a NVIDIA GPU. Picking with crYOLO is also quite fast on the CPU. On my local machine (Intel i9) it takes roughly 1 second per micrograph and on our low-performance notebooks (Intel i3) 4 seconds.  +
- +
-After you followed the crYOLO installation instructions just replace tensorflow-gpu by tensorflow: +
-<code> +
-pip uninstall tensorflow-gpu +
-pip install tensorflow==1.10.1 +
-</code> +
- +
-Now crYOLO should work on the CPU as well! +
- +
-====== Start picking! ====== +
- +
-Use the **__''[[pipeline:window:cryolo|step-by-step tutorial]]''__** to get started!+
  
downloads/cryolo_1.txt · Last modified: 2021/02/19 09:43 by twagner