This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
gpu_isac [2020/06/18 12:13] fschoenfeld |
gpu_isac [2020/09/23 14:41] fschoenfeld |
||
---|---|---|---|
Line 3: | Line 3: | ||
===== Overview ===== | ===== Overview ===== | ||
- | **ISAC** (Iterative Stable Alignment and Clustering) is a 2D classification algorithm | + | **ISAC** (//Iterative Stable Alignment and Clustering//) is a **2D classification algorithm**. It sorts a given stack of cryo-EM particles into different |
- | alternating equal size k-means clustering and repeated 2D alignment routines. | + | |
< | < | ||
Line 11: | Line 10: | ||
</ | </ | ||
- | **ISAC2** is an improved version of ISAC, and the default tool to produce 2D class averages in the **[[http:// | + | ===== ISAC versions ===== |
- | **GPU ISAC** is designed to run ISAC2 on a single workstation | + | * **ISAC** is the initial version as described in the original paper. At this point this implementation is obsolete and has been replaced |
- | <note important> | + | |
- | **Beta version:** The currently available | + | |
- | </ | + | * **GPU ISAC** was developed to run ISAC2 on a single workstation by outsourcing its computationally expensive bottleneck calculations to any available GPUs, while simultaneously keeping its MPI-based CPU parallelization otherwise intact. GPU ISAC is provided as an add-on to SPHIRE that can be installed manually (see below). |
===== Download & Installation ===== | ===== Download & Installation ===== | ||
<note important> | <note important> | ||
- | | + | **Before you start**, please note the following |
- | * **SPHIRE required:** In order to use GPU ISAC, SPHIRE needs to be installed first and the following instructions assume that this is the case. You can find the SPHIRE download and installation instructions [[http:// | + | ---- |
- | //**Note:** To confirm | + | * **CUDA:** These installation instructions assume that CUDA is already installed on your system. You can confirm |
+ | |||
+ | * **SPHIRE:** In order to use GPU ISAC, SPHIRE needs to be installed. You can find the SPHIRE download and installation instructions [[http://sphire.mpg.de/ | ||
</ | </ | ||
+ | |||
+ | ---- | ||
=== Download === | === Download === | ||
- | * The GPU ISAC beta version | + | * GPU ISAC is currently |
- | * A printout of the installation notes below can be found {{: | + | |
+ | ---- | ||
=== Installation === | === Installation === | ||
+ | Before you start, make sure your SPHIRE environment is activated. | ||
+ | <hidden How to activate your SPHIRE environment:> | ||
+ | * During the SPHIRE installation, | ||
+ | * Look for your SPHIRE environment and activate it using either: < | ||
+ | </ | ||
+ | \\ | ||
+ | |||
+ | GPU ISAC comes with a handy installation script that can be used as follows: | ||
- **Extract the archive** to your chosen GPU ISAC installation folder. | - **Extract the archive** to your chosen GPU ISAC installation folder. | ||
- **Open a terminal** and navigate to your installation folder. | - **Open a terminal** and navigate to your installation folder. | ||
- | - **Untar the archive**: < | + | - **Run the installation |
- | - **Check CUDA path variables** using: | + | |
- | - < | + | |
- | - < | + | |
- | - If the path variables do **not** contain these path variables, you can add them like so: < | + | |
- | export LD_LIBRARY_PATH=/ | + | |
- | * Where '' | + | |
- | - **Compile the GPU ISAC C++/CUDA library: | + | |
- | - '' | + | |
- | - '' | + | |
- | - **Adjust sparx libraries** to work with the C++/CUDA library we just compiled: | + | |
- | - '' | + | |
- | - '' | + | |
- | - '' | + | |
- | - **Set the correct libraries and environment: | + | |
- | - '' | + | |
- | - '' | + | |
- | - < | + | |
- | - < | + | |
+ | All done! | ||
- | ===== Usage ===== | + | ===== Running GPU ISAC ===== |
An example call to use GPU ISAC looks as follows: | An example call to use GPU ISAC looks as follows: | ||
< | < | ||
- | mpirun | + | mpirun |
</ | </ | ||
- | More readable: | + | Using the following mix of both mandatory and optional parameters (see below to learn which is which): |
< | < | ||
- | mpirun | + | mpirun |
- | bdb: | + | bdb: |
+ | path/ | ||
--CTF | --CTF | ||
-–radius=160 | -–radius=160 | ||
- | --target_radius=29 | ||
- | --target_nx=76 | ||
--img_per_grp=100 | --img_per_grp=100 | ||
--minimum_grp_size=60 | --minimum_grp_size=60 | ||
- | --thld_err=0.7 | ||
- | --center_method=0 | ||
--gpu_devices=0, | --gpu_devices=0, | ||
</ | </ | ||
Line 85: | Line 77: | ||
**[ ! ] - Mandatory** parameters in the GPU ISAC call: | **[ ! ] - Mandatory** parameters in the GPU ISAC call: | ||
- | * Replace | + | * '' |
- | * Replace | + | * ''/ |
- | * Replace | + | * '' |
- | * Adjust the number in ''< | + | * '' |
+ | * ''< | ||
+ | * ''< | ||
- | **[?] - Optional** parameters | + | <hidden What GPUs do I have and what are their system id values?> |
+ | You can use '' | ||
- | * In '' | + | {{ : |
- | * Using ''< | + | |
- | * You can also use ''< | + | Above: Example output of '' |
- | * Similarly, | + | </ |
+ | \\ | ||
+ | |||
+ | **[?] - Optional** parameters recommended to be used when running GPU ISAC: | ||
+ | * Use the ''< | ||
+ | * Use ''< | ||
+ | * Use ''< | ||
< | < | ||
- | * The full list of **ISAC2 parameters** can be found [[http:// | + | * An up to date list of **all GPU ISAC parameters** can always be printed by using the '' |
+ | |||
+ | * The online documentation | ||
| | ||
* **Additional utilities** that are helpful when using any version of ISAC can be found [[http:// | * **Additional utilities** that are helpful when using any version of ISAC can be found [[http:// | ||
Line 104: | Line 107: | ||
* More information about **using ISAC for 2D classification** can also be found in the ISAC chapter of the official [[ftp:// | * More information about **using ISAC for 2D classification** can also be found in the ISAC chapter of the official [[ftp:// | ||
</ | </ | ||
+ | |||
+ | ===== Examples ===== | ||
+ | |||
+ | **EXAMPLE 01: Test run** | ||
+ | |||
+ | This example is a test run that can be used to confirm GPU ISAC was installed successfully. It is a small stack that contains 64 artificial faces and is already included in the GPU ISAC installation package. You can process it using GPU ISAC as follows: | ||
+ | |||
+ | - In your terminal, navigate to your GPU ISAC installation folder:< | ||
+ | - Run GPU ISAC:< | ||
+ | |||
+ | Note that we don't care about the quality of any produced averages here; this test is used to make sure there are no runtime issues before a more time consuming run is executed. | ||
+ | |||
+ | ---- | ||
+ | |||
+ | **EXAMPLE 02: TcdA1 toxin data** | ||
+ | |||
+ | This example uses the [[https:// | ||
+ | |||
+ | After downloading the data you'll notice that the extracted folder contains a multitude of subfolders. For the purposes of this example we are only interested in the '' | ||
+ | |||
+ | You can process this stack using GPU ISAC as follows: | ||
+ | |||
+ | - In your terminal, navigate to your GPU ISAC installation folder:< | ||
+ | - Run GPU ISAC:< | ||
+ | * Replace ''/ | ||
+ | * Optional: Replace ''< | ||
+ | |||
+ | The final averages can then be found in '' | ||
+ | |||
+ | {{: | ||
+ | Above: 95 class averages produced when processing the above data set using GPU ISAC. The particle stack contains 11,003 particles and the averages were computed within 6 minutes (Intel i9-7020X CPU and 2x GeForce GTX 1080 GPUs). | ||
+ | |||
+ | ===== Usage ===== | ||
+ | |||
+ | Next to producing high quality 2D class averages, GPU ISAC is also an excellent tool to screen your data which allows you to: | ||
+ | |||
+ | * Quickly generate **2D class averages**. | ||
+ | * Quickly identify **suitable parameters** for your data set. | ||
+ | * Quickly gauge the **quality of your data** set before spending time on more costly processing steps. | ||
+ | |||
+ | <hidden Well, " | ||
+ | Clustering cryo-EM data is a difficult problem that involves many different parameters and often it is unclear how these impact the resulting 2D class averages. In GPU ISAC the most relevant parameters to fiddle with are: | ||
+ | |||
+ | * **Class size:** The class (or cluster) size ''< | ||
+ | * **Threshold error:** The ''< | ||
+ | |||
+ | Since GPU ISAC processes small stacks of about 10,000 to 20,000 particles fairly quickly, you can try several runs with different values for ''< | ||
+ | Once you are happy with the results, you can use these parameters for a full-sized run of (GPU) ISAC. Good luck! :) | ||
+ | </ | ||
+ | \\ | ||
+ | |||
+ | ===== GPU ISAC output files ===== | ||
+ | |||
+ | GPU ISAC produces a multitude of output files that can be used to analyze the success of running the program, even while it is still ongoing. These include the following: | ||
+ | |||
+ | * **Main iteration folders:** As GPU ISAC is running, it performs multiple "main iterations" | ||
+ | * In both the main iteration folders and the base output folder you will find '' | ||
+ | * **The final averages** are stored in '' | ||
+ | |||
+ | ===== Release notes ===== | ||
+ | |||
+ | **GPU ISAC limitations** | ||
+ | |||
+ | * The current develpoment goal of GPU ISAC is to run as fast as possible on a single machine. Because of this priority, **GPU ISAC does not yet run on multiple nodes**. This is planned to change as soon as the currently known bottlenecks have all been converted to run on the available GPUs. | ||
+ | |||
+ | ------ | ||
+ | |||
+ | **Known issues** | ||
+ | |||
+ | * In some cases **when using CUDA version 11, GPU ISAC receives a kill signal interrupt**. We're investigating the issue but recommend to use a lower version (confirmed working when using CUDA 9 and 10) until it is resolved. You can use '' | ||
+ | |||
+ | ------ | ||
+ | |||
+ | **GPU ISAC v2.3.1 & v2.3.2 (hotfix releases)** | ||
+ | |||
+ | * Changed data handling, which results in a **massive reduction in overall memory usage** and an **increased pre-alignment performance**. | ||
+ | * Fixed use of '' | ||
+ | * Fixed error in the pre-alignment progress bar that made it seem as if it did not run to completion. | ||
+ | * Minimum class size is now 60% of the full class size, if no minimum class size was specified by the user. | ||
+ | |||
+ | **GPU ISAC v2.3** | ||
+ | |||
+ | * Updated GPU ISAC install package for version 2.3. | ||
+ | * Fixed multiple issues occurring when handling larger data sets (200k and upwards). | ||
+ | * Added memory checks to the output to document GPU ISAC memory use. | ||
+ | * Batched input read, part I: Input is now read in batches with pre-processing already applied during reading. This means that we can now deal with data sets that do not fit into system RAM (as long as the compressed data does). | ||
+ | * Batched input read, part II: Input reading and processing is spread across all MPI processes, while actual processing (pre-alignment) is spread only across GPU processes. This ensures data is processed using all CPU & GPU resources available to the used machine. | ||
+ | * Multiple bug fixes for increased stability and functionality. | ||
+ | * Cleanup of old code for increased readability and maintainability. | ||
+ | |||
+ | **GPU ISAC v2.2** | ||
+ | |||
+ | * Created an easy installation package for GPU ISAC including a quick test to confirm a successful installation. | ||
+ | * Added more sanity checks to catch invalid parameter combinations and abort the run immediately. | ||
+ | * Updated the transformation stack in GPU alignment functions for higher quality averages. | ||
+ | * Multiple bug fixes for increased stability and functionality. | ||
+ | |||
+ | **GPU ISAC " | ||
+ | |||
+ | * Initially released beta version of GPU ISAC. | ||
+ | |||
+ |