gpu_isac

This version (2020/06/30 16:06) was approved by mstabrin.The Previously approved version (2020/06/18 12:35) is available.Diff

ISAC (Iterative Stable Alignment and Clustering) is a 2D classification algorithm to sort cryo-EM particles into classes depicting the same view of a target protein. It is based around iterations of alternating equal size k-means clustering and repeated 2D alignment routines.

You can find the details of the ISAC algorithm in this paper. To cite ISAC, use the following: Yang, Z., Fang, J., Chittuluru, J., Asturias, F. J. and Penczek, P. A. (2012) Iterative stable alignment and clustering of 2D transmission electron microscope images. Structure 20, 237–247.

ISAC2 is an improved version of ISAC, and the default tool to produce 2D class averages in the SPHIRE (git) software package and the TranSPHIRE automated pipeline for processing cryo-EM data. ISAC2 is a CPU-only implementation that is usually run on a computer cluster.

GPU ISAC is designed to run ISAC2 on a single workstation by outsourcing its computationally expensive calculations to any available GPUs.

Beta version: The currently available version of GPU ISAC is a beta version. A significantly improved and significantly faster version is being tested right now and will be released shortly.
  • CUDA required: The following installation instructions assume that CUDA is already installed on your system.
  • SPHIRE required: In order to use GPU ISAC, SPHIRE needs to be installed first and the following instructions assume that this is the case. You can find the SPHIRE download and installation instructions here.

Note: To confirm a working version of SPHIRE 1.3 (or later), use which sphire in your terminal. It should return to you the path to your SPHIRE installation.

Download

  • The GPU ISAC beta version is currently distributed as a .tar file that can be found here.
  • A printout of the installation notes below can be found here.

Installation

  1. Extract the archive to your chosen GPU ISAC installation folder.
  2. Open a terminal and navigate to your installation folder.
  3. Untar the archive:
    tar -xf GPU_ISAC_CHIMERA.tar
  4. Check CUDA path variables using:
    1. echo $PATH

      Confirm your PATH variable contains the path to your cuda/bin folder.

    2. echo $LD_LIBRARY_PATH

      Confirm your LD_LIBRARY_PATH variable contain the path to your cuda/lib64 folder.

    3. If the path variables do not contain these path variables, you can add them like so:
      export PATH=/path/to/cuda/bin:${PATH}
      export LD_LIBRARY_PATH=/path/to/cuda/lib64:${LD_LIBRARY_PATH}

      Where path/to/cuda/bin and path/to/cuda/lib64 need to be replaced with the real paths to the respective folders. If you do not know where to find them, by default they should be located in /usr/local/cuda.

  5. Compile the GPU ISAC C++/CUDA library:
    1. cd vChimera/cuda

      Note: This assumes you did not change directories after unpacking the .tar archive.

    2. nvcc gpu_aln_common.cu gpu_aln_noref.cu -o gpu_aln_pack.so -shared -Xcompiler -fPIC -lcufft -std=c++11
  6. Adjust sparx libraries to work with the C++/CUDA library we just compiled:
    1. cd ../eman2/sparx/libpy
    2. sed -i.bkp "s|/home/schoenf/work/code/cuISAC/cuda|$(realpath ../../../cuda)|g" applications.py
    3. sed -i.bkp2 's|statistics.sum_oe( data, "a", CTF, EMData(), myid=myid|statistics.sum_oe( data, "a", CTF, EMData()|g' applications.py
  7. Set the correct libraries and environment:
    1. cd ../bin

      Note: We are now in the /eman2/sparx/bin folder of your GPU ISAC installation folder.

    2. ln -rs ../libpy/* .

      Note: Don't forget the dot at the end!

    3. sed -i.bkp "s|/home/schoenf/applications/sphire/v1.1/envs/sphire_1.3/bin|$(dirname $(which sphire))|g" sxisac2_gpu.py
    4. sed -i.bkp2 "s/^\(.*options, args.*\)$/\1\n    os.environ['CUDA_VISIBLE_DEVICES'] = options.gpu_devices\n    options.gpu_devices = ','.join(map(str, range(len(options.gpu_devices.split(',')))))/g" sxisac2_gpu.py

An example call to use GPU ISAC looks as follows:

mpirun -np 6 /path/to/sxisac2_gpu.py bdb:path/to/stack out_dir --CTF -–radius=160 --target_radius=29 --target_nx=76 --img_per_grp=100 --minimum_grp_size=60 --thld_err=0.7 --center_method=0 --gpu_devices=0,1

More readable:

mpirun -np 6 /path/to/sxisac2_gpu.py
bdb:path/to/stack out_dir
--CTF 
-–radius=160
--target_radius=29
--target_nx=76
--img_per_grp=100
--minimum_grp_size=60
--thld_err=0.7
--center_method=0
--gpu_devices=0,1

[ ! ] - Mandatory parameters in the GPU ISAC call:

  • Replace /path/to/sxisac2_gpu.py with the path to your sxisac2_gpu.py file.
  • Replace path/to/stack with the path to your input .bdb stack. If you are using an .hdf stack, you need to remove the bdb: prefix.
  • Replace out_dir with the path to your preferred output directory.
  • Adjust the number in --radius=160 to the radius of your particle (in pixels).

[?] - Optional parameters in the GPU ISAC call:

  • In mpirun –np 6 the number can be set to the number of your CPU processors (e.g., if you have a quad core CPU, you would use 4 here).
  • Using --gpu_devices you can set what GPUs to use. This example uses two GPUs with id values 0 and 1, respectively. You can check the id values of your available GPUs by executing nvidia-smi in your terminal (GPUs are sorted by capability, with 0 being your strongest GPU).
  • You can also use --img_per_grp to limit the maximum size of individual classes. Empirically, a class size of 100-200 (30-50 for negative stain) particles has been proven successful when dealing with around 100,000 particles.
  • Similarly, you can also use --minimum_grp_size to limit the minimum size of individual classes. In general, this value should be around 50-60% of your maximum class size.
  • The full list of ISAC2 parameters can be found here.
  • Additional utilities that are helpful when using any version of ISAC can be found here.
  • More information about using ISAC for 2D classification can also be found in the ISAC chapter of the official SPHIRE tutorial (link to .pdf file).
  • gpu_isac.txt
  • Last modified: 2020/06/29 14:22
  • by mstabrin