User Tools

Site Tools

pipeline:isac:sxisac2

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Last revision Both sides next revision
pipeline:isac:sxisac2 [2018/06/20 13:12]
127.0.0.1 external edit 		pipeline:isac:sxisac2 [2020/08/19 16:39]
fschoenfeld
Line 1: Line 1:
 ~~NOTOC~~ ~~NOTOC~~
  
-===== sxisac2 =====+===== sp_isac2 =====
 ISAC2 - 2D Clustering: Iterative Stable Alignment and Clustering (ISAC) of a 2D image stack. ISAC2 - 2D Clustering: Iterative Stable Alignment and Clustering (ISAC) of a 2D image stack.
  
Line 9: Line 9:
 Usage in command line Usage in command line
  
-  sxisac2.py stack_file output_directory --radius=particle_radius --img_per_grp=img_per_grp --CTF --xr=xr --thld_err=thld_err --target_radius=target_radius --target_nx=target_nx --VPP --ir=ir --rs=rs --yr=yr --ts=ts --maxit=maxit --center_method=center_method --dst=dst --FL=FL --FH=FH --FF=FF --init_iter=init_iter --iter_reali=iter_reali --stab_ali=stab_ali --minimum_grp_size --rand_seed=rand_seed --skip_prealignment --restart+  sp_isac2.py stack_file output_directory --radius=particle_radius --img_per_grp=img_per_grp --CTF --filament_width=filament_width --xr=xr --thld_err=thld_err --target_radius=target_radius --target_nx=target_nx --VPP --ir=ir --rs=rs --yr=yr --ts=ts --maxit=maxit --center_method=center_method --dst=dst --FL=FL --FH=FH --FF=FF --init_iter=init_iter --iter_reali=iter_reali --stab_ali=stab_ali --minimum_grp_size --rand_seed=rand_seed --skip_prealignment --restart --filament_mask_ignore 
  
 \\ \\
 ===== Typical usage ===== ===== Typical usage =====
  
-sxisac2 exists only in MPI version.+sp_isac2 exists only in MPI version.
  
 \\ __1. Conventional TEM dataset__: \\ __1. Conventional TEM dataset__:
  
-  mpirun -np 96 sxisac2.py bdb:stack isac2_outdir --radius=120 --CTF+  mpirun -np 96 sp_isac2.py bdb:stack isac2_outdir --radius=120 --CTF
  
 Note: ISAC2 will change the size of input data such that they fit into box size 76x76 by default (see Description below). Note: ISAC2 will change the size of input data such that they fit into box size 76x76 by default (see Description below).
Line 24: Line 24:
 \\ __2. Phase Plate TEM dataset__: \\ __2. Phase Plate TEM dataset__:
  
-  mpirun -np 96 sxisac2.py bdb:stack isac2_outdir --radius=120 --VPP+  mpirun -np 96 sp_isac2.py bdb:stack isac2_outdir --radius=120 --VPP
  
 Note: CTF and VPP options cannot be used together. Note: CTF and VPP options cannot be used together.
Line 31: Line 31:
 ===== Input ===== ===== Input =====
 === Main Parameters === === Main Parameters ===
-  ; stack_file : Input image stack: The images must to be square (//nx//=//ny//). The stack can be either in bdb or hdf format. (default required string) +  ; stack_file : Input image stack: Images must to be square (//nx//=//ny//). The stack can be either in .bdb or in .hdf format. (default required string) 
-  ; output_directory : Output directory: The directory will be automatically created and the results will be written here. If the directory already exists, results will be written there, possibly overwriting previous runs. (default required string)+  ; output_directory : Output directory: General ISAC output directory to store all results. If the directory already exists ISC will only run in continuation mode (see advanced parameter restart). (default required string)
  
-  ; %%--%%radius : Particle radius [Pixels]: Radius of the particle (pixels). There is no default value and so a sensible number has to be provided. (default required int) +  ; %%--%%radius : Particle radius [Pixels] : Radius of the particle in pixels. ISAC cannot offer a default here since the value will depend on the particle im question. (default required int) 
-  ; %%--%%img_per_grp : Images per class: Number of images per class in an ideal situation. In practice, it defines the maximum size of the classes or the number of classes //K// = //N// / //img_per_grp//, where //N// is the total number of images in the input stack. (default 200) +  ; %%--%%img_per_grp : Images per class: Ideally the number of images per. In practice this value determines the number of classes //K// = //N// / //img_per_grp//, where //N// is the total number of images in the input stack. (default 200) 
-  ; %%--%%CTF : CTF phase flipping: If set, the data will be phase-flipped using CTF information included in the image headers. (default False) +  ; %%--%%CTF : CTF phase flipping : Use for cryo datasets. If set to True the data will be phase-flipped using CTF information included in the image headers. Cannot be used together with the VPP option. (default False) : %%--%%VPP==False 
-  ; %%--%%xr : Translation search range [Pixels]: The translational search range. Set by the program by default. (default 1) +  ; %%--%%VPP : Phase Plate data : Use this option if the dataset is taken with phase plate. Cannot be used togehter with the CTF option. (default False) : %%--%%CTF==False 
-  ; %%--%%thld_err : Pixel error threshold [Pixels]: Used for checking stability. It is defined as the root mean square of distances between corresponding pixels from set of found transformations and theirs average transformation, depends linearly on square of radius (parameter //ou//). //units// - //pixels//. (default 0.7) +  ; %%--%%filament_width : Filament width [Pixels]: When this is set to a non-default value ISAC assumes helical data in which case particle images will be subjected to rectangular masking of the given filament_width value. (default -1)
-  ; %%--%%target_radius : Target particle radius [Pixels]: Particle radius used by ISAC2 to process the data. The images will be resized to fit this radius (default 29) +
-  ; %%--%%target_nx : Target particle image size [Pixels]: Image size used by ISAC2 to process the data. The images will be resized according to target particle radius and then cut/padded to achieve the target image size. When //xr// > 0, the final image size for ISAC2 processing is //target_nx// + //xr// - 1  (default 76)+
  
 \\ \\
 === Advanced Parameters === === Advanced Parameters ===
-  ; %%--%%VPP : Phase Plate data: Please use this option if the dataset is taken with Phase Plate. (default False) +  ; %%--%%xr : Translation search range [Pixels] : The translational search range. Change with care; higher values will incur significantly higher processing costs. (default 1) 
-  ; %%--%%ir : Inner ring [Pixels]: Inner of the resampling to polar coordinates. (default 1) +  ; %%--%%thld_err : Pixel error threshold [Pixels]: Used as a threshold value when checking cluster stability. The pixel error is defined as the root mean square of distances between corresponding pixels from set of found transformations and their average transformation; it depends linearly on square of radius (parameter //ou//). //units// - //pixels//. (default 0.7) 
-  ; %%--%%rs : Ring step [Pixels]: Step of the resampling to polar coordinates. (default 1) +  ; %%--%%target_radius : Target particle radius [Pixels] : Particle radius used by ISAC2 to process the data. All particle images will be re-scaled to match their particle radius with this radius. (default 29) 
-  ; %%--%%yr : Y search range [Pixels]: The translational search range in the y direction. Set as //xr// by default. (default -1) +  ; %%--%%target_nx : Target particle image size [Pixels] : Image size used by ISAC2 to process the data. particle images will first be resized according to target particle radius (see above) and then cropped or padded to achieve the target image size. When //xr// > 0, the final image size for ISAC2 processing is //target_nx// + //xr// - 1  (default 76) 
-  ; %%--%%ts : Search step [Pixels]: Translational search step. (default 1.0) +  ; %%--%%ir : Inner ring [Pixels] : Radius of the inner-most ring when resampling images to polar coordinates. (default 1) 
-  ; %%--%%maxit : Reference-free alignment iterations: The number of iterations for reference-free alignment. (default 30) +  ; %%--%%rs : Ring step [Pixels] : Radius step size when resampling images to polar coordinates. (default 1) 
-  ; %%--%%center_method : Centering method: Method to center global 2D average during the initial prealignment of the data (0: no centering; -1: average shift method; please see center_2D in utilities.py for methods 1-7). (default -1) +  ; %%--%%yr : Y search range [Pixels] : The translational search range in the y direction. Set to the value of //xr// by default. (default -1) 
-  ; %%--%%dst : Discrete angle used for within-group alignment: Discrete angle used for within-group alignment. (default 90.0) +  ; %%--%%ts : Search step [Pixels] : Translational search step. (default 1.0) 
-  ; %%--%%FL : Lowest filter frequency [1/Pixel]: Lowest frequency used for the tangent filter. (default 0.2) +  ; %%--%%maxit : Reference-free alignment iterations: The number of iterations for reference-free alignments. (default 30) 
-  ; %%--%%FH : Highest filter frequency [1/Pixel]: Highest frequency used for the tangent filter. (default 0.45) +  ; %%--%%center_method : Centering method : Method to center global 2D average during the initial prealignment of the data (0: no centering; -1: average shift method; please see center_2D in utilities.py for methods 1-7). (default -1) 
-  ; %%--%%FF : Tangent filter fall-off: The fall-off of the tangent filter. (default 0.2) +  ; %%--%%dst : Discrete angle used for within-group alignment : Discrete angle used for within-group alignment. (default 90.0) 
-  ; %%--%%init_iter : Maximum generations: Maximum number of generation iterations performed for a given subset. (default 7) +  ; %%--%%FL : Lowest filter frequency [1/Pixel] : Lowest frequency used for the tangent filter. (default 0.2) 
-  ; %%--%%iter_reali : SAC stability check interval: Defines every how many iterations the SAC stability checking is performed. (default 1) +  ; %%--%%FH : Highest filter frequency [1/Pixel] : Highest frequency used for the tangent filter. (default 0.45) 
-  ; %%--%%stab_ali : Number of alignments for stability check: The number of alignments when checking stability. (default 5) +  ; %%--%%FF : Tangent filter fall-off : The fall-off of the tangent filter. (default 0.2) 
-  ; %%--%%minimum_grp_size : Minimum size of reproducible class: Minimum size of reproducible class. (default 60)+  ; %%--%%init_iter : Maximum generations : Maximum number of generation iterations performed for a given subset. (default 7) 
 +  ; %%--%%iter_reali : SAC stability check interval : Defines every how many iterations the SAC stability checking is performed. (default 1) 
 +  ; %%--%%stab_ali : Number of alignments for stability check : The number of alignment runs when checking stability. (default 5) 
 +  ; %%--%%minimum_grp_size : Minimum size of reproducible classes : Minimum size of reproducible classes. (default 60)
   ; %%--%%rand_seed : Seed: Random seed set before calculations. Useful for testing purposes. By default, ISAC2 sets a random seed number. (default none)   ; %%--%%rand_seed : Seed: Random seed set before calculations. Useful for testing purposes. By default, ISAC2 sets a random seed number. (default none)
   ; %%--%%skip_prealignment : Do pre-alignment: Indicate if pre-alignment should be used or not. Do not use pre-alignment if images are already centered. The //2dalignment// directory will still be generated but the parameters will be zero. By default, do pre-alignment. (default False question reversed in GUI)   ; %%--%%skip_prealignment : Do pre-alignment: Indicate if pre-alignment should be used or not. Do not use pre-alignment if images are already centered. The //2dalignment// directory will still be generated but the parameters will be zero. By default, do pre-alignment. (default False question reversed in GUI)
   ; %%--%%restart : Restart run: 0: Restart ISAC2 after the last completed main iteration (i.e. the directory must contain //finished// file); //k//: Restart ISAC2 after //k//-th main iteration, it has to be completed (i.e. the directory must contain //finished// file), and higer iterations will be removed; Default: Do not restart. (default -1)   ; %%--%%restart : Restart run: 0: Restart ISAC2 after the last completed main iteration (i.e. the directory must contain //finished// file); //k//: Restart ISAC2 after //k//-th main iteration, it has to be completed (i.e. the directory must contain //finished// file), and higer iterations will be removed; Default: Do not restart. (default -1)
 +  ; %%--%%filament_mask_ignore : Ignore filament masking (filament use only): ONLY RELEVANT IF parameter filament_width is set to a non-default value. When processing helical particle images rectangular masking is used (A) to normalize and (B) to mask the actual particle images. The latter can be disabled by setting this flag to True. (default True question reversed in GUI)
  
 \\ \\
 ===== Output ===== ===== Output =====
-For each generation of running the program, there are two phases. The first phase is an exploratory phase. In this phase, we set the criteria to be very loose and try to find as much candidate class averages as possible. (OBSOLETE? This phase typically should have 10 to 20 rounds). The candidate class averages are stored in class_averages_candidate_generation_n.hdf.+For each generation of running the program, there are two phases. The first phase is an exploratory phase. In this phase, we set the criteria to be very loose and try to find as much candidate class averages as possible. The candidate class averages are stored in class_averages_candidate_generation_n.hdf.
  
-The second phase is where the actual class averages are generated (OBSOLETE?!!, it typically have 3~9 iterations of matching). The first half of iterations are 2-way matching, the second half of iterations are 3-way matching, and the last iteration is 4-way matching. In the second phase, three files will be generated:+The second phase is where the actual class averages are generated. The candidate groups are tested for their stability by repeated randomly initialized reference-free alignment and those deemed stable as set aside as output and member images of those that did not pass the test are returned to the overall pool of data and processed again starting from the first phase.
  
 \\ \\
Line 90: Line 92:
   - The alignment shift parameters will be applied to the input data.   - The alignment shift parameters will be applied to the input data.
   - **IMPORTANT**: Input aligned images will be resized such that the original user-provided radius will be now target_radius and the box size target_nx + xr - 1.  The pixel size of the modified data is thus original_pixel_size * original_radius_size / target_radius.   - **IMPORTANT**: Input aligned images will be resized such that the original user-provided radius will be now target_radius and the box size target_nx + xr - 1.  The pixel size of the modified data is thus original_pixel_size * original_radius_size / target_radius.
-    * The pseudo-code for adjusting the size of the radius and the size of the images is as follows: +    * The pseudo-code for adjusting the size of the radius and the size of the images is as follows: <code> 
-    * shrink_ratio = target_radius / original_radius_size +shrink_ratio   = target_radius / original_radius_size 
-    * new_pixel_size = original_pixel_size * shrink_ratio +new_pixel_size = original_pixel_size / shrink_ratio 
-    * if shrink_ratio is different than 1: resample images using shrink_ratio +new_box_size   = original_box_size * shrink_ratio 
-    * if new_pixel_size > target_nx : cut image to be target_nx in size +IF shrink_ratio =/= 1.0: resample images using shrink_ratio 
-    * if new_pixel_size < target_nx : pad image to be target_nx in size+IF new_box_size > target_nx : CUT image to be target_nx in size 
 +IF new_box_size < target_nx : PAD image to be target_nx in size</code>
     * The target_radius and target_nx options allow the user to finely adjust the image so that it contains enough background information.     * The target_radius and target_nx options allow the user to finely adjust the image so that it contains enough background information.
   - The program will iterate through generations of ISAC2 by alternating two steps. The outcome of these two steps is in subdirectory generation_*** (stars replaced by the current generation number).   - The program will iterate through generations of ISAC2 by alternating two steps. The outcome of these two steps is in subdirectory generation_*** (stars replaced by the current generation number).
Line 108: Line 111:
 ==== Time and Memory ==== ==== Time and Memory ====
  
-Unfortunately, ISAC2 is very time- and memory-consuming. (OBSLETE?!! For example, on my cluster, it takes 15 hours to process 50,000 64x64 particles on 256 processors. Therefore, before embarking on the big dataset, it is recommended to run a test dataset (about 2,000~5,000 particles) first to get a rough idea of timing.)+ISAC2 is very time- and memory-consuming.  For example, it may take 15 hours to process 50,000 particles using 256 processors. Therefore, before embarking on the big dataset, it is recommended to run a test dataset (about 20,000 particles) first to get a rough idea of timing.
  
 \\ \\
Line 115: Line 118:
   - Delete averages whose member particles should not be included in the output.    - Delete averages whose member particles should not be included in the output. 
   - Save the selected subset under a new name,say select1.hdf   - Save the selected subset under a new name,say select1.hdf
-  - Retrieve IDs of member particles and store them in a text file ohk.txt: +  - Retrieve IDs of member particles and store them in a text file ok.txt: 
-    * $ sxprocess.py --isacselect class_averages.hdf ok.txt+    * $ sp_process.py --isacselect class_averages.hdf ok.txt
   - Create a vritual stack containng selected particles:   - Create a vritual stack containng selected particles:
-    * $ e2bdb.py bdb:data --makevstack:bdb:select1  --list=ohk.txt+    * $ e2bdb.py bdb:data --makevstack:bdb:select1  --list=ok.txt
  
 The same steps can be performed on files containing candidate class averages. The same steps can be performed on files containing candidate class averages.
Line 128: Line 131:
   - Retrieve original image numbers in the selected ISAC2 group.  The output is list3_12.txt, which will contain image numbers in the main stack (bdb:test) and thus of the tilted counterparts in the tilted stack.  First, change directory to the subdirectory of the main run that contains results of the generation 3.  Note bdb:../data is the file in the main output directory containing original (reduced size) particles.   - Retrieve original image numbers in the selected ISAC2 group.  The output is list3_12.txt, which will contain image numbers in the main stack (bdb:test) and thus of the tilted counterparts in the tilted stack.  First, change directory to the subdirectory of the main run that contains results of the generation 3.  Note bdb:../data is the file in the main output directory containing original (reduced size) particles.
     * $ cd generation_0003     * $ cd generation_0003
-    * $ sxprocess.py  bdb:../data class_averages_generation_3.hdf  list3_12.txt  --isacgroup=12  --params=originalid+    * $ sp_process.py  bdb:../data class_averages_generation_3.hdf  list3_12.txt  --isacgroup=12  --params=originalid
   - Extract the identified images from the main stack (into subdirectory RCT, has to be created):   - Extract the identified images from the main stack (into subdirectory RCT, has to be created):
     * $ e2bdb.py bdb:test  --makevstack=bdb:RCT/group3_12  --list=list3_12.txt     * $ e2bdb.py bdb:test  --makevstack=bdb:RCT/group3_12  --list=list3_12.txt
Line 134: Line 137:
     * $ e2proc2d.py --split=12 --first=12 --last=12 class_averages_generation3.hdf  group3_12.hdf     * $ e2proc2d.py --split=12 --first=12 --last=12 class_averages_generation3.hdf  group3_12.hdf
   - Align particles using the corresponding class average from ISAC2 as a template (please adjust the parameters):   - Align particles using the corresponding class average from ISAC2 as a template (please adjust the parameters):
-    * $ sxali2d.py bdb:RCT/group3_12 None --ou=28 --xr=3 --ts=1 --maxit=1  --template=group3_12.12.hdf +    * $ sp_ali2d.py bdb:RCT/group3_12 None --ou=28 --xr=3 --ts=1 --maxit=1  --template=group3_12.12.hdf 
-  - Extract the needed alignment parameters.  The order is phi,sx,sy,mirror.  sx and mirror are used to transfer to tilted images. +  - Extract the needed alignment parameters.  The order is phi,sp_,sy,mirror.  sp_ and mirror are used to transfer to tilted images. 
-    * $ sxheader.py  group3_12.12.hdf  --params=xform.align2d  --export=params_group3_12.txt+    * $ sp_header.py  group3_12.12.hdf  --params=xform.align2d  --export=params_group3_12.txt
  
 \\ \\
Line 160: Line 163:
 \\ \\
 ==== Files ==== ==== Files ====
-sparx/bin/sxisac2.py, sparx/bin/sxisac.py, sparx/bin/isac.py+sparx/bin/sp_isac2.py, sparx/bin/sp_isac.py, sparx/bin/isac.py
  
 \\ \\
 ==== See also ==== ==== See also ====
-[[pipeline:isac:sxcompute_isac_avg|sxcompute_isac_avg]], [[pipeline:isac:sxpipe_isac_substack|sxpipe isac_substack]], and [[pipeline:isac:sxisac|sxisac]]+[[pipeline:isac:sxcompute_isac_avg|sp_compute_isac_avg]], [[pipeline:isac:sxpipe_isac_substack|sp_pipe isac_substack]], and [[pipeline:isac:sxisac|sp_isac]]
  
 \\ \\
pipeline/isac/sxisac2.txt · Last modified: 2020/08/19 16:40 by fschoenfeld

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: GNU Free Documentation License 1.3
GNU Free Documentation License 1.3 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki