User Tools

Site Tools

pipeline:sort3d:sxrsort3d

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
pipeline:sort3d:sxrsort3d [2018/02/16 10:11]
127.0.0.1 external edit 		pipeline:sort3d:sxrsort3d [2018/02/23 18:12]
moriya
Line 1: Line 1:
 ~~NOTOC~~ ~~NOTOC~~
  
-===== sxsort3d ===== +===== sxrsort3d ===== 
-3D Clustering - SORT3D DEPTH: Sorting heterogeneous 3D dataset by checking the reproducible members of two independent runs of K-means clustering with minimum group size constraint. Sorting requires the 3D reconstruction parameters have been determined already.+3D Clustering - RSORT3D: Sort out 3D heterogeneity of 2D data whose 3D reconstruction parameters (xform.projection) have been determined already using 3D sorting.
  
 \\ \\
 ===== Usage ===== ===== Usage =====
  
-Usage1 in command line+Usage in command line
  
-  sxrsort3d_depth.py   --refinement_dir=refinemen_out_dir  --output_dir=master_dir  --niter_for_sorting=num_of_sorting_iterations  --mask3D=mask3d_file  --focus=focus3d_file  --radius=outer_radius  --sym=symmetry  --img_per_group=img_per_grp --minimum_grp_size=minimum_grp_size --depth_order=depth_of_order --noctf=no_ctf  --instack=input_stack_file --memory_per_node=memeory_per_node --orientation_groups=number_of_orientation_groups --not_include_unaccounted=not_include_unaccounted --stop_mgskmeans_percentage=MGSKmeans_stop_ratio --swap_ratio=accounted_vs_unaccounted_swap_ratio --notapplybckgnoise=do_not_use_background_noise  --do_swap_au=turn_on_swap_accounted_vs_unaccounted  +  sxrsort3d.py  stack  outdir  mask  --previous_run1=sort3d_run1_directory  --previous_run2=sort3d_run2_directory  --focus=3D_focus_mask  --radius=outer_radius  --delta=angular_step  --CTF  --sym=symmetry  --number_of_images_per_group=images_per_group  --nxinit=nxinit  --smallest_group=smallest_group  --chunkdir=chunkdir  --ir=inner_radius  --maxit=max_iter  --rs=ring_step  --xr=xr  --yr=yr  --ts=ts  --an=angular_neighborhood  --center=centring_method  --nassign=nassign  --nrefine=nrefine  --stoprnct=stop_percent  --function=user_function  --independent=independent_runs  --low_pass_filter=low_pass_filter  --unaccounted  --seed=random_seed  --group_size_for_unaccounted=group_size_for_unaccounted  --sausage  --PWadjustment=PWadjustment  --protein_shape=protein_shape  --upscale=upscale  --wn=wn  --interpolation=method
- +
-Usage2 in command line +
- +
-  sxrsort3d_depth.py   --instack=input_stack_file  --output_dir=master_dir --mask3D=mask3d_file  --focus=focus3d_file  --radius=outer_radius  --sym=symmetry  --number_of_images_per_group=num_of_images_per_group --minimum_grp_size=minimum_grp_size --depth_order=depth_of_order --nxinit=initial_image_size --noctf=no_ctf   --memory_per_node=memeory_per_node --orientation_groups=number_of_orientation_groups --not_include_unaccounted=not_include_unaccounted --stop_mgskmeans_percentage=MGSKmeans_stop_ratio --swap_ratio=accounted_vs_unaccounted_swap_ratio --notapplybckgnoise=do_not_use_background_noise  --do_swap_au=turn_on_swap_accounted_vs_unaccounted+
  
 \\ \\
Line 20: Line 16:
 sxrsort3d.py exists only in MPI version. sxrsort3d.py exists only in MPI version.
  
-\\ __Initiate sorting from a SPHIRE/SPARX refinement__: In this mode, one can select arbitrary iteration of a 3D refinement directory. Typically, it is the master directory of a sxmeridien refinement via —niter_for_sorting option.  +  mpirun -np 192 sxrsort3d.py bdb:data rsort3d mask.hdf --previous_run1=sort3d1 --previous_run2=sort3d2 --radius=88 --maxit=25 --independent=3 --CTF --number_of_images_per_group=11000 --low_pass_filter=.20 --chunkdir=./ --sym=c4 --PWadjustment=pwrec.txt
-  mpirun  -np  176  sxrsort3d.py  --refinement_method=SPARX  --refinement_dir=meridien_outdir  --niter_for_sorting=30  --radius=120  --sym=c5  --number_of_images_per_group=6000  --smallest_group=1500  --nindependent=5  --interpolation=trl  --low_pass_filter=0.25+
  
-\\ __Initiate sorting from a data stack__: Currently, this mode is not supported by SPHIRE GUI. +\\
-  mpirun  -np  176  sxrsort3d.py  --instack=bdb:data  --mask3D=mask3d.hdf  --focus=focus3d.hdf  -radius=29  --sym=c1  --nxinit=64  --number_of_images_per_group=2000  --nindependent=3  --low_pass_filter=0.25  --interpolation=4nn  --comparison_method=cross  --Kmeans_lpf=adhoc +
- +
-\\ __Initiate sorting from a relion refinement__: For this mode, please provide relion refinement directory. The program will pick up the results of the last iteration and start sorting. Currently, this mode is not supported by SPHIRE GUI. +
-  mpirun  -np  160  sxrsort3d.py  --refinement_method=relion  --refinement_dir=relion_outdir  --radius=120  --sym=c5  --nindependent=3  --number_of_images_per_group=6000 +
- +
-\\ NOTE - How to continue sxmeridien refinement using sorting results: Please use --ctrefromsort3d option of sxmeridien, then specify the directory where you wish to continue the refinement to --oldrefdir option and a subset of data to —-subset option. The command will load the refinement information from the directory and continue refinement. Optinally, you can specify the iteration number for continuing refinement using -—ctrefromiter option, which is not necessarily be the same iteration where you used for the 3D sorting. Also, one can modify refinement parameters of the selected iteration through the other options.  +
-  mpirun  -np  88  sxmeridien.py  --ctrefromsort3d  --oldrefdir=meridien_outdir  --ctrefromiter=20  --subset=Clusters3.txt ''' <<BR>><<BR>> +
- +
-\\ +
 ===== Input ===== ===== Input =====
-=== Main Paramaters === +=== Main Parameters === 
-  ; refinement_method : Input 3D refinement method: Valid values are 'SPARX' and 'relion'. Currently, SHPIRE GUI (sxgui) supports only 'SPARX'. (default none) +  ; stack : Input images stack: (default required string) 
-  ; refinement_dir : Input 3D refinement directory: Usually the master output directory of sxmeridien. (default none) +  ; outdir : Output directory: There is a log.txt that describes the sequences of computations in the program. (default required string) 
-  ; masterdir : Output directory: The master output directory for sorting. (default none) +  ; mask : 3D mask: (default none) 
-  ; niter_for_sorting : 3D refinement iteration: Specify an iteration number of 3D refinement where the 3D alignment parameters should be extracted for this sorting. By default, it uses iteration achieved best resolution. (default -1) + 
-  ; mask3D : 3D mask: File path of the global 3D mask for clustering. (default none) +  ; %%--%%previous_run1 : Directory of 1st sort3d run: (default required string) 
-  ; focus : Focus 3D mask: File path of a binary 3D mask for focused clustering. (default none) +  ; %%--%%previous_run2 : Directory of 2nd sort3d run: (default required string) 
-  ; radius : Outer radius for rotational correlation [Pixels]: Particle radius in pixel for rotational correlation. The value must be smaller than half the box size. (default -1) +  ; %%--%%focus : Focus 3D mask: Mask used for focused clustering (default none) 
-  ; sym : Point-group symmetry: Point group symmetry of the structure. (default c1)  +  ; %%--%%radius : Outer radius for rotational correlation [Pixels]: Must be smaller than half the box size. Please set to the radius of the particle. (default -1) 
-  ; number_of_images_per_group : Images per group: The number of images per a group. This value is critical for successful 3D clustering. (default 1000)  +  ; %%--%%delta : Angular step for projections: (default '2') 
-  ; smallest_group : Smallest group size: Minimum number of members for being identified as a group. This value must be smaller than the number of images per a group (number_of_images_per_group). (default 500)  +  ; %%--%%CTF : Use CTF: Do a full CTF correction during the alignment. (default False)  
-  ; nxinit : Initial image size for sorting [Pixels]: If it is necessary to speed up the processing time, set a non-zero positive integer to this option. Then, the program will reduce image size of original data by resampling to the specified size. By default, program determines the value from resolution. (default -1)+  ; %%--%%sym : Point-group symmetry: (default c1)  
 +  ; %%--%%number_of_images_per_group : Images per group: Critical number of images per group, defined by user. (default 1000)  
 +  ; %%--%%nxinit : Initial image size for sorting: (default 64) 
 +  ; %%--%%smallest_group : Smallest group size: Minimum members for identified group. (default 500)  
 +  ; %%--%%chunkdir : Directory containing chunk files: Typically, specify meridien output directory. A chunk file contains a list of particle IDs belonging to associated halfset. This information is used for computing margin of error. (default none)
  
 \\ \\
 === Advanced Parameters === === Advanced Parameters ===
-  ; low_pass_filter : Low-pass filter frequency [1/Pixel]: Absolute frequency cutoff of the low-pass filter used on the original image size for the 3D sorting. (default -1.0) +  ; %%--%%ir : Inner radius for rotational correlation [Pixels]: Must be bigger than 1. (default 1) 
-  ; Kmeans_lpf : Low-pass filter method for K-means: Low-pass filter method for K-means clustering. Valid values are 'adaptive', 'max', 'min', 'adhoc', and 'avg'. (default adaptive) +  ; %%--%%maxit : Maximum iterations: (default 50) 
-  ; nindependent : Independent runs: Number of independent runs for Equal Sized K-means clustering. The value must be an odd number larger than 2. (default 3)  +  ; %%--%%rs : Step between rings in rotational correlation: Must be bigger than 0. (default 1) 
-  ; noctf : No CTF correction: Use this option if full CTF correction should not be applied during the 3D clustering. By default, the program will do full CTF correction. (default False)  +  ; %%--%%xr : X search range [Pixels]: The translational search range in the x direction will take place in a +/xr range. (default '1') 
-  ; PWadjustment : Reference power spectrum file path: Path of text file containing 1D reference power spectrum of a PDB structure or EM map. The power spectrum will be used as reference to adjust the power spectra of clustered volumes. (default none)  +  ; %%--%%yr : Y search range [Pixels]: The translational search range in the y direction. If omitted it will be set as xr. (default '-1') 
-  ; interpolation : 3D reconstruction interpolation method: Interpolation method for 3D reconstruction. Valid values are 'trl' and '4nn'. (default 4nn)  +  ; %%--%%ts : Translational search step [Pixels]: The search will be performed in -xr, -xr+ts, 0, xr-ts, xr, can be fractional. (default '0.25') 
-  ; comparison_method : Comparison method: Similarity measurement for the comparison between reprojected reference images and particle images. Valid values are 'cross' (cross-correlaton coefficients) and 'eucd' (Euclidean distance). (default cross)  +  ; %%--%%an : Local angular search width [Degrees]: This defines the neighbourhood where the local angular search will be performed. (default '-1') 
-  ; instack : Input images stack: File path of particle stack for sorting. This option is not currently supported by SHPIRE GUI (sxgui). (default none)+  ; %%--%%center : Centering method: 0 - if you do not want the volume to be centered, 1 - center the volume using the center of gravity. (default 0) 
 +  ; %%--%%nassign : Number of reassignment iterations: Performed for each angular step. (default 1) 
 +  ; %%--%%nrefine : Number of alignment iterations: Performed for each angular step. (default 0) 
 +  ; %%--%%stoprnct : Assignment convergence threshold [%]: Used to asses convergence of the run. It is the minimum percentage of assignment change required to stop the run.  (default 3.0)  
 +  ; %%--%%function : Reference preparation function: Function used to prepare the reference volume. (default do_volume_mrk05)  
 +  ; %%--%%independent : Number of independent runs: Number of independent equal-Kmeans(default 3)  
 +  ; %%--%%low_pass_filter : Low-pass filter frequency [1/Pixel]: Low-pass filter used for the 3D sorting on the original image size. (default -1.0) 
 +  ; %%--%%unaccounted : Reconstruct unaccounted images: (default False)  
 +  ; %%--%%seed : Random seed: Seed used for the initial random assignment for EQ Kmeans. The program generates a random integer by default. (default -1)  
 +  ; %%--%%group_size_for_unaccounted : Unaccounted particles group size: (default 500)  
 +  ; %%--%%sausage : Use sausage filter: (default False) 
 +  ; %%--%%PWadjustment : Power spectrum reference: Text file containing a 1D reference power spectrum. (default none)  
 +  ; %%--%%protein_shape : Protein Shape: It defines protein preferred orientation angles. "g" is for globular proteins and "f" is for filament proteins. (default g) 
 +  ; %%--%%upscale : Power spectrum adjustment strength: This parameters adjusts how strongly the power spectrum of the volume should be modified to match the reference. A value of 1 brings the volume's power spectrum completely to the reference, while a value of 0 means no modification.  (default 0.5)  
 +  ; %%--%%wn : Target image size [Pixels]: If different than 0, then the images will be rescaled to fit this size. (default 0)  
 +  ; %%--%%interpolation : 3D interpolation method: Method interpolation in 3D. Options are tr1 or 4nn. (default 4nn)
  
 \\ \\
 ===== Output ===== ===== Output =====
-Please use --masterdir option to specify the output directory. The results will be written here. This directory will be created automatically if it does not exist  Here, you can find a log.txt that describes the sequences of computations in the program.  
  
 \\ \\
 ===== Description ===== ===== Description =====
-sxrsort3d finds out stable members by carrying out two-way comparison of two independent sxsort3d runs.+sxrsort3d.py finds out stable members by carrying out two-way comparison of two independent sxsort3d.py runs.
  
 For small tested datasets (real and simulated ribosome data around 10K particles), it gives 70%-90% reproducibility. However, this rate also depends on the choice of number of images per group and number of particles in the smallest group. For small tested datasets (real and simulated ribosome data around 10K particles), it gives 70%-90% reproducibility. However, this rate also depends on the choice of number of images per group and number of particles in the smallest group.
 +
 +\\
 +=== Time and Memory ===
 +On lonestar cluster of TACC, using 264 cpus, it takes about 2 hours and 23 minutes to accomplish 95953 128x128 images for one sxsort3d.py independent run, 2 hours 24 minutes to accomplish one independent sxrsort3d.py run with number_of_images_per_group set as 30000. 
  
 \\ \\
Line 75: Line 83:
 ==== Reference ==== ==== Reference ====
 Not published yet. Not published yet.
 +
 +\\
 +==== Developer Notes ====
  
 \\ \\
Line 86: Line 97:
 \\ \\
 ==== Files ==== ==== Files ====
-sxrsort3d.py+sparx/bin/sxrsort3d.py
  
 \\ \\
 ==== See also ==== ==== See also ====
-[[sxsort3d|sxsort3d.py]]+[[pipeline:meridien:sxmeridien|sxmeridien]], [[pipeline:utilities:sxheader|sxheader]], [[[pipeline:sort3d:sx3dvariability|sx3dvariability]], [[pipeline:sort3d:sxsort3d|sxsort3d]], and [[pipeline:sort3d:sxsort3d_depth|sxsort3d_depth]].
  
 \\ \\
 ==== Maturity ==== ==== Maturity ====
-beta:: Under development. It has been tested, The test cases/examples are available upon request. Please let us know if there are any bugs.+Alpha:: Under development.
  
 \\ \\
-==== Known Bugs ==== +==== Bugs ==== 
-None so far.+There are no known bugs so far.
  
 \\ \\
  
pipeline/sort3d/sxrsort3d.txt · Last modified: 2018/06/20 13:12 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: GNU Free Documentation License 1.3
GNU Free Documentation License 1.3 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki