This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Last revision Both sides next revision | ||
pipeline:utilities:sxprocess [2018/11/20 10:18] shaikh [Typical usage] |
pipeline:utilities:sxprocess [2019/04/02 09:40] lusnig |
||
---|---|---|---|
Line 1: | Line 1: | ||
~~NOTOC~~ | ~~NOTOC~~ | ||
- | ===== sxprocess | + | ===== sp_process |
Miscellaneous Commands: Carry out various SPARX commands on image series, and generate data and initialize database for demo script. | Miscellaneous Commands: Carry out various SPARX commands on image series, and generate data and initialize database for demo script. | ||
Line 9: | Line 9: | ||
Usage in command line | Usage in command line | ||
- | | + | |
\\ | \\ | ||
===== Typical usage ===== | ===== Typical usage ===== | ||
- | sxprocess | + | sp_process |
1. Phase flip a stack of images and write output to new file: | 1. Phase flip a stack of images and write output to new file: | ||
- | | + | |
2. Change size of image or map (resample, decimation or interpolation up). The process also changes the pixel size and window size accordingly. | 2. Change size of image or map (resample, decimation or interpolation up). The process also changes the pixel size and window size accordingly. | ||
- | | + | |
3. Compute average power spectrum of a stack of 2D images with optional padding (option wn) with zeroes. | 3. Compute average power spectrum of a stack of 2D images with optional padding (option wn) with zeroes. | ||
- | | + | |
4. Generate a stack of projections bdb:data and micrographs with prefix mic (i.e., mic0.hdf, mic1.hdf etc) from structure input_structure.hdf, | 4. Generate a stack of projections bdb:data and micrographs with prefix mic (i.e., mic0.hdf, mic1.hdf etc) from structure input_structure.hdf, | ||
- | | + | |
5. Retrieve original image numbers in the selected ISAC group (here group 12 from generation 3). | 5. Retrieve original image numbers in the selected ISAC group (here group 12 from generation 3). | ||
- | | + | |
| | ||
- | | + | |
7. Adjust rotationally averaged power spectrum of an image to that of a reference image or a reference 1D power spectrum stored in an ASCII file. Optionally use a tangent low-pass filter. | 7. Adjust rotationally averaged power spectrum of an image to that of a reference image or a reference 1D power spectrum stored in an ASCII file. Optionally use a tangent low-pass filter. | ||
- | | + | |
- | | + | |
8. Generate a 1D rotationally averaged power spectrum of an image. | 8. Generate a 1D rotationally averaged power spectrum of an image. | ||
- | | + | |
\\ Output will contain three columns: | \\ Output will contain three columns: | ||
Line 56: | Line 56: | ||
9. Apply 3D transformation (rotation and/or shift) to a set of orientation parameters associated with projection data. | 9. Apply 3D transformation (rotation and/or shift) to a set of orientation parameters associated with projection data. | ||
- | | + | |
The output file is then imported and 3D transformed map computed. | The output file is then imported and 3D transformed map computed. | ||
- | | + | |
- | mpirun | + | mpirun |
The reconstructed map is in the position of the map computed using the input.txt parameters and then transformed with rot_shift3D(vol, | The reconstructed map is in the position of the map computed using the input.txt parameters and then transformed with rot_shift3D(vol, | ||
- | 10. Import ctf parameters from the output of sxcter | + | 10. Import ctf parameters from the output of sp_cter |
\\ There are three possible input files formats: (1) all particles are in one stack, (2) and/or (3) particles are in stacks, each stack corresponds to a single micrograph. In each case the particles should contain a name of the micrograph of origin stores using attribute name ' | \\ There are three possible input files formats: (1) all particles are in one stack, (2) and/or (3) particles are in stacks, each stack corresponds to a single micrograph. In each case the particles should contain a name of the micrograph of origin stores using attribute name ' | ||
- | | + | |
\\ Output will be a vritual stack bdb:Gdata. | \\ Output will be a vritual stack bdb:Gdata. | ||
- | | + | |
- | To concatenate output files: | + | To concatenate output files, |
cd directory | cd directory | ||
Line 83: | Line 83: | ||
11. Scale 3D shifts. The shifts in the input five columns text file with 3D orientation parameters will be DIVIDED by the scale factor. | 11. Scale 3D shifts. The shifts in the input five columns text file with 3D orientation parameters will be DIVIDED by the scale factor. | ||
- | | + | |
12. Generate soft-edged 3D mask from input 3D map automatically or using the user-provided threshold. | 12. Generate soft-edged 3D mask from input 3D map automatically or using the user-provided threshold. | ||
Line 89: | Line 89: | ||
\\ Automatically compute the threshold to intially obtain the largest density cluster. | \\ Automatically compute the threshold to intially obtain the largest density cluster. | ||
- | | + | |
\\ Use the user-provided threshold to intially obtain the largest density cluster. | \\ Use the user-provided threshold to intially obtain the largest density cluster. | ||
- | | + | |
13. Generate binary 3D mask from input 3D map using the user-provided threshold. | 13. Generate binary 3D mask from input 3D map using the user-provided threshold. | ||
- | | + | |
14. PostRefiner - Post-refine maps or images by enhancing the power spectrum after 2D averaging, 3D refinement, or 3D sorting run. | 14. PostRefiner - Post-refine maps or images by enhancing the power spectrum after 2D averaging, 3D refinement, or 3D sorting run. | ||
Line 112: | Line 112: | ||
\\ Options are independent of each others. | \\ Options are independent of each others. | ||
* %%--%%do_adaptive_mask : =True when it is restored, the program adaptively creates adaptive mask file using summed two maps. This takes a couple of minutes. For map with dimension of 384*384*384, | * %%--%%do_adaptive_mask : =True when it is restored, the program adaptively creates adaptive mask file using summed two maps. This takes a couple of minutes. For map with dimension of 384*384*384, | ||
- | * %%--%%mask_threshold | + | * %%--%%threshold |
* %%--%%mtf | * %%--%%mtf | ||
* %%--%%fsc_adj | * %%--%%fsc_adj | ||
Line 118: | Line 118: | ||
* %%--%%fl | * %%--%%fl | ||
- | | + | |
- | | + | |
- | | + | |
\\ (2) Cluster Maps Mode or Single Map Mode: | \\ (2) Cluster Maps Mode or Single Map Mode: | ||
Line 135: | Line 135: | ||
\\ Note that this mode is mainly designed for SORT3D_DEPTH outputs but also applicable to ANY maps. | \\ Note that this mode is mainly designed for SORT3D_DEPTH outputs but also applicable to ANY maps. | ||
- | | + | |
\\ To process one single map, simply specify the input volume path (without wild card ' | \\ To process one single map, simply specify the input volume path (without wild card ' | ||
- | | + | |
\\ (3) Images Mode - for 2D images | \\ (3) Images Mode - for 2D images | ||
Line 145: | Line 145: | ||
15. Window stack file -reduce size of images without changing the pixel size. | 15. Window stack file -reduce size of images without changing the pixel size. | ||
- | sxprocess.py input.hdf output.hdf --box=new_box_size | + | sp_process.py input.hdf output.hdf --box=new_box_size |
16. Pad stack file --pad images to a larger size and set surround background to request value (default 0.0). | 16. Pad stack file --pad images to a larger size and set surround background to request value (default 0.0). | ||
- | sxprocess.py input.hdf output.hdf --box=new_box_size --background=3.0 | + | sp_process.py input.hdf output.hdf --box=new_box_size --background=3.0 |
17. Create angular distribution .build file | 17. Create angular distribution .build file | ||
- | sxprocess.py --angular_distribution | + | sp_process.py --angular_distribution |
18. Subtract from images in the first stack images in the second stack and write results to the third stack. | 18. Subtract from images in the first stack images in the second stack and write results to the third stack. | ||
- | \\ If the name of the output stack is the same as the second stack, the results will be written to the second stack (it will be overwritten). | + | If the name of the output stack is the same as the second stack, the results will be written to the second |
+ | stack (it will be overwritten). | ||
- | sxprocess.py bdb: | + | sp_process.py bdb: |
19. Balance angular distribution. | 19. Balance angular distribution. | ||
- | \\ Input ASCII file with 3D orientation parameters, compute a histogram of distribution of angles using user-provided angular step, retain a subset of randomly selected projection direction per angular bin using user-provided threshold, and write the list of the all retained projection directions. | + | Input ASCII file with 3D orientation parameters, compute a histogram |
+ | of distribution of angles using user-provided angular step, retain a subset of randomly selected | ||
+ | projection direction per angular bin using user-provided threshold, and write the list of the all | ||
+ | retained projection directions. | ||
+ | with options makevstack and list). | ||
+ | |||
+ | sp_process.py --balance_angular_distribution | ||
- | sxprocess.py --balance_angular_distribution | ||
- | \\ In order to create a substack with retained images, use e2bdb.py with options makevstack and list. | ||
\\ | \\ | ||
Line 191: | Line 196: | ||
; %%--%%rotpw : Compute rotationally averaged power spectrum of the input image: Store in output text file with name as specified. (default none) | ; %%--%%rotpw : Compute rotationally averaged power spectrum of the input image: Store in output text file with name as specified. (default none) | ||
- | ; %%--%%importctf : Import | + | ; %%--%%importctf : Import |
; %%--%%input : Input particle image stack file: CTF parameters will be imported into headers of images in the stack. (default none) | ; %%--%%input : Input particle image stack file: CTF parameters will be imported into headers of images in the stack. (default none) | ||
- | ; %%--%%defocuserror : Defocus error threshold: Exclude micrographs whose relative defocus error as estimated by sxcter | + | ; %%--%%defocuserror : Defocus error threshold: Exclude micrographs whose relative defocus error as estimated by sp_cter |
- | ; %%--%%astigmatismerror : Astigmatism error threshold: Set to zero astigmatism for micrographs whose astigmatism angular error as estimated by sxcter | + | ; %%--%%astigmatismerror : Astigmatism error threshold: Set to zero astigmatism for micrographs whose astigmatism angular error as estimated by sp_cter |
; %%--%%scale : Divide shifts in input 3D orientation parameters text file by the specified scale factor: (default -1.0) | ; %%--%%scale : Divide shifts in input 3D orientation parameters text file by the specified scale factor: (default -1.0) | ||
; %%--%%adaptive_mask : Create soft-edged 3D mask: Create soft-edged 3D mask from the input structure. (default False) | ; %%--%%adaptive_mask : Create soft-edged 3D mask: Create soft-edged 3D mask from the input structure. (default False) | ||
- | ; %%--%%nsigma | + | ; %%--%%use_mol_mass |
- | ; %%--%%threshold : Binarization threshold: Defines the threshold used in the first step of the processing to generate a binary version of the input structure. If the value is lower-equal than the default, the option will be ignored and the threshold will be set according to nsigma method above. (default -9999.0) | + | ; %%--%%threshold : Binarization threshold: Defines the threshold used in the first step of the processing to generate a binary version of the input structure. If the value is lower-equal than the default, the option will be ignored and the threshold will be set according to nsigma method above. (default -9999.0) |
- | ; %%--%%ndilation | + | ; %%--%%mol_mass |
- | ; %%--%%kernel_size | + | ; %%--%%nsigma : Density standard deviation threshold: Defines |
- | ; %%--%%gauss_standard_dev | + | ; %%--%%ndilation |
+ | ; %%--%%edge_width | ||
+ | ; %%--%%edge_type : Soft-edge type: The type of soft-edge for moon-eliminator 3D mask and a moon-eliminated soft-edged 3D mask. Available methods are (1) \' | ||
; %%--%%binary_mask : Create binary 3D mask: Create binary 3D mask from the input structure. (default False) | ; %%--%%binary_mask : Create binary 3D mask: Create binary 3D mask from the input structure. (default False) | ||
- | ; %%--%%bin_threshold : Binarization threshold: Defines the threshold used in the first step of the processing to generate a binary version of the input structure. (default 0.0) | + | ; %%--%%nerosion |
- | ; %%--%%ne | + | |
- | ; %%--%%nd : Dilation cycles: After erosing the binary structure is dilated back to smooth the surface and match the original size. (default 0) | + | |
; %%--%%combinemaps : Post-refine structures or images: Post-refine structures or averages by enhancing their high-frequencies after 2D alignment, 3D refinement, or 3D sorting. Available modes are (1) Halfset Volumes Mode, (2) Cluster Volumes Mode or Single Volumes Mode, and (3) Images Mode. (1) The Halfset Volumes Mode combines a pair of unfiltered odd & even 3D density maps, then enhance the power spectrum at high frequencies. B-factor can be automatically estimated from these unfiltered halfset maps. This mode requires two arguments; use unfiltered hal-maps produced by MERIDIEN. (2) The Cluster Volumes Mode or Single Volumes Mode enhances the power spectrum of cluster maps, produced by SORT3D_DEPTH, | ; %%--%%combinemaps : Post-refine structures or images: Post-refine structures or averages by enhancing their high-frequencies after 2D alignment, 3D refinement, or 3D sorting. Available modes are (1) Halfset Volumes Mode, (2) Cluster Volumes Mode or Single Volumes Mode, and (3) Images Mode. (1) The Halfset Volumes Mode combines a pair of unfiltered odd & even 3D density maps, then enhance the power spectrum at high frequencies. B-factor can be automatically estimated from these unfiltered halfset maps. This mode requires two arguments; use unfiltered hal-maps produced by MERIDIEN. (2) The Cluster Volumes Mode or Single Volumes Mode enhances the power spectrum of cluster maps, produced by SORT3D_DEPTH, | ||
Line 214: | Line 219: | ||
; %%--%%output : Output file name: File name of output final post-refined structure. (default vol_combined.hdf) | ; %%--%%output : Output file name: File name of output final post-refined structure. (default vol_combined.hdf) | ||
; %%--%%pixel_size : Pixel size [A]: Pixel size of input data. (default 0.0) | ; %%--%%pixel_size : Pixel size [A]: Pixel size of input data. (default 0.0) | ||
- | ; %%--%%mask : 3D mask file: Path to user-provided mask. (default none) | + | ; %%--%%mask : 3D mask file: Path to user-provided mask. (default none) : %%--%%do_adaptive_mask==False |
- | ; %%--%%do_adaptive_mask : Apply adaptive mask: Program creates mask adaptively with given density threshold. (default False) | + | ; %%--%%do_adaptive_mask : Apply adaptive mask: Program creates mask adaptively with given density threshold. (default False) |
- | ; %%--%%mask_threshold | + | ; %%--%%do_adaptive_mask |
- | ; %%--%%cosine_edge | + | ; %%--%%do_approx |
- | ; %%--%%dilation : Surface dilation size [Pixels]: Size of surface dilation or erosion. Effective only with --do_adaptive_mask option. (default 6.0) | + | |
; %%--%%mtf : MTF file: Path to file contains the MTF (modulation transfer function) of the detector used. (default none) | ; %%--%%mtf : MTF file: Path to file contains the MTF (modulation transfer function) of the detector used. (default none) | ||
; %%--%%fsc_adj : Apply FSC-based low-pass filter: Applies an FSC-based low-pass filter to the merged map before the B-factor estimation. Effective only in Halfset Volumes Mode. (default False) | ; %%--%%fsc_adj : Apply FSC-based low-pass filter: Applies an FSC-based low-pass filter to the merged map before the B-factor estimation. Effective only in Halfset Volumes Mode. (default False) | ||
; %%--%%B_enhance : B-factor enhancement: | ; %%--%%B_enhance : B-factor enhancement: | ||
- | ; %%--%%B_start : B-factor estimation lower limit [A]: Frequency in Angstrom defining lower boundary of B-factor estimation. Effective only in Halfset Volumes Mode with --B_enhance=0.0. (default 10.0) | + | ; %%--%%B_start : B-factor estimation lower limit [A]: Frequency in Angstrom defining lower boundary of B-factor estimation. Effective only in Halfset Volumes Mode with --B_enhance=0.0. (default 10.0) : %%--%%B_enhance==0.0 |
- | ; %%--%%B_stop : B-factor estimation upper limit [A]: Frequency in Angstrom defining upper boundary of B-factor estimation. Recommended to set the upper boundary to the frequency where fsc is smaller than 0.0. Effective only in Halfset Volumes Mode with --B_enhance=0.0. (default | + | ; %%--%%B_stop : B-factor estimation upper limit [A]: Frequency in Angstrom defining upper boundary of B-factor estimation. Recommended to set the upper boundary to the frequency where fsc is smaller than 0.0. Effective only in Halfset Volumes Mode with --B_enhance=0.0. (default |
; %%--%%fl : Low-pass filter frequency [A]: 0.0: low-pass filter to resolution (valid only in Halfset Volumes Mode); A value larger than 0.5: low-pass filter to the value in Angstrom; -1.0: no low-pass filter. (default 0.0) | ; %%--%%fl : Low-pass filter frequency [A]: 0.0: low-pass filter to resolution (valid only in Halfset Volumes Mode); A value larger than 0.5: low-pass filter to the value in Angstrom; -1.0: no low-pass filter. (default 0.0) | ||
- | ; %%--%%aa : Low-pass filter fall-off [1/Pixels]: Low-pass filter fall-off. Effective only when --fl option is not -1.0. (default 0.01) | + | ; %%--%%aa : Low-pass filter fall-off [1/Pixels]: Low-pass filter fall-off. Effective only when --fl option is not -1.0. (default 0.01) : %%--fl%%!=-1.0 |
; %%--%%window_stack : Window stack images using a smaller window size: (default False) | ; %%--%%window_stack : Window stack images using a smaller window size: (default False) | ||
; %%--%%box : New window size: (default 0) | ; %%--%%box : New window size: (default 0) | ||
- | ; %%--%%balance_angular_distribution : Balance Angular Distribution: | + | ; %%--%%balance_angular_distribution : Balance Angular Distribution: |
- | ; %%--%%max_occupy : Maximum orientations per reference angle: Maximum number of angular orientations | + | ; %%--%%max_occupy : Maximum orientations per reference angle: Maximum number of particles |
- | ; %%--%%angstep : Angular step of reference angles: angular step of reference angles, i.e., number of bins of angular histogram. (default | + | ; %%--%%angstep |
+ | ; %%--%%symmetry : Point-group symmetry | ||
\\ | \\ | ||
Line 246: | Line 251: | ||
; %%--%%makedb : Generate a database file containing a set of parameters: One argument is required, name of key with which the database will be created. Fill in database with parameters specified as %%--%%makedb param1=value1: | ; %%--%%makedb : Generate a database file containing a set of parameters: One argument is required, name of key with which the database will be created. Fill in database with parameters specified as %%--%%makedb param1=value1: | ||
- | ; %%--%%transformparams : Transform 3D projection orientation parameters: Using six 3D parameters (phi, | + | ; %%--%%transformparams : Transform 3D projection orientation parameters: Using six 3D parameters (phi, |
\\ | \\ | ||
Line 274: | Line 279: | ||
\\ | \\ | ||
==== Files ==== | ==== Files ==== | ||
- | sparx/bin/sxprocess.py | + | sparx/bin/sp_process.py |
\\ | \\ | ||
Line 288: | Line 293: | ||
\\ | \\ | ||
+ |