Differences

This shows you the differences between two versions of the page.

--- pipeline:window:sxrewindow [2018/08/07 22:03]
moriya
+++ pipeline:window:sxrewindow [2019/04/02 09:23]
lusnig
@@ Line 1: / Line 1: @@
 ~~NOTOC~~
-===== sxpipe restacking =====
+===== sp_rewindow =====
-Restacking: Generate all necessary information to restack the input stack (i.e. particle image ID list, CTF parameters list, projection parameters list) while appling micrograph selection list. Optionally, the command can directly output the virtual stack.  In addition, this command can be used to generate all parameters files for reboxing (i.e. original/centered particle coordinates list files, CTF parameters list, original/centered projection parameters list as well as micrograph selection file). Optionally, user can provided a 3D shift to recenter the projection parameters and so the particle coordinates.
+Particle Reextraction: Rewindow particles from micrographs using the information stored in rebox files.
 \\
@@ Line 9: / Line 9: @@
 Usage in command line
-  sxpipe.py  restacking  input_bdb_stack_path  output_directory  --selection_list=FILE_PATH  --shift3d_x=SHIFT3D_X  --shift3d_y=SHIFT3D_Y  --shift3d_z=SHIFT3D_Z  --save_vstack  --sv_vstack_basename=BASENAME  --reboxing  --rb_box_size=BOX_SIZE
+  sp_rewindow.py  input_micrograph_pattern  input_rebox_pattern  output_directory  --selection_list=SELECTION_TEXT_FILE  --box_size=BOX_SIZE  --skip_invert  --mic_resample_ratio=RATIO  --swap_ctf_params=CTER_PARTRES_FILE_PATH  --check_consistency
 \\
 ===== Typical usage =====
-sxpipe restacking does not support MPI.
+sp_rewindow exists in both MPI and non-MPI versions (Running MPI version does not require --MPI flag).
-Do restacking to generate the virtual stack of only particle images associated to the micrographs in the selection list.
+There are several modes to run the program:
-  sxpipe.py restacking 'bdb:outdir_window#data' 'outdir_pipe_restacking' --selection_list='micrograph_selection_list.txt' --save_vstack --sv_vstack_basename='vstack_selected'
+\\ __1. All Micrographs Mode - Process all micrographs in a directory:__
+\\ Specify path pattern of input micrographs and rebox files with a wild card (*).. The path pattern must be enclosed by single quotes (') or double quotes ("). (Note: sp_gui.py automatically adds single quotes (')). The substring at the variable part must be same between a associated pair of input micrograph and rebox file. bdb files cannot be selected as input micrographs. Finally, specify output directory where all outputs should be saved. In this mode, all micrographs matching the path pattern will be processed.
+  mpirun  -np  32  sp_rewindow.py  './mic*.hdf'  'outdir_rebox/centered_rebox/mic*_centered_rebox.rbx'  outdir_rewindow  --box_size=64
-Generate the reboxing information of all particle images in the input stack.
+\\ You can also ignore per-particle CTF information stored in rebox files and use CTF information stored in the CTER partres.txt instead.
+  mpirun  -np  32  sp_rewindow.py  './mic*.hdf'  'outdir_rebox/centered_rebox/mic*_centered_rebox.rbx'  outdir_rewindow  --box_size=64  --swap_ctf_params='outdir_cter/partres.txt'
-  sxpipe.py restacking 'bdb:outdir_window#data' 'outdir_pipe_restacking' --shift3d_z=20 --reboxing --rb_box_size=352
+\\ __2. Selected Micrographs Mode - Process all micrographs in a selection list file:__
+\\ In addition to input micrographs path pattern, rebox files path pattern, and output directry arguments, specify a name of micrograph selection list text file using --selection_list option. In this mode, only micrographs in the selection list which matches the file name part of the pattern (ignoring the directory paths) will be processed. If a micrograph name in the selection list does not exists in the directory specified by the micrograph path pattern, processing of the micrograph will be skipped.
+  mpirun  -np  32  sp_rewindow.py  './mic*.hdf'  'outdir_rebox/centered_rebox/mic*_centered_rebox.rbx'  outdir_rewindow  --selection_list=mic_list.txt  --box_size=64
-Generate the reboxing information of only particle images associated to the micrographs in the selection list.
+\\ __3. Single Micrograph Mode - Process a single micrograph:__
+\\ In addition to input micrographs path pattern, rebox files path pattern, CTF paramters source, and output directry arguments, specify a single micrograph name using --selection_list option. In this mode, only the specified single micrograph will be processed. If this micrograph name does not matches the file name part of the pattern (ignoring the directory paths), the process will exit without processing it. If this micrograph name matches the file name part of the pattern but does not exists in the directory which specified by the micrograph path pattern, again the process will exit without processing it. Use single processor for this mode.
-  sxpipe.py restacking 'bdb:outdir_window#data' 'outdir_pipe_restacking' --selection_list='micrograph_selection_list.txt' --shift3d_z=20 --reboxing --rb_box_size=352
+  sp_rewindow.py  './mic*.hdf'  'outdir_rebox/centered_rebox/mic*_centered_rebox.rbx'  outdir_rewindow  --selection_list=mic0.hdf  --box_size=64
-Do both restacking and reboxing to generate both the virtual stack and reboxing information of only particle images associated to the micrographs in the selection list.
+\\ For negative staining data, use --skip_invert.
+  mpirun  -np  32  sp_rewindow.py  './mic*.hdf'  'outdir_rebox/centered_rebox/mic*_centered_rebox.rbx'  outdir_rewindow  --box_size=64  --skip_invert
-  sxpipe.py restacking 'bdb:outdir_window#data' 'outdir_pipe_restacking' --selection_list='micrograph_selection_list.txt' --save_vstack --sv_vstack_basename='vstack_selected' --shift3d_z=20 --reboxing --rb_box_size=352
+\\ **NOTE:** Please remember to adjust box_size also when setting mic_resample_ratio < 0.1.
 \\
 ===== Input =====
 === Main Parameters ===
-  ; input_bdb_stack_path : Input BDB image stack: Specify the input BDB image stack. (default required string)
+  ; input_micrograph_pattern : Input micrograph path pattern: Specify path pattern of input micrographs with a wild card (*).. The path pattern must be enclosed by single quotes (') or double quotes ("). (Note: sp_gui.py automatically adds single quotes (')). The substring at the variable part must be same between the associated pair of input micrograph and rebox file. bdb files cannot be selected as input micrographs. (default required string)
-  ; output_directory : Output directory: The results will be written here. This directory will be created automatically and it must not exist previously. (default required string)
+  ; input_rebox_pattern : Input rebox path pattern: Specify path pattern of input rebox files with a wild card (*).. The path pattern must be enclosed by single quotes (') or double quotes ("). (Note: sp_gui.py automatically adds single quotes (')). The substring at the variable part must be same between the associated input micrograph. (default required string)
+  ; output_directory : Output directory: The results will be written here. It cannot be an existing one. (default required string)
-  ; %%--%%selection_list : Micrograph/Movie selection file: Specify path to text file containing a list of selected micrograph/movie names or paths. The particles associated with the micrographs/movies in this list will be processed. The file extension must be //.txt//. The directory path of each entry will be ignored if there are any. (default none)
+  ; %%--%%selection_list : Micrograph selection file: Specify a name of micrograph selection list text file for Selected Micrographs Mode. The file extension must be '.txt'. Alternatively, the file name of a single micrograph can be specified for Single Micrograph Mode. (default none)
-  ; %%--%%shift3d_x : 3D x-shift [Pixels]: Provide 3D x-shift corresponding to shifting the 3D volume along x-axis. (default 0)
+  ; %%--%%box_size : Particle box size [Pixels]: The x and y dimensions of square area to be windowed. The box size after resampling is assumed when mic_resample_ratio < 1.0. (default 256)
-  ; %%--%%shift3d_y : 3D y-shift [Pixels]: Provide 3D y-shift corresponding to shifting the 3D volume along y-axis. (default 0)
+  ; %%--%%skip_invert : Invert image contrast: Indicate if image contrast should be inverted or not. Do not invert for negative staining data. By default, the image contrast will be inverted for cryo data. (default False question reversed in GUI)
-  ; %%--%%shift3d_z : 3D z-shift [Pixels]: Provide 3D z-shift corresponding to shifting the 3D volume along z-axis. (default 0)
-  ; %%--%%save_vstack : Save virtual stack: Use this option to save the virtual stack. By default, the virtual stack will not be generated. (default False)
-  ; %%--%%sv_vstack_basename : Virtual stack basename: For %%--%%save_vstack, specify the basename of output virtual stack file. It cannot be empty string or only white spaces. (default vstack)
-  ; %%--%%reboxing : Generate reboxing information: Prepare reboxing by extracting coordinates from the input stack headers, then center them according to projection parameters in the header and user-provided 3D shift. If the headers do not contain projection parameters, the program assumes the prjection parameters are all zeros (null alignment). (default False)
-  ; %%--%%rb_box_size : Particle box size [Pixels]: For %%--%%reboxing option, specify the x and y dimensions of square area to be windowed. (default 0)
 \\
 === Advanced Parameters ===
+  ; %%--%%mic_resample_ratio : Image size reduction factor (<1): Use a value between 0.0 and 1.0 (excluding 0.0). The new pixel size will be automatically recalculated and stored in CTF paramers when mic_resample_ratio < 1.0 is used. (default 1.0)
+  ; %%--%%swap_ctf_params : Swap CTF parameters: Swaps CTF parameters by setting the CTF parameters in the specified CTER partres file while ignoring the CTF parameters in the input rebox parameters file. Typically, specify the file produced by sp_cter and normally called partres.txt. Alternatively, enter pixel size [A/Pixels] to simulate ideal CTF. By default, the program uses the CTF parameters in the input rebox parameters file. (default None)
+  ; %%--%%check_consistency : Check consistency of dataset: Create a text file containing the list of Micrograph ID entries might have inconsitency among the provided dataset. (i.e. mic_consistency_check_info_TIMESTAMP.txt). (default False)
 \\
 ===== Output =====
-=== List of output Files ===
+Inside the output directory, the program will write a local bdb stack image for each micrograph. The files are named micrograph name with "_ptcls" suffix. For example, ''outdir_rewindow/EMAN2DB/mic0_ptcls.bdb'' or ''bdb:outdir_rewindow#mic0_ptcls''
-|| **File Name** || **Discription** ||
-|| micrographs_in_input_stack.txt || Micrograph selection text file containing the list of micrograph basenames extracted from the input stack. This file can be used as micrograph selection text file of [[pipeline:cter:sxcter|sxcter]], [[pipeline:cter:sxgui_cter|sxgui_cter]], [[pipeline:window:sxwindow|sxwindow]], and [[pipeline:utilities:sxpipe_organize_micrographs|sxpipe organize_micrographs]]. ||
-|| micrographs_in_output_dataset.txt || Micrograph selection text file containing the list of micrograph in the output dataset. This file can be used as micrograph selection text file of [[pipeline:cter:sxcter|sxcter]], [[pipeline:cter:sxgui_cter|sxgui_cter]], [[pipeline:window:sxwindow|sxwindow]], and [[pipeline:utilities:sxpipe_organize_micrographs|sxpipe organize_micrographs]]. ||
-|| input_stack_particle_id_for_output_dataset.txt || Particle selection text file containing the list of particle image IDs used for generating the output dataset from the input stack. This file can be used as particle selection list text file of [[pipeline:utilities:e2bdb|e2bdb]]. ||
-|| ctf_params_for_output_dataset.txt || CTF parameters text file containing CTF parameters of all particle images in the output dateset. This projection parameters can be used as inputs of [[pipeline:window:sxwindow|sxwindow]] or imported to the reboxed stack using [[pipeline:utilities:sxheader|sxheader]]. ||
-|| original_proj_params_for_output_dataset.txt || Projection parameters text file containing **original** projection parameters of all particle images in the output dateset. This projection parameters can be imported to the reboxed stack using [[pipeline:utilities:sxheader|sxheader]]. ||
-|| centered_proj_params_for_output_dataset.txt || Projection parameters text file containing **centered** projection parameters of all particle images in the output dateset. This projection parameters can be imported to the reboxed stack using [[pipeline:utilities:sxheader|sxheader]]. ||
-|| original/${MICROGRAPH_ROOTNAME}_original.box || The list of original coordinates stored in the headers of particle images in the stack. These files can be used as inputs of [[pipeline:window:sxwindow|sxwindow]]. ||
-|| centered/${MICROGRAPH_ROOTNAME}_centered.box || The list of centered (or transformed) coordinates. These files can be used as inputs of [[pipeline:window:sxwindow|sxwindow]]. ||
-|| EMAN2DB/${VSTACK_BASENAME}.dbd || The output virtual stack. This stack can be directly used as input stack of [[pipeline:meridien:sxmeridien|sxmeridien]] local refinement. ||
 \\
 ===== Description =====
 This command executes the following processes:
-  - Extract the following information stored in the header of each particle image.
+  - Takes micrographs and rebox files as input and writes the particle images to a local stack.
-    * source micrograph path (ptcl_source_image).
+  - Normalization is done by subtracting the average of the region outside of a circle with radius half the box size.
-    * CTF parameters if exist (ctf).
+  - Micrographs are applied a Gaussian high-pass filter equivalent to the box size.
-    * projection parameters if exist (xform.projection).
-    * box center coordinates within the micrograph (ptcl_source_coord).
-  - Save the list of extracted micrograph names to an output file.
-  - If provided, apply the selection list to extracted micrographs.
-  - Save the list of selected micrograph names to an output file.
-  - Extract only particle image associating to selected micrographs.
-  - Save the list of selected particle image IDs to an output file.
-  - Save the CTF parameters of selected particle images to output file.
-  - Save the original projection parameters of selected particle images to output file.
-  - Transform the projection parameters of selected particle images  based on user-provided 3D shift, and then save the results to output files.
-  - Convert the center coordinates to EMAN1 box coordinates format, and then save the results to output files.
-  - Transform the coordinates based on the projection parameters and user-provided 3D shift, and then save the results to output files.
-  - Create the output virtual stack if necessary
 \\
@@ Line 92: / Line 76: @@
 \\
 ==== Developer Notes ====
-=== 2018/03/05 Toshio Moriya ===
-Wish list:
-  * Setting %%--%%box_size to zero (default value) should save the coordinates file in SPHIRE format
 \\
 ==== Author / Maintainer ====
-Christos Gatsogiannis and Toshio Moriya
+Toshio Moriya
 \\
 ==== Keywords ====
-Category 1:: APPLICATIONS
+Category 1:: FUNDAMENTALS, UTILITIES, APPLICATIONS
 \\
 ==== Files ====
-sparx/bin/sxpipe.py
+sparx/bin/sp_rewindow.py
 \\
 ==== See also ====
-[[pipeline:window:sxwindow|sxwindow]], [[pipeline:cter:sxcter|sxcter]], [[pipeline:cter:sxgui_cter|sxgui_cter]], [[pipeline:utilities:sxpipe_organize_micrographs|sxpipe organize_micrographs]], [[pipeline:utilities:e2bdb|e2bdb]], and [[pipeline:meridien:sxmeridien|sxmeridien]].
+[[pipeline:utilities:sp_pipe_restacking]], [[pipeline:cter:sp_cter|sp_cter]], and [[pipeline:cter:sp_gui_cter]]
 \\

User Tools

Site Tools

Differences

Page Tools