nextflow scripts
The following scripts are provided as part of the nextflow package:
beamform.nf
help:
beamform.nf: A pipeline that will beamform and splice on all input pointings.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
--publish_fits
Publish to the fits files to the vcs subdirectory.
Pointing arguments (one is required):
--pointings A comma-separated list of pointings with the RA and Dec separated
by _ in the format HH:MM:SS_+DD:MM:SS, e.g.
"19:23:48.53_-20:31:52.95,19:23:40.00_-20:31:50.00" [default: None]
--pointing_file
A file containing pointings with the RA and Dec separated by _
in the format HH:MM:SS_+DD:MM:SS on each line, e.g.
"19:23:48.53_-20:31:52.95\n19:23:40.00_-20:31:50.00" [default: None]
Beamforming types arguments (optional):
--summed Sum the Stoke paramters [default: false]
--incoh Also produce an incoherent beam [default: false]
--ipfb Also produce a high time resolution Inverse Polyphase Filter Bank beam
[default: false]
--offringa Use offringa calibration solution instead of RTS [default: false]
Optional arguments:
--vcstools_version
The vcstools module version to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
benchmark_beamformer.nf
help:
beamform.nf: A pipeline that will beamform and splice on all input pointings.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Optional arguments:
--summed Add this flag if you the beamformer output as summed polarisations
(only Stokes I). This reduces the data size by a factor of 4.
[default: False]
--ipfb Perform an the inverse PFB to produce high time resolution beamformed
vdif files [default: false]
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
candidate_TOAs.nf
help:
candidate_TOAs.nf: A pipeline that will generate pulsar TOAs to be used for timing in tempo2
Argurments:
--fits_file The fits file to process. If this is not supplied, the pipeline will beamform
for you if you have the required beamforming arguments (see beamform.nf -h)
--fits_file_dir
A base directory of all fits files to process. Will search subdirectory to
find fits files.
--std_profile
The standard profile to convolve your pulses with to get the TOAs
[default: /astro/mwavcs/pulsar_search/psr2_timing/psr2_1275178816.profile]
--time_split Split the observation in time of size --subint to get several TOAs
--subint Size in seconds to split the observation into if using --time_split.
If not supplied will make a guesstimate of a reasonable subint with the presto SN.
--chan_split Split the observation in frequency of size --nchan coarse channels
This method isn't well test and may break
--ncchan Number of coarse channels to split the obs into if using --chan_split.
[default: 1]
--eph The ephermis file to fold on the obs. If you don't have one use --period and --dm
--period The topo period of the pulsar in seconds.
--dm The dispersion measure of the pulsar.
--out_dir Where the TOAs will be output
[default: .//no_obsid_candidates/<obsid>]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
classifier.nf
help:
classifier.nf: A pipeline that will use the LOTAAS classifier to sort candidates
into positive and negative detections.
Required argurments:
--cand_dir The directory containing the pfd files [no default]
Optional arguments:
--out_dir The output directory [default: cand_dir]
data_processing_pipeline.nf
help:
beamform_fov_sources.nf: A pipeline that will beamform on all pulsars in the FOV
and perform a search on all pulsar candidates.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Optional arguments:
--search_radius
The radius to search (create beams within) in degrees to account for ionosphere.
[default: 0.00001 degrees (doesn't make a grid)]
--only_cand_search
Only search for pulsar candidates (no known pulsar processing
[default: False]
--publish_fits
Publish to the fits directory (/astro on Galaxy).
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
--out_dir Where the search candidates will be output
[default: .//no_obsid_candidates]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
find_candidate_position.nf
help:
mwa_search_pipeline.nf: A pipeline that will beamform and perform a pulsar search
in the entire FOV.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
--summed Add this flag if you the beamformer output as summed polarisations
(only Stokes I). This reduces the data size by a factor of 4.
[default: False]
Required pointing arguments:
--pointings A comma sepertated list of pointings with the RA and Dec seperated
by _ in the format HH:MM:SS_+DD:MM:SS, e.g.
"19:23:48.53_-20:31:52.95,19:23:40.00_-20:31:50.00" [default: None]
--pointing_file
A file containing pointings with the RA and Dec seperated by _
in the format HH:MM:SS_+DD:MM:SS on each line, e.g.
"19:23:48.53_-20:31:52.95\n19:23:40.00_-20:31:50.00" [default: None]
Pointing grid arguments:
--pointing_grid
Pointing which grid.py will make a loop of pointings around eg.
"19:23:48.53_-20:31:52.95" [default: None]
--fraction The fraction of the FWHM to space the grid by [default: 0.8]
--loops The number of loops of beamd to surround the centre pointing [default: 1]
Presto and dspsr options:
--bins Number of bins to use [default: 128]
--period Period in seconds to fold on [default: 0.90004]
--dm The dispersion measure to use [default: 23.123]
--subint The number of subints to use in pmdp [default: 60]
--nchan The number of subchans to use in pmdp [default: 48]
Optional arguments:
--out_dir Output directory for the candidates files
[default: .//<obsid>_candidates]
--ipfb Perform an the inverse PFB to produce high time resolution beamformed
vdif files [default: false]
--publish_fits
Publish to the fits directory (/group on Galaxy).
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
mwa_search_pipeline.nf
help:
mwa_search_pipeline.nf: A pipeline that will beamform and perform a pulsar search
in the entire FOV.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Pointing arguments (one is required):
--pointings A comma-separated list of pointings with the RA and Dec separated
by _ in the format HH:MM:SS_+DD:MM:SS, e.g.
"19:23:48.53_-20:31:52.95,19:23:40.00_-20:31:50.00" [default: None]
--pointing_file
A file containing pointings with the RA and Dec separated by _
in the format HH:MM:SS_+DD:MM:SS on each line, e.g.
"19:23:48.53_-20:31:52.95\n19:23:40.00_-20:31:50.00" [default: None]
--bestprof_pointings
A directory that contains bestprof files that you would like to
follow up. The pipeline will beamform on their pointings, prepfold
on their DM and period, and perform a blind search. [default: None]
Beamforming types arguments (optional):
--summed Sum the Stoke paramters [default: false]
--incoh Also produce an incoherent beam [default: false]
--ipfb Also produce a high time resolution Inverse Polyphase Filter Bank beam
[default: false]
--offringa Use offringa calibration solution instead of RTS [default: false]
Dedispersion arguments (optional):
--dm_min Minimum DM to search over [default: 1]
--dm_max Maximum DM to search over [default: 250]
--dm_min_step
Minimum DM step size (Delta DM) [default: 0.02]
--dm_max_step
Maximum DM step size (Delta DM) [default: 0.5]
--max_dms_per_job
Maximum number of DM steps a single job will procces.
Lowering this will reduce memory usage and increase parellelisation.
[default: 5000]
Pulsar search arguments (optional):
--sp Perform only a single pulse search [default: false]
--cand Label given to output files [default: Blind]
--nharm Number of harmonics to search [default: 16]
--min_period
Min period to search for in sec (ANTF min = 0.0013)
[default: 0.001]
--max_period
Max period to search for in sec (ANTF max = 23.5)
[default: 30]
--zmax Maximum acceleration to search (0 will do a simpler periodic search).
I recomend you use 200 and set --max_dms_per_job 32
[default: 0]
Other arguments (optional):
--out_dir Output directory for the candidates files
[default: .//no_obsid_candidates]
--publish_fits
Publish to the fits files to the vcs subdirectory.
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
pdmp.nf
help:
Script compilation error
- file : /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/nextflow/pdmp.nf
- cause: Workflow 'publish' is not supported anymore use process 'publishDir' instead @ line 19, column 9.
pdmp_wf.out to: params.out_dir
^
Script compilation error
- file : /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/nextflow/pdmp.nf
- cause: Workflow malformed parameter definition @ line 19, column 9.
pdmp_wf.out to: params.out_dir
^
2 errors
pulsar_search.nf
help:
pulsar_search.nf: A pipeline perform a pulsar search on a single input fits file.
The fits files must be in the format
<obsid>_<pointing>_ch<min_chan>-<max_chan>_00??.fits
Required argurments:
--fits_file The fits file to search [no default]
Dedispersion arguments (optional):
--dm_min Minimum DM to search over [default: 1]
--dm_max Maximum DM to search over [default: 250]
--dm_min_step
Minimum DM step size (Delta DM) [default: 0.02]
--dm_max_step
Maximum DM step size (Delta DM) [default: 0.5]
--max_dms_per_job
Maximum number of DM steps a single job will process.
Lowering this will reduce memory usage and increase parellelisation.
[default: 5000]
Pulsar search arguments (optional):
--sp Perform only a single pulse search [default: false]
--cand Label given to output files [default: Blind]
--nharm Number of harmonics to search [default: 16]
--min_period
Min period to search for in sec (ANTF min = 0.0013)
[default: 0.001]
--max_period
Max period to search for in sec (ANTF max = 23.5)
[default: 30]
--zmax Maximum acceleration to search (0 will do a simpler periodic search).
I recomend you use 200 and set --max_dms_per_job 32
[default: 0]
Optional arguments:
--cand Candidate name to do a targeted search [default: Blind]
--out_dir Output directory for the candidates files
[default: .//<obsid>_candidates]
--mwa_search_version
The mwa_search module bersion to use [default: master]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
sun_signal_search.nf
help:
WARN: Access to undefined parameter `fits_file_dir` -- Initialise it to a default value eg. `params.fits_file_dir = some_value`
WARN: Access to undefined parameter `fits_files` -- Initialise it to a default value eg. `params.fits_files = some_value`
WARN: Access to undefined parameter `scratch` -- Initialise it to a default value eg. `params.scratch = some_value`
pulsar_search.nf: A pipeline perform a pulsar search on the input fits files.
The fits files must be in the format
<obsid>_<pointing>_ch<min_chan>-<max_chan>_00??.fits
Required argurments:
--obsid Observation ID you want to process [no default]
--pointings The pointing to search in with the RA and Dec seperated
by _ in the format HH:MM:SS_+DD:MM:SS
Optional arguments:
--cand Candidate name to do a targeted search [default: Blind]
--sp Perform a single pulse search [default false]
--fits_file_dir
Directory containing the fits files. Use this if the fits files
are not in the default directory :
.//<obsid>/pointings/null
--scratch Change the default directory to:
.//<obsid>/pointings/null
--dm_min Minimum DM to search over [default: 1]
--dm_max Maximum DM to search over [default: 250]
--dm_min_step
Minimum DM step size (Delta DM) [default: 0.1]
--out_dir Output directory for the candidates files
[default: .//<obsid>_candidates]
--mwa_search_version
The mwa_search module bersion to use [default: master]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
test_beamformer.nf
help:
test_beamformer.nf: A pipeline that will test the beamformer by comparing the binaires of the output psrfits
and vdif files with previous runs and the signal-to-noise of pulsar detections
Optional arguments:
--vcstools_version
The vcstools module version to use [default: master]
--version_compare
The beamformer version to compare the current beamformer with.
The previous runs are contained in .//beamformer_tests
[default: V2.3_FEE2016]
--publish_version
Publish the current beamformer run to .//beamformer_tests so it can be compared in future runs.
Make sure the --version_compare is different than any other previous runs.
[default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]
vcs_download.nf
help:
vcs_download.nf: A pipeline that will download vcs data and untar or recombine it if required.
Required argurments:
--obsid Observation ID you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Optional arguments:
--download_dir
The directory of already downloaded data (with the ASVO).
--increment Increment in seconds (how much we process at once). [default: 64]
--max_jobs Number of maximum jobs of each type to run at once (to limit IO). [default: 12]
--parallel_dl
Number of parallel downloads to envoke. [default: 3]
--untar_jobs
Number of parallel jobs when untaring downloaded tarballs. [default: 2]
--keep_tarball
Keep the tarballs after unpacking. [default: false]
--keep_raw
Keep the raw data after recombining. [default: false]
--vcstools_version
The vcstools module version to use [default: master]
--mwa_voltage_version
The mwa-voltage module version to use [default: master]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/latest/docs/work]