nextflow scripts
The following scripts are provided as part of the nextflow package:
beamform.nf
help:
WARN: Access to undefined parameter `scratch_basedir` -- Initialise it to a default value eg. `params.scratch_basedir = some_value`
beamform.nf: A pipeline that will beamform and splice on all input pointings.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--pointings A comma sepertated list of pointings with the RA and Dec seperated
by _ in the format HH:MM:SS_+DD:MM:SS, e.g.
"19:23:48.53_-20:31:52.95,19:23:40.00_-20:31:50.00" [default: None]
--pointing_file
A file containing pointings with the RA and Dec seperated by _
in the format HH:MM:SS_+DD:MM:SS on each line, e.g.
"19:23:48.53_-20:31:52.95\n19:23:40.00_-20:31:50.00" [default: None]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
--publish_fits
Publish to the fits directory (/group on Galaxy). Include this
option.
Optional arguments:
--summed Add this flag if you the beamformer output as summed polarisations
(only Stokes I). This reduces the data size by a factor of 4.
[default: False]
--ipfb Perform an the inverse PFB to produce high time resolution beamformed
vdif files [default: false]
--publish_fits_scratch
Publish to the scratch fits directory (/astro on Galaxy). Use this
instead of --publish_fits_scratch
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]
benchmark_beamformer.nf
help:
WARN: Access to undefined parameter `scratch_basedir` -- Initialise it to a default value eg. `params.scratch_basedir = some_value`
beamform.nf: A pipeline that will beamform and splice on all input pointings.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Optional arguments:
--summed Add this flag if you the beamformer output as summed polarisations
(only Stokes I). This reduces the data size by a factor of 4.
[default: False]
--ipfb Perform an the inverse PFB to produce high time resolution beamformed
vdif files [default: false]
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]
candidate_TOAs.nf
help:
WARN: Access to undefined parameter `max_pointings` -- Initialise it to a default value eg. `params.max_pointings = some_value`
WARN: Access to undefined parameter `bm_read` -- Initialise it to a default value eg. `params.bm_read = some_value`
WARN: Access to undefined parameter `bm_cal` -- Initialise it to a default value eg. `params.bm_cal = some_value`
WARN: Access to undefined parameter `bm_beam` -- Initialise it to a default value eg. `params.bm_beam = some_value`
ERROR ~ Cannot execute null+null
-- Check script '/home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/nextflow/./beamform_module.nf' at line: 61 or see '.nextflow.log' file for more details
classifier.nf
help:
classifier.nf: A pipeline that will use the LOTAAS classifier to sort candidates
into positive and negative detections.
Required argurments:
--cand_dir The directory containing the pfd files [no default]
Optional arguments:
--out_dir The output directory [default: cand_dir]
data_processing_pipeline.nf
help:
WARN: Access to undefined parameter `scratch_basedir` -- Initialise it to a default value eg. `params.scratch_basedir = some_value`
WARN: Access to undefined parameter `search_dir` -- Initialise it to a default value eg. `params.search_dir = some_value`
beamform_fov_sources.nf: A pipeline that will beamform on all pulsars in the FOV
and perform a search on all pulsar candidates.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Optional arguments:
--search_radius
The radius to search (create beams within) in degrees to account for ionosphere.
[default: 0.00001 degrees (doesn't make a grid)]
--only_cand_search
Only search for pulsar candidates (no known pulsar processing
[default: False]
--publish_fits
Publish to the fits directory (/group on Galaxy). Use this instead
of --publish_fits_scratch
--publish_fits_scratch
Publish to the scratch fits directory (/astro on Galaxy). Include
this option.
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
--out_dir Where the search candidates will be output
[default: null/no_obsid_candidates]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]
find_candidate_position.nf
help:
WARN: Access to undefined parameter `search_dir` -- Initialise it to a default value eg. `params.search_dir = some_value`
mwa_search_pipeline.nf: A pipeline that will beamform and perform a pulsar search
in the entire FOV.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
--summed Add this flag if you the beamformer output as summed polarisations
(only Stokes I). This reduces the data size by a factor of 4.
[default: False]
Required pointing arguments:
--pointings A comma sepertated list of pointings with the RA and Dec seperated
by _ in the format HH:MM:SS_+DD:MM:SS, e.g.
"19:23:48.53_-20:31:52.95,19:23:40.00_-20:31:50.00" [default: None]
--pointing_file
A file containing pointings with the RA and Dec seperated by _
in the format HH:MM:SS_+DD:MM:SS on each line, e.g.
"19:23:48.53_-20:31:52.95\n19:23:40.00_-20:31:50.00" [default: None]
Pointing grid arguments:
--pointing_grid
Pointing which grid.py will make a loop of pointings around eg.
"19:23:48.53_-20:31:52.95" [default: None]
--fraction The fraction of the FWHM to space the grid by [default: 0.8]
--loops The number of loops of beamd to surround the centre pointing [default: 1]
Presto and dspsr options:
--bins Number of bins to use [default: 128]
--period Period in seconds to fold on [default: 0.90004]
--dm The dispersion measure to use [default: 23.123]
--subint The number of subints to use in pmdp [default: 60]
--nchan The number of subchans to use in pmdp [default: 48]
Optional arguments:
--out_dir Output directory for the candidates files
[default: null/<obsid>_candidates]
--ipfb Perform an the inverse PFB to produce high time resolution beamformed
vdif files [default: false]
--publish_fits
Publish to the fits directory (/group on Galaxy).
--publish_fits_scratch
Publish to the scratch fits directory (/astro on Galaxy).
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]
mwa_search_pipeline.nf
help:
WARN: Access to undefined parameter `scratch_basedir` -- Initialise it to a default value eg. `params.scratch_basedir = some_value`
WARN: Access to undefined parameter `search_dir` -- Initialise it to a default value eg. `params.search_dir = some_value`
mwa_search_pipeline.nf: A pipeline that will beamform and perform a pulsar search
in the entire FOV.
Required argurments:
--obsid Observation ID you want to process [no default]
--calid Observation ID of calibrator you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Pointing arguments (one is required):
--pointings A comma-separated list of pointings with the RA and Dec separated
by _ in the format HH:MM:SS_+DD:MM:SS, e.g.
"19:23:48.53_-20:31:52.95,19:23:40.00_-20:31:50.00" [default: None]
--pointing_file
A file containing pointings with the RA and Dec separated by _
in the format HH:MM:SS_+DD:MM:SS on each line, e.g.
"19:23:48.53_-20:31:52.95\n19:23:40.00_-20:31:50.00" [default: None]
--bestprof_pointings
A directory that contains bestprof files that you would like to
follow up. The pipeline will beamform on their pointings, prepfold
on their DM and period, and perform a blind search. [default: None]
Dedispersion arguments (optional):
--dm_min Minimum DM to search over [default: 1]
--dm_max Maximum DM to search over [default: 250]
--dm_min_step
Minimum DM step size (Delta DM) [default: 0.02]
--dm_max_step
Maximum DM step size (Delta DM) [default: 0.5]
--max_dms_per_job
Maximum number of DM steps a single job will procces.
Lowering this will reduce memory usage and increase parellelisation.
[default: 5000]
Optional arguments:
--summed Add this flag if you the beamformer output as summed polarisations
(only Stokes I). This reduces the data size by a factor of 4.
[default: False]
--zmax The acceleration range to search over. If you would like to perform
an acceleration search I recomend you use 200 and set
--max_dms_per_job 32
[default: 0 (no acceleration search)]
--out_dir Output directory for the candidates files
[default: null/<obsid>_candidates]
--ipfb Perform an the inverse PFB to produce high time resolution beamformed
vdif files [default: false]
--publish_fits
Publish to the fits directory (/group on Galaxy).
--publish_fits_scratch
Publish to the scratch fits directory (/astro on Galaxy).
--vcstools_version
The vcstools module version to use [default: master]
--mwa_search_version
The mwa_search module bersion to use [default: master]
--no_combined_check
Don't check if all the combined files are available [default: false]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]
pdmp.nf
help:
ERROR ~ Script compilation error
- file : /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/nextflow/pdmp.nf
- cause: Workflow 'publish' is not supported anymore use process 'publishDir' instead @ line 27, column 9.
pdmp_wf.out to: params.out_dir
^
Script compilation error
- file : /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/nextflow/pdmp.nf
- cause: Workflow malformed parameter definition @ line 27, column 9.
pdmp_wf.out to: params.out_dir
^
2 errors
-- Check '.nextflow.log' file for details
pulsar_search.nf
help:
WARN: Access to undefined parameter `search_dir` -- Initialise it to a default value eg. `params.search_dir = some_value`
pulsar_search.nf: A pipeline perform a pulsar search on a single input fits file.
The fits files must be in the format
<obsid>_<pointing>_ch<min_chan>-<max_chan>_00??.fits
Required argurments:
--obsid Observation ID you want to process [no default]
--fits_file The fits file to search [no default]
--dur Duration of the fits file in seconds [no default]
Dedispersion arguments (optional):
--dm_min Minimum DM to search over [default: 1]
--dm_max Maximum DM to search over [default: 250]
--dm_min_step
Minimum DM step size (Delta DM) [default: 0.02]
--dm_max_step
Maximum DM step size (Delta DM) [default: 0.5]
--max_dms_per_job
Maximum number of DM steps a single job will process.
Lowering this will reduce memory usage and increase parellelisation.
[default: 5000]
Optional arguments:
--cand Candidate name to do a targeted search [default: Blind]
--sp Perform a single pulse search [default false]
--zmax The acceleration range to search over. If you would like to perform
an acceleration search I recomend you use 200 and set
--max_dms_per_job 32
[default: 0 (no acceleration search)]
--out_dir Output directory for the candidates files
[default: null/<obsid>_candidates]
--mwa_search_version
The mwa_search module bersion to use [default: master]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]
sun_signal_search.nf
help:
WARN: Access to undefined parameter `fits_files` -- Initialise it to a default value eg. `params.fits_files = some_value`
WARN: Access to undefined parameter `search_dir` -- Initialise it to a default value eg. `params.search_dir = some_value`
WARN: Access to undefined parameter `basedir` -- Initialise it to a default value eg. `params.basedir = some_value`
WARN: Access to undefined parameter `scratch_basedir` -- Initialise it to a default value eg. `params.scratch_basedir = some_value`
pulsar_search.nf: A pipeline perform a pulsar search on the input fits files.
The fits files must be in the format
<obsid>_<pointing>_ch<min_chan>-<max_chan>_00??.fits
Required argurments:
--obsid Observation ID you want to process [no default]
--pointings The pointing to search in with the RA and Dec seperated
by _ in the format HH:MM:SS_+DD:MM:SS
Optional arguments:
--cand Candidate name to do a targeted search [default: Blind]
--sp Perform a single pulse search [default false]
--fits_file_dir
Directory containing the fits files. Use this if the fits files
are not in the default directory :
null/<obsid>/pointings/null
--scratch Change the default directory to:
null/<obsid>/pointings/null
--dm_min Minimum DM to search over [default: 1]
--dm_max Maximum DM to search over [default: 250]
--dm_min_step
Minimum DM step size (Delta DM) [default: 0.1]
--out_dir Output directory for the candidates files
[default: null/<obsid>_candidates]
--mwa_search_version
The mwa_search module bersion to use [default: master]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]
test_beamformer.nf
help:
WARN: Access to undefined parameter `bm_read` -- Initialise it to a default value eg. `params.bm_read = some_value`
WARN: Access to undefined parameter `bm_cal` -- Initialise it to a default value eg. `params.bm_cal = some_value`
WARN: Access to undefined parameter `bm_beam` -- Initialise it to a default value eg. `params.bm_beam = some_value`
ERROR ~ Cannot execute null+null
-- Check script '/home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/nextflow/test_beamformer.nf' at line: 17 or see '.nextflow.log' file for more details
vcs_download.nf
help:
vcs_download.nf: A pipeline that will download vcs data and untar or recombine it if required.
Required argurments:
--obsid Observation ID you want to process [no default]
--begin First GPS time to process [no default]
--end Last GPS time to process [no default]
--all Use entire observation span. Use instead of -b & -e. [default: false]
Optional arguments:
--download_dir
The directory of already downloaded data (with the ASVO).
--increment Increment in seconds (how much we process at once). [default: 64]
--max_jobs Number of maximum jobs of each type to run at once (to limit IO). [default: 12]
--parallel_dl
Number of parallel downloads to envoke. [default: 3]
--untar_jobs
Number of parallel jobs when untaring downloaded tarballs. [default: 2]
--keep_tarball
Keep the tarballs after unpacking. [default: false]
--keep_raw
Keep the raw data after recombining. [default: false]
--vcstools_version
The vcstools module version to use [default: master]
--mwa_voltage_version
The mwa-voltage module version to use [default: master]
-w The Nextflow work directory. Delete the directory once the processs
is finished [default: /home/docs/checkouts/readthedocs.org/user_builds/mwa-search/checkouts/development/docs/work]