dpp modules
helper_relaunch
helper_archive
- dpp.helper_archive.archive_fit(cfg, archive_path, cliptype='verbose')[source]
Fits a profile to the supplied archive and adds it to cfg. Cliptype options found in prof_utils.py
- dpp.helper_archive.archive_to_fits(ar_file, extension='fits', container='/pawsey/mwa/singularity/dspsr/dspsr.sif')[source]
Returns bash commands to turn an arhive file to a fits file
- dpp.helper_archive.fits_to_archive(fits_dir, ar_name, bins, dm, period, out_dir, memory=4000, total=1.0, seek=0, vdif=False, container='/pawsey/mwa/singularity/dspsr/dspsr.sif')[source]
Returns bash commands to fold on a fits file using dspsr
helper_RVMfit
plotting_toolkit
- exception dpp.plotting_toolkit.NoEPNDBError[source]
Raise when a pulsar has not been found on the EPNDB
- dpp.plotting_toolkit.add_ascii_to_dict(pulsar_dict, ascii_archive, freq)[source]
Adds an ascii archive to a pulsar dict generated from get_data_from_epndb
- Parameters:
- pulsar_dict: dictionary
A dictionary generated from get_data_from_epndb
- ascii_archive: string
The pathname of the ascii archive to use
- freq: float
The frequency of the observation in MHz
- Returns:
- pulsar_dict: dictionary
The input dictionary with the new information added and sorted by frequency
- lin_pol: list
The linear poilarisation of the ascii archive
- dpp.plotting_toolkit.add_intensity_to_dict(pulsar_dict, profile, freq)[source]
Adds a stokes I profile to the pulsar dictionary
- Parameters:
- pulsar_dict: dictionary
A dictionary generated by get_data_from_epndb
- profile: list
The stokes I profile to add to the pulsar dictionary
- freq: float
The frequency of the profile being added in MHz
- Returns:
- pulsar_dict: dictionary
The same dictionary but with the new profile added and sorted
- dpp.plotting_toolkit.calc_lin_pa(stokes_Q, stokes_U)[source]
Calculates the linear polsarization and position angle from stokes Q and U components
- Parameters:
- stokes_Q: list
A list of the Stokes Q values
- stokes_U: list
A list of the Stokes U values
- Returns:
- lin_pol: list
The linear polarization
- pa: list
The position angles
- dpp.plotting_toolkit.clip_nopol_epn_data(pulsar_dict)[source]
Deletes any profile without enough information for polarimetry
- Parameters:
- pulsar_dict: dictionary
A dictionary generated from get_data_from_epndb
- Returns:
- pulsar_dict: dictionary
The same dictionary but with the insufficient profiles removed
- dpp.plotting_toolkit.get_data_from_epndb(pulsar)[source]
Searches the EPN database and returns all information of the chosen pulsar
- Parameters:
- pulsar: string
The name of the pulsar to search for
- Returns:
- pulsar_dict: dictionary
A dictionary in which each value is a list corresponding to a different entry on the databse. Keys:
- I: list
An list of stokes I values for the profile
- Q: list
An list of stokes Q values for the profile
- U: list
An list of stokes U values for the profile
- V: list
An list of stokes V values for the profile
- freq: float
The frequency of the observation in MHz
- dm: float
The measured dispersion measure
- rm: float
The measured rotation measure. Returns 0.0 for no rm measurement
- site: string
The location the pulsar was observed at
- dpp.plotting_toolkit.lin_pol_from_dict(pulsar_dict)[source]
Calculates the linear polarisation of eachprofile in the pulsar dict
- Parameters:
- pulsar_dict: dictionary
A dictionary generated from get_data_from_epndb
- Returns:
- lin: list
A list containing the linear polarisation profile for each profile in pulsar_dict
- dpp.plotting_toolkit.plot_archive_stokes(archive, pulsar=None, freq=None, obsid=None, out_dir='./', rvm_fit=None, rm=None, rm_e=None)[source]
Plots a polarimetry profile as a .png using a full-stokes ascii text file
- Parameters:
- archive: string
The path to the ascii text file
- pulsar: string
OPTIONAL - The name of the pulsar. Default: None
- freq: float
OPTIONAL - The frequency of the observation. Default: None
- obsid: int
OPTIONAL - The observation ID. Default: None
- out_dir: string
OPTIONAL - The directory to ouput the .png to
- rvm_fit: dictionary
OPTIONAL - A dictionary generated from stokes_fold.read_rvm_fit_file. Supplying this will plot the fitted RVM. Default: None
- rm: float
OPTIONAL - The rm used to correct the plot. If supplied will create a stamp on the plot. Default: None
- rm_e: float
OPTIONAL - The uncertainty in rm. Default: None
- Returns:
- fig_path: string
The path of the output .png file
- dpp.plotting_toolkit.plot_bestprof(bestprof, freq=None, out_dir='./')[source]
Plots a .pfd.bestprof file from a presto output and saves as a .png
- Parameters:
- bestprof: string
The path to the bestprof file
- freq: float
OPTIONAL - The frequency of the observation. Default: None
- out_dir: string
OPTIONAL - The directory to save the file to. Default: ‘./’
- Returns:
- fig_path: string
The path of the .png plot
- dpp.plotting_toolkit.plot_profile(I, pulsar=None, freq=None, obsid=None, out_dir='./')[source]
Plots an ascii text file and saves as a .png
- Parameters:
- I: list
The list of stokes I values to plot
- pulsar: string
OPTIONAL - The name of the pulsar
- freq: float
OPTIONAL - The frequency of the observation. Default: None
- obsid: int
OPTIONAL - The ID of the MWA observation. Default: None
- out_dir: string
OPTIONAL - The directory to save the file to. Default: ‘./’
- Returns:
- fig_path: string
The path of the .png plot
- dpp.plotting_toolkit.plot_rvm_chi_map(chis, alphas, betas, name='RVM_chi_map_plot.png', my_chi=None, my_alpha=None, my_beta=None)[source]
Plots a chi map generated from RVM fitting
- Parameters:
- chi: list
A lsit of the chisquare values
- alpha: list
A list of the alpha values
- beta: list
A list of the beta values
- name: string
OPTIONAL - The pathname of the output plot. Defalt: ‘RVM_chi_map_plot.png’
- dof: float
OPTIONAL - The degrees of freedom. Used to display a reduces chi
- Returns:
- name: string
The pathname of the ouput plot
- dpp.plotting_toolkit.plot_stack(frequencies, profs_y, pulsar_name, out_dir='./', mybuffer=0.75, ignore_duplicates=True, special_freqs=None, ignore_freqs=None, label='')[source]
Plots multiple profiles stacked on top of one anothre in order of frequency. Saves as a .png
- Parameters:
- frequencies: list
The frequencies of the profiles to plot
- profs_y: list
The Intesities of the profiles
- pulsar_name: string
The name of the pulsar
- out_dir: string
OPTIONAL - The directory to output the .png to. Default: ‘./’
- mybuffer: float
OPTIONAL - The separation in intensity between profiles. Default:0.75
- ignore_dupicates: boolean
OPTIONAL - If True, will not plot duplicate frequencies. Default: True
- special_freqs: list
OPTIONAL - Any frequencies to be highlighted in the plot. Default: None
- ignore_freqs: list
OPTIONAL - Any frequencies to not plot. Default: None
- label: string
OPTIONAL - A string to identify the output file. Default: ‘’
- Returns:
- fig_name: string
The path of the saved .png
- dpp.plotting_toolkit.plot_stack_pol(frequencies, I_y, lin_y, circ_y, pulsar_name, out_dir='./', mybuffer=1.1, ignore_duplicates=True, ignore_freqs=None, label='')[source]
Plots multiple profiles stacked on top of one anothre in order of frequency. Saves as a .png
- Parameters:
- frequencies: list
The list of frequencies in MHz
- I_y: list
A list containing the stokes I intensities for each frequency
- lin_y: list
A list containing the linear polarisation for each frequency
- circ_y: list
A list containing the circular polarisation for each frequency
- pulsar_name: string
The J name of the pulsar
- out_dir: string
OPTIONAL - The directory to output the .png to. Default: ‘./’
- mybuffer: float
OPTIONAL - The separation in intensity between profiles. Default:0.75
- ignore_dupicates: boolean
OPTIONAL - If True, will not plot duplicate frequencies. Default: True
- special_freqs: list
OPTIONAL - Any frequencies to be highlighted in the plot. Default: None
- ignore_freqs: list
OPTIONAL - Any frequencies to not plot. Default: None
- label: string
OPTIONAL - A string to identify the output file. Default: ‘’
- Returns:
- fig_name: string
The path of the saved .png
- dpp.plotting_toolkit.read_ascii_archive(archive, roll=True, norm=True)[source]
Reads an ascii archive and calculates linear polarisation. Will also calculate PA if it’s not in the archive.
- Parameters:
- archive: string
The pathname of the ascii archive
- roll: boolean
If True, will align the archive at the centre
- norm: boolean
If True, will normalise the archive wrt stokes I
- Returns:
- sI: list
The stokes I values of the archive
- sQ: list
The stokes Q values of the archive
- sU: list
The stokes U values of the archive
- sV: list
The stokes V values of the archive
- lin_pol: list
The linear polarisation of the stokes values
- pa: list
Tries to find in the archive, if not there calculates from stokes values. Returns are in radians
- pa_err: list
If pa is in archive, this is the error in the pa. Otherwise empty
- dpp.plotting_toolkit.roll_data(data, idx_to_roll=None, roll_to=None)[source]
Rolls a list of data to about some index to some new index.
- Parameters:
- data: list
The data to roll
- idx_to_roll: int
OPTIONAL - The index number of the original array to roll. If None, uses max value index. Default: None
- roll_to: int
OPTIONAL - The index to move the ‘idx_to_roll’ to. If None, rolls to centre of list. Default: None
helper_source_info
- dpp.helper_source_info.bin_sampling_limit(pulsar, sampling_rate=0.0001, query=None)[source]
Finds the sampling limit of the input pulsar in units of number of bins
helper_classify
- exception dpp.helper_classify.ClassifierFilesNotFoundError[source]
Raise when no classifier files are found
- exception dpp.helper_classify.InvalidClassifyBinsError[source]
Raise when the bin count for the classifier inputs is incorrect
- dpp.helper_classify.add_classify_to_commands(cfg, container='/pawsey/mwa/singularity/lofar_pulsar_ml/lofar_pulsar_ml.sif')[source]
Makes the classify commands
helper_obs_info
- dpp.helper_obs_info.apply_offset(pointing_list, offset, angle_offset)[source]
Apply an offset to the input list of pointings
- Parameters:
- pointing_list: list
A list of pointings where each pointing contains an RA and a Dec in the format ‘hh:mm:ss.ss_dd:mm:ss.ss’
- offset: float
The offset to apply to all pointings in arcseconds
- angle_offset: float
The angle of the offset to apply to all pointings in degrees where zero is north
- Returns:
- offset_pointing_list: list
A list of pointings where each pointing contains an RA and a Dec in the format ‘hh:mm:ss.ss_dd:mm:ss.ss’
- dpp.helper_obs_info.find_beg_end(obsid, base_path=None)[source]
looks through the comined files of the obsid to find the beginning and end gps times
- Parameters:
- obsid: int
The observation ID
- base_path: string
OPTIONAL - The system’s base working pat
- Returns:
- beg: int
The beginning time for on-disk files
- end: int
The end time for on-disk files
- dpp.helper_obs_info.find_fold_times(pulsars, obsid, beg, end, min_z_power=(0.3, 0.1), metadata=None, full_meta=None, query=None)[source]
Finds the fractional time the pulsar is in the beam at some zenith normalized power
- Parameters:
- pulsar: list
Pulsar J names of pulsars to evaluate
- obsid: int
The observation ID
- beg: int
The beginning of the observation time in gps time
- end: int
The end of the observation time in gps time
- min_z_power: tuple/list
OPTIONAL - evaluated the pulsar as ‘in the beam’ at this normalized zenith power. If None will use [0.3, 0.1] Default: None
- Returns:
- fold_times_dict: dict
- keys:
- psr: dict
- keys:
- power: float
The power of the recorded enter and leave times
- enter: float
The normalised time the pulsar enters the beam
- exit: float
The normalisd time the pulsar leaves the beam
- dpp.helper_obs_info.find_pulsars_in_fov(obsid, psrbeg, psrend, fwhm=None, search_radius=0.02, meta_data=None, full_meta=None, no_known_pulsars=False, no_search_cands=False, offset=0, angle_offset=0)[source]
Find all pulsars in the field of view and return all the pointings sorted into vdif and normal lists:
- Parameters:
- obsid: int
The observation ID
- psrbeg: int
The begining of the observation you are processing in GPS time
- psrend: int
The end of the observation you are processing in GPS time
- fwhm: float
OPTIONAL - The FWHM of the beam in degrees. Default None: Value will be estimated
- search_radius: float
OPTIONAL - The radius to search (create beams within) in degrees to account for ionosphere. Default: 0.02 degrees
- meta_data: list
OPTIONAL - Comon observation metadata in the format from vcstools.metadb_utils.get_common_obs_metadata Default None: Will perform the metadata call
- full_meta: dict
OPTIONAL - Full observation metadata in the format from vcstools.metadb_utils.getmeta Default None: Will perform the metadata call
- no_known_pulsars: bool
OPTIONAL - Will return no known pulsars Default: False
- no_search_cands: bool
OPTIONAL - Will return no search candidates Default: False
- Returns:
- list of lists:
- [pulsar_name_list,
pulsar_pointing_list, vdif_name_list, vdif_pointing_list, sp_name_list, sp_pointing_list]
- dpp.helper_obs_info.find_pulsars_power(obsid, powers=None, names_ra_dec=None, metadata_list=None)[source]
Finds the beam power information for pulsars in a specific obsid
- Parameters:
- obsid: int
The observation ID
- powers: list/tuple
OPTIONAL - A list of minimum beam powers to evaluate the pulsar coverage at. If none, will use [0.3, 0.1]. Default: None
- names_ra_dec: list
- OPTIONAL - A list of puslars and their RA and Dec values to evaluate (generated from get_source_alog).
If none, will look for all pulsars. Default: None
- metadata: list
A list of the output of get_common_obs_metadata for the input obsid
- Returns:
- pulsar_power_dict: dictionary
- Contains keys - power
- Contains key - obsid
- Contains one list for each pulsar found in that power
Each list is constructed as [jname, enter, exit, max_power]
- meta_data: list
A list of the output of get_common_obs_metadata for the input obsid
- dpp.helper_obs_info.get_pointings_required(source_ra, source_dec, fwhm, search_radius)[source]
Gets the number of grid pointings required to cover the search radius
- Parameters:
- source_ra, source_dec: string
A string separated representing the RA and dec respectively. Expected format is ‘hh:mm[:ss.s]’
- fwhm: float
FWHM of the tied-array beam in degrees. Can be calculated in the calc_ta_fwhm function
- search_radius: float
The radius of the circle that you would like to search
- Returns:
- pointing_list_list: list of lists
A list of pointings where each pointing contains an RA and a Dec in the format ‘hh:mm:ss.ss’ [[RA, Dec]]
- dpp.helper_obs_info.get_sources_in_fov(obsid, source_type, fwhm)[source]
Find all sources of the input type in the observations field-of-view
- Parameters:
- obsid: str
observation ID to search in
- source_type: str
the source type input to grab_source_alog()
- fwhm: float
FWHM of the tied-array beam in degrees. Can be calculated in the calc_ta_fwhm function
- Returns:
- list:
- name_list: list
A list of pulsars in the FOV
- pointing_list: list
A list of pointings corresponding to the pulsars in name_list
helper_logging
- class dpp.helper_logging.LoggerWriter(writer)[source]
I don’t know how this works, I got it from stackoverflow It’s used to redirect stderr/stdout to a logging instance https://stackoverflow.com/questions/47325506/making-python-loggers-log-all-stdout-and-stderr-messages
Methods
flush
write
helper_RM
helper_database
helper_files
- dpp.helper_files.clean_cfg(cfg)[source]
Remove any lists in the cfg that are empty and deletes the directories
- dpp.helper_files.file_precursor(kwargs, psr)[source]
Creates a common precursor string for files and directories using kwargs from observation_processing_pipeline.py and a pulsar name
- dpp.helper_files.find_config_files(obsid, label='')[source]
Searches the obsid/dpp directories to find any config (.yaml) files
helper_checks
- exception dpp.helper_checks.FitsNotFoundError[source]
Raise when fits files are not found in pointing directory
- exception dpp.helper_checks.InvalidPAFileError[source]
Raise when a paswing file is not valid for any reason
- exception dpp.helper_checks.PointingNotFoundError[source]
Raise when a pointing directory is not found
- dpp.helper_checks.check_all_beamformed_fits(cfg)[source]
Checks that fits files exist in all pointings Raises FileNotFoundError
- dpp.helper_checks.check_file_dir_exists(file_dir)[source]
Checks if a file or directory exists Raises FileNotFoundError
- dpp.helper_checks.check_paswing(paswing_file)[source]
Checks existence of paswing file and if all PA points in a paswing file are identical. Raises InvalidFileError
- dpp.helper_checks.check_pdirs_exist(cfg)[source]
Checks that the required pointing directory symlinks exist. If not, attempts to create them. Raises PointingNotFoundError
helper_terminate
helper_prepfold
- dpp.helper_prepfold.add_prepfold_to_commands(prep_kwargs, psr_name, pointing, eph=None, eph_name=None, presto_container='', binary=False)[source]
Adds prepfold commands to a list. If eph is not None, will use -par
- dpp.helper_prepfold.common_kwargs(cfg, bin_count, pointing)[source]
Creates a prepfold-friendly dictionary of common arguments to pass to prepfold
- dpp.helper_prepfold.initial_folds(cfg)[source]
Loops through the required initial folds and submits them to slurm
- dpp.helper_prepfold.post_folds(cfg)[source]
Loops through the required post folds and submits them to slurm
helper_status
- dpp.helper_status.cfg_status(psr_dir)[source]
Checks a cfg to see how it ended and returns the status code
- dpp.helper_status.message_from_status(status)[source]
Takes a status code and returns the defualt error message for it
helper_bestprof
- exception dpp.helper_bestprof.NoUsableFoldsError[source]
Raise when no usable folds are found in a pipe
- dpp.helper_bestprof.bestprof_fit(cfg, cliptype='verbose')[source]
Fits a profile to the best bestprof and adds it to cfg. Cliptype options found in prof_utils.py
- dpp.helper_bestprof.bestprof_info(filename)[source]
Finds various information on a .bestprof file
- Parameters:
- filename: string
The path of the bestprof file
- Returns:
- info_dict: dictionary
A dictionary consisting of the following:
- obsid: int
The ID of the observation
- puslar: string
The J name of the pulsar
- nbins: int
The number of bins used to fold this profile
- chi: float
The reduced Chi squared value of the fold
- sn: float
The signal to noise ratio of the fold
- dm: float
The pulsar’s dispersion measure
- period: float
The pulsar’s period
- period_error: float
The error in the pulsar’s period measurement
- dpp.helper_bestprof.classify_init_bestprof(cfg)[source]
Determines whether an iniital fold is a detection based on its PRESTO output
helper_config
- dpp.helper_config.create_cfgs_main(kwargs, psrs_pointing_dict)[source]
uses kwargs from observation_processing_pipeline.py
- dpp.helper_config.create_edited_eph(pulsar_name)[source]
Created a string version of ‘psrcat -e2’ and removes the last line