Imaging

Attention

MeasurementSets produced by the ASKAPsoft pipeline need modification before using tools like WSClean. This can be done using FixMS, which is called internally by Arrakis.

Arrakis provides an interface to the WSClean imaging software, with convencince functions for imaging mutliple ASKAP beams simultaneously. There are two main interaces for running the imaging pipeline:

The spice_image CLI and API

Attention

This will only run using a sequential Prefect task runner. i.e. Only one beam will be imaged at a time. See either the Python API below, or the spice_process usage further below for parallel imaging.

This can be run using:

$ spice_image -h
usage: spice_image [-h] [--temp_dir_wsclean TEMP_DIR_WSCLEAN] [--temp_dir_images TEMP_DIR_IMAGES] [--psf_cutoff PSF_CUTOFF] [--robust ROBUST] [--nchan NCHAN] [--pols POLS] [--size SIZE] [--scale SCALE] [--mgain MGAIN] [--niter NITER] [--nmiter NMITER] [--auto_mask AUTO_MASK]
                   [--auto_threshold AUTO_THRESHOLD] [--local_rms] [--local_rms_window LOCAL_RMS_WINDOW] [--force_mask_rounds FORCE_MASK_ROUNDS] [--gridder {direct-ft,idg,wgridder,tuned-wgridder,wstacking}] [--taper TAPER] [--minuv MINUV] [--parallel PARALLEL] [--purge] [--mpi]
                   [--multiscale] [--multiscale_scale_bias MULTISCALE_SCALE_BIAS] [--multiscale_scales MULTISCALE_SCALES] [--absmem ABSMEM] [--make_residual_cubes] [--ms_glob_pattern MS_GLOB_PATTERN] [--data_column DATA_COLUMN] [--no_mf_weighting] [--skip_fix_ms]
                   [--num_beams NUM_BEAMS] [--disable_pol_local_rms] [--disable_pol_force_mask_rounds] [--hosted-wsclean HOSTED_WSCLEAN | --local_wsclean LOCAL_WSCLEAN]
                   msdir datadir


    mmm   mmm   mmm   mmm   mmm
    )-(   )-(   )-(   )-(   )-(
   ( S ) ( P ) ( I ) ( C ) ( E )
   |   | |   | |   | |   | |   |
   |___| |___| |___| |___| |___|
    mmm     mmm     mmm     mmm
    )-(     )-(     )-(     )-(
   ( R )   ( A )   ( C )   ( S )
   |   |   |   |   |   |   |   |
   |___|   |___|   |___|   |___|

    Arrkis imager


options:
  -h, --help            show this help message and exit
  --hosted-wsclean HOSTED_WSCLEAN
                        Docker or Singularity image for wsclean (default: docker://alecthomson/wsclean:latest)
  --local_wsclean LOCAL_WSCLEAN
                        Path to local wsclean Singularity image (default: None)

imaging arguments:
  msdir                 Directory containing MS files
  --temp_dir_wsclean TEMP_DIR_WSCLEAN
                        Temporary directory for WSClean to store intermediate files (default: None)
  --temp_dir_images TEMP_DIR_IMAGES
                        Temporary directory for to store intermediate image files (default: None)
  --psf_cutoff PSF_CUTOFF
                        Cutoff for smoothing in units of arcseconds.  (default: None)
  --robust ROBUST
  --nchan NCHAN
  --pols POLS
  --size SIZE
  --scale SCALE
  --mgain MGAIN
  --niter NITER
  --nmiter NMITER
  --auto_mask AUTO_MASK
  --auto_threshold AUTO_THRESHOLD
  --local_rms
  --local_rms_window LOCAL_RMS_WINDOW
  --force_mask_rounds FORCE_MASK_ROUNDS
  --gridder {direct-ft,idg,wgridder,tuned-wgridder,wstacking}
  --taper TAPER
  --minuv MINUV
  --parallel PARALLEL
  --purge               Purge intermediate files (default: False)
  --mpi                 Use MPI (default: False)
  --multiscale          Use multiscale clean (default: False)
  --multiscale_scale_bias MULTISCALE_SCALE_BIAS
                        The multiscale scale bias term provided to wsclean.  (default: None)
  --multiscale_scales MULTISCALE_SCALES
                        The scales used in the multiscale clean.  (default: 0,2,4,8,16,32,64,128)
  --absmem ABSMEM       Absolute memory limit in GB (default: None)
  --make_residual_cubes
                        Create residual cubes as well as cubes from restored images.  (default: False)
  --ms_glob_pattern MS_GLOB_PATTERN
                        The pattern used to search for measurement sets.  (default: scienceData*_averaged_cal.leakage.ms)
  --data_column DATA_COLUMN
                        Which column in the measurement set to image.  (default: CORRECTED_DATA)
  --no_mf_weighting     Do not use multi-frequency weighting.  (default: False)
  --skip_fix_ms         Do not apply the ASKAP MS corrections from the package fixms.  (default: False)
  --num_beams NUM_BEAMS
                        Number of beams to image (default: 36)
  --disable_pol_local_rms
                        Disable local RMS for polarisation images (default: False)
  --disable_pol_force_mask_rounds
                        Disable force mask rounds for polarisation images (default: False)

workdir arguments:
  datadir               Directory to create/find full-size images and 'cutout' directory

You may instead prefer to use the Python API, which is more flexible and allows for parallel imaging. You will need to set up your own Prefect task-runner for this. Here is a (very) minimal example:

from prefect.task_runners import SequentialTaskRunner
from arrakis.imager import main as imager_flow

def main(...):

    task_runner = SequentialTaskRunner()
    imager_flow.with_options(
        task_runner=task_runner
    )(
        ... # Add your arguments here
    )

You can find the full list of arguments in the API docs here: arrakis.imager.main.

The spice_process CLI

It is also possible to run just the imaging part of the pipeline using a the spice_process command line tool, as described in Running the pipeline. You will need to envoke the argument –imager_only, along with the other imaging arguments. This will run the imaging pipeline in parallel, using the Dask task runner defined in your config file of choice. Here is an example pipeline config for only imaging:

# SB8593.yaml
imager_only: true
ms_glob_pattern: 'scienceData_SB8593_RACS_1347-37A.beam*_averaged_cal.leakage.split.ms'
imager_dask_config: petrichor.yaml
mgain: 0.7
force_mask_rounds: 8
nmiter: 15
niter: 500000
local_rms: true
auto_mask: 4
local_rms_window: 60
auto_threshold: 1
size: 6144
scale: 2.5
robust: -0.5
pols: IQU
gridder: wgridder
minuv: 200
local_wsclean: wsclean_force_mask.sif
multiscale: true
multiscale_scale_bias: 0.7
multiscale_scales: "0,2,4,8,16,32,64,128"
purge: false
absmem: 100
nchan: 36
psf_cutoff: 30
skip_fix_ms: false
data_column: CORRECTED_DATA
disable_pol_local_rms: true
disable_pol_force_mask_rounds: false
temp_dir_images: /dev/shm
temp_dir_wsclean: /dev/shm

You would then run the pipeline using:

spice_process \
    --config SB8593.yaml \
    /path/to/ms/files/ \
    /path/to/work/dir/ \
    RACS_1347-37A