Calibration Updates Pipeline 2026 A step-by-step guide for new users

This guide walks you through running the full pipeline — from SLURM job submission on HPC all the way to a published HTML leakage report — for the first time. It is written as a narrative complement to the project README, which contains exhaustive flag references and script tables. Read this first, then use the README as a day-to-day reference.

Overview

The pipeline measures on-axis polarisation leakage across ASKAP beams and reference fields. It has two side-by-side experiments for each cohort — a group of scheduling blocks that share the same holography reference weights, identified by their ref_ws ID (e.g. ref_ws-4788, ref_ws-5316):

Experiment	What it does	Run order
`baseline`	Processes data without Q-corrections. The resulting leakage measurements reveal the raw X/Y amplitude imbalance and are used to derive the correction factors (a CSV file).	Always run first
`qcorr`	Re-processes the same data with Q-corrections applied (using the CSV from the baseline run). The resulting report quantifies residual leakage after correction.	Run after `baseline` has produced its CSV

Each experiment runs through four stages:

Stage	Where	Purpose
1 — refField	HPC	Generate bandpass + leakage tables from reference-field observations
2 — 1934	HPC	Apply tables to 1934 control data set and generate calibrated visibilities.
3 — assess	HPC	Write out the baseline-averaged I,Q,U&V visibility spectra per beam into text files (`.txt` for bandpass calibrated and `.lcal.txt` for bandpass+on-axis leakage calibrated visibilities of the control dataset), and per-beam diagnostic plots: `_stokes.png` (Stokes I,Q,U,V spectra) and `_pol-degree.png` (polarization degree) for each variant.
4 — copy/combine	Local	Rsync outputs locally; invoke `combine_beam_outputs.py` to consolidate all 36 beams into single-file summary products: a `leakage_stats.png` panel showing channel-averaged leakage levels across all beams at a glance, a `.pdf` with every beam's Stokes and polarization-degree spectra plots laid out on one page, and `.mp4`/`.gif` animations that step through beams sequentially.

After stage 4, all work is local: build the HTML report, generate dQ plots and PAF overlays, then publish.

Prerequisites

Software and environment

Clone the repo and create a Python virtual environment:

git clone https://github.com/wasimraja81/mstool.git
cd mstool
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

The .venv must be active whenever you run local analysis scripts.
HPC access (Setonix / Pawsey) with ssh key and sbatch available.
rsync for stage-4 transfers.

Directory layout

All scripts self-locate using dirname "$0" or realpath — they can be invoked from any working directory, regardless of where you have cloned the repo.

Recommended practice: if you are working inside the repo, run from a scratch/ directory at the repository root. This keeps any generated files (logs, temporary outputs) out of the tracked tree.

mkdir -p scratch
cd scratch

You can equally run the scripts from outside the repo — just supply the correct path to the script, for example:

bash ~/repos/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh stage1

Key concepts

Manifests

A manifest is a tab-delimited text file that maps each scheduling block tuple to its processing parameters. Manifests live in projects/calibration-updates-2026/manifests/ and are named by their holography reference weights ID:

File	Cohort (ref_ws)
`manifest_ref_ws-4788.txt`	ref_ws-4788
`manifest_ref_ws-5316.txt`	ref_ws-5316

Each row in the manifest defines one scheduling block tuple. Per-row columns are idx, sb_ref, sb_1934, sb_holo, sb_target_1934, followed by optional key=value tokens — ODC_WEIGHT=<id>, REF_FIELDNAME=<name>, AMP_STRATEGY=<value>, DO_PREFLAG_REFTABLE=<true|false> — specified per row in any order. Global directives at the top of the file (ODC_WEIGHT_ID, LOCAL_BASE, HPC_BASE_DIR, REMOTE, AMP_STRATEGY) are legacy fallbacks; prefer setting these per-row for clarity. All scripts accept --manifest PATH to select a different manifest.

Extract from manifest_ref_ws-4788.txt:

# SB combinations
# idx  sb_ref  sb_1934  sb_holo  sb_target_1934  [optional key=value tokens]
# ODC-5229
14 81082 77045 76554 81089 ODC_WEIGHT=5229 REF_FIELDNAME=REF_0324-28
15 81083 77045 76554 81089 ODC_WEIGHT=5229 REF_FIELDNAME=REF_0236-28
16 81084 77045 76554 81089 ODC_WEIGHT=5229 REF_FIELDNAME=REF_0324-28

# ODC-5231
19 81099 77045 76554 81107 ODC_WEIGHT=5231 REF_FIELDNAME=REF_0236-28
20 81100 77045 76554 81107 ODC_WEIGHT=5231 REF_FIELDNAME=REF_0324-28
21 81104 77045 76554 81107 ODC_WEIGHT=5231 REF_FIELDNAME=REF_1212-28

Every stage script accepts three flags that let you process a subset of manifest rows rather than all of them in one go — useful for testing on a single scheduling block, reprocessing a failed row, or splitting a large cohort across multiple jobs:

--start-index / --end-index — select a contiguous row range by manifest index (the first column of each data row).
--exclude-indices — exclude specific rows from the selected range (e.g. 24-29 to skip rows 24 through 29).

These three flags accept the same values across all four stages — use the same numbers throughout a run so that each stage operates on exactly the same set of scheduling blocks.

The `--experiment` flag

Every stage-1 through stage-4 run script accepts:

--experiment  baseline | qcorr

This single flag controls the entire data path:

baseline — standard processing; HPC work directory and local results directory are the paths given in the manifest.
qcorr — Q-corrections enabled; the pipeline appends -qcorr to both the HPC work directory and the local results directory, keeping the two experiments cleanly separated. The Q-correction CSV is auto-derived from the manifest cohort ID.

Important: run baseline first and generate the correction CSV before attempting a qcorr run. The qcorr pipeline validates the CSV and every field name at startup — it will exit with a clear error message before submitting any SLURM jobs if something is missing.

Cohort wrapper scripts

For reproducible, auditable runs the recommended approach is to use the pre-built cohort wrappers in cohorts/. Each wrapper script encodes the manifest, experiment, index range, and Q-correction variant for a specific cohort as a named function for each stage:

cohorts/
  ref_ws-4788_baseline.sh   # ref_ws-4788 baseline run
  ref_ws-4788_qcorr.sh      # ref_ws-4788 Q-correction run
  ref_ws-5316_baseline.sh   # ref_ws-5316 baseline run

Usage — call the wrapper with the name of the stage you want to run (~/mstool is the repository root — adjust if you cloned elsewhere):

bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh stage1
bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh stage2
# … and so on

The wrappers call the same underlying run_stage-*.sh scripts with all arguments pre-filled; you can inspect them to see the exact flags passed, or pass your own flags directly to the run scripts for non-standard runs.

First run: baseline experiment

Work through the stages in order. Stages 1–3 submit SLURM jobs to the HPC; wait for each to complete before moving to the next.

Stage 1 — reference field HPC

Generates bandpass and leakage tables from reference-field observations.

# Using the cohort wrapper (recommended):
bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh stage1

# Or directly:
bash ~/mstool/projects/calibration-updates-2026/scripts/run_stage-1.sh \
    --manifest    ~/mstool/projects/calibration-updates-2026/manifests/manifest_ref_ws-4788.txt \
    --start-index 0 \
    --end-index   49 \
    --experiment  baseline

This submits one SLURM job per scheduling block row in the index range. Monitor job status on the HPC with squeue -u $USER.

Tip: pass --help to any run_stage-*.sh script to see all available options, including --exclude-indices for skipping problematic rows.

Stage 2 — 1934 calibration HPC

Applies the bandpass and leakage tables to 1934 data and quality-checks the calibration.

bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh stage2

Wait: do not start stage 2 until all stage-1 jobs have completed successfully. Check the HPC job log files for any failures before proceeding.

Stage 3 — assessment HPC

Runs averageMS.py per beam; produces per-beam .txt and .lcal.txt channel measurement files and PNG plots in the HPC work directory.

bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh stage3

Stage 4 — copy and combine Local

Rsyncs the HPC assessment outputs to your local machine, then runs combine_beam_outputs.py to assemble the per-SB_REF results into the combined data structures that the report pipeline consumes.

bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh stage4

Add --copy-metadata on the first run (or when you need PAF overlay plots): it also rsyncs the metadata/ directories from the HPC — required for PAF beam-overlay images in the report.

# Directly, with metadata:
bash ~/mstool/projects/calibration-updates-2026/scripts/run_stage-4.sh \
    --manifest      ~/mstool/projects/calibration-updates-2026/manifests/manifest_ref_ws-4788.txt \
    --start-index   0 \
    --end-index     49 \
    --experiment    baseline \
    --copy-metadata

After stage 4 all subsequent steps run entirely locally — no HPC connection required.

Building the HTML report

The HTML report is built in a single command that runs the full local post-processing pipeline: Phase 1 master table → Phase 2 isolation tables → leakage cube → footprint heatmaps → PAF overlays → dQ plots → HTML.

bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh report

# Or directly:
bash ~/mstool/projects/calibration-updates-2026/scripts/run_html_report.sh \
    --manifest   ~/mstool/projects/calibration-updates-2026/manifests/manifest_ref_ws-4788.txt \
    --experiment baseline

Once generated, open phase3/index.html (under your LOCAL_BASE directory) directly in any browser to preview the report before publishing.

The output artefacts are written under your LOCAL_BASE directory:

phase3/index.html — the main report
phase3/plots/ — footprint heatmaps, PAF overlays, dQ plots, beam movies
phase2/leakage_cube.nc — 3-D NetCDF4 leakage cube (xarray-compatible)

Adding Q-corrections (qcorr experiment)

Once the baseline run is complete and the correction CSV has been generated by plot_dQ_vs_beam.py (called automatically inside run_html_report.sh), you can run the qcorr experiment.

The CSV is expected at:
/askapbuffer/payne/raj030/askap-calibration-updates/ref_ws-4788/dq_du_correction_factors.csv
where ref_ws-4788 is derived automatically from the manifest filename. The pipeline validates the file exists and that every field in the manifest has entries in it — before submitting any SLURM jobs.

The workflow mirrors the baseline run exactly — just use the qcorr cohort wrapper:

bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_qcorr.sh stage1
# … wait for HPC …
bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_qcorr.sh stage2
# … wait for HPC …
bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_qcorr.sh stage3
bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_qcorr.sh stage4
bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_qcorr.sh report

The -qcorr suffix is appended automatically to both the HPC work directory and local results directory, so both experiments coexist without overwriting each other. The local results will appear under e.g. ~/DATA/reffield-average/assess_1934-ref_ws-4788-qcorr/.

Publishing the report

When the report looks good, publish it to GitHub Pages:

bash ~/mstool/projects/calibration-updates-2026/cohorts/ref_ws-4788_baseline.sh publish

# Or directly:
bash ~/mstool/projects/calibration-updates-2026/scripts/publish_report.sh \
    --manifest   ~/mstool/projects/calibration-updates-2026/manifests/manifest_ref_ws-4788.txt \
    --experiment baseline

The script rsyncs the assembled package to wasimraja81/askap-leakage-report on the develop branch. To go live, merge develop → main on GitHub (either via a PR or from the command line in the ~/github-wasimraja81/askap-leakage-report clone). GitHub Pages redeploys automatically within 1–3 minutes.

The live report is at https://wasimraja81.github.io/askap-leakage-report/.

Troubleshooting

Symptom	Likely cause and fix
Stage 1 exits with "Q correction ref_ws not found in CSV"	The holography reference weights ID (`ref_ws`) resolved from the manifest is not present in the correction CSV. Check that the CSV was generated for this cohort and that `BP_UPDATE_Q_CORR_REF_WS` / `PARENT_WEIGHT_SCIENCE` resolves to the correct integer.
Stage 1 exits with "No CSV rows found for field='REF_…'"	A field name in the manifest is absent from the correction CSV. The error message lists all available fields. Add the missing field to the CSV or exclude that row with `--exclude-indices`.
PAF overlays missing in the HTML report	Stage 4 was not run with `--copy-metadata`, so the local `metadata/` directories were not populated. Re-run stage 4 with `--copy-metadata`, then re-run the report.
Stage-4 rsync fails for some rows	Check HPC connectivity and that stage 3 completed for those rows. Re-run stage 4 with a narrowed `--start-index / --end-index` for the failing subset.
"BP_UPDATE_Q_CORR_CSV no longer found" inside SLURM job	The correction CSV was present when `processDefaults.sh` ran but was subsequently removed or moved. Restore the file and resubmit.

Further reference

Project README — exhaustive script and flag reference, output artefact list, PAF visualisation docs, publishing one-time setup
Repository root README — repo layout, mstool Python scripts, installation
--help on any run_stage-*.sh, run_html_report.sh, or publish_report.sh for the full flag list
Inspect the cohort wrapper scripts in cohorts/ to see which flags are passed for each experiment

Calibration Updates Pipeline 2026 · CSIRO ATNF · wasimraja81/mstool

Calibration Updates Pipeline 2026 A step-by-step guide for new users

Overview

Prerequisites

Software and environment

Directory layout

Key concepts

Manifests

The --experiment flag

Cohort wrapper scripts

First run: baseline experiment

Stage 1 — reference field HPC

Stage 2 — 1934 calibration HPC

Stage 3 — assessment HPC

Stage 4 — copy and combine Local

Building the HTML report

Adding Q-corrections (qcorr experiment)

Publishing the report

Troubleshooting

Further reference

The `--experiment` flag