Skip to content

Numba-enhanced Data Averaging (and more!) - #1694

Open
kartographer wants to merge 7 commits into
mainfrom
must_average_faster
Open

Numba-enhanced Data Averaging (and more!)#1694
kartographer wants to merge 7 commits into
mainfrom
must_average_faster

Conversation

@kartographer

Copy link
Copy Markdown
Contributor

Description

A new numba kernel for averaging data (with arbitrary grouping along one axis) has been added and integrated into UVData.frequency_average and UVData.downsample_in_time, which should substantially increase its speed (via multithreaded processing) and significantly reduce the memory footprint of the averaging step.

Motivation and Context

The primary motivation for this was addressing the speed and memory issue with frequency averaging, given the prior report about MWA data suffering a ~6x increase in memory footprint, although because the new kernel is somewhat flexibly designed to handle irregularly grouped arrays, I took the opportunity to add a few additional features to frequency_average (including an option to sum across nsample_array, which is appropriate for when it's being used to carry weights, as is the case in a few different filetypes).

Closes #730
Closes #1311
Closes #1624

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

New feature checklist:

  • I have added or updated the docstrings associated with my feature using the numpy docstring format.
  • I have updated the tutorial to highlight my new feature (if appropriate).
  • I have added/updated tests to cover my new feature.
  • I have updated the CHANGELOG.

@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.93%. Comparing base (18f4f6a) to head (32fcd10).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1694   +/-   ##
=======================================
  Coverage   99.93%   99.93%           
=======================================
  Files          67       69    +2     
  Lines       22982    23038   +56     
=======================================
+ Hits        22968    23024   +56     
  Misses         14       14           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

High memory usage when frequency averaging MWA correlator fits files Make UVData.frequency_average more flexible Frequency averaging

2 participants