BUG - MAG L1C validation tests

[maxinelasp]

Description of the issue

Validation tests for MAG L1C are still failing. Gap-filling algorithm needs updates to pass them.

Steps to reproduce the issue

No response

Expected behavior (What should happen)

No response

Actual behavior (What does happen)

No response

Code Snippet:

Code

Additional notes, affected areas, and suggested fixes

No response

[greglucas]

This has been requested as high-priority for the MAG team and they are early in commissioning which pushes the timeline sooner as well. What are the details for reproducing this? Do the current tests not pass / are xfailed, so the goal is to get the tests to pass? Or are there some new tests/validation data that would help to debug this situation?

[maxinelasp]

Yes, there are 3 validation tests in an "XFAIL" state. Here are my priorities for this ticket:

• Implement proper CIC filter delay compensation ***
• Improve buffer size calculation (algorithm specifies 2x normal mode cadence seconds) ***
• Fix gaps at beginning/end of day by accessing previous/next day files **
• Handle cases with only one input file (missing burst or norm) **
• Use FILLVAL instead of zeros for missing data **

These are roughly in priority order.

[alastairtree]

Notes on the history of this ticket to help find all the issues. "Version 2" of L1C was not completed so many of the tests fail and the algorithm implememented does not meet all the steps outlined in the algorithm document. The tests should most of these gaps in implementation which is why serveral are still failing.

We have been in continuous burst mode for serval weeks so L1C is currently easy/reliable, but once we are in routine ops there will be one of more periods of N hours of burst mode with in a days data, with one or more periods of normal mode and each block might span over day boundaries. Those cases are in part not implemented/may cause failures. And some fixes have been made since we last checked this so current state is not clear.

L1C should:

• downsample so that each L1C day continues the previous. Currently It starts a fresh instead of trying to continue the cadence from the previous day. Ideally every day of L1C data continues a clean pattern of 2 vectors a second (say) so 500ms gap between any 2 adjacent vectors in a month (say), meaning all downsampled Burst->Normal sections carry on a time series from the previous normal section if possible, even if that started in the previous day.
• it needs to handle blocks of burst mode spanning over day boundaries
• it needs to handle gaps in both data streams, and either stream missing/arriving out of order.
• one, 2 or many burst, normal and no-science/mag-off transitions in a day
• a day can start and end in any mode (burst 64-8, burst 128-128,, normal 2-2, MAG OFF etc)

Serveral of the above are overlapping cases but the point is it has to be able to handle all of them without erroring.

All of L1C is higher priority than the single sensor bug #2255

[alastairtree]

On 5th and 6th Dec 2025 MAG went between BM and NM and BM. As a result of the issues in this ticket there are now gaps in MAG data for those days.