BUG - Data from 2025-12-29 inside MAG L1B 2025-12-20 CDF files

[alastairtree]

Description of the issue

The latest version L1B mag data files for burst mode on day 2025-12-20 contains epochs like 2025-12-29T23:59:58.667793920. This should never happen! These files should container 20th Dec + 30min on either side on 19/21 Dec. The previous version (1) of these files data looks better at first glance but presumably v2 was generated for good reason so we probably need a v3.

Files affected:

• imap_mag_l1b_burst-magi_20251220_v002.cdf
• imap_mag_l1b_burst-mago_20251220_v002.cdf
• imap_mag_l1a_burst-mago_20251220_v002.cdf
• imap_mag_l1a_burst-magi_20251220_v002.cdf

Metadata for the files looks legit:

But the epochs are very wrong - out by 9 days, and probably the vectors are wrong as well. Is this data from 9 days in the future?!

Please can the SDC:

1. Regenerate and replace/remove these very dodgy files
2. Explain how this has occurred & reassure us this can't happen again - maybe we need a defensive step at the end of the pipeline?

Steps to reproduce the issue

No response

Expected vs Actual behavior

No response

Code Snippet (If applicable)

Code

Additional notes, affected areas, and suggested fixes

No response

[sapols]

@alastairtree I have more-or-less gotten to the bottom of this: it was a bad L0 file. I'll paste my full report below, which probably contains more details than is necessary, just to not leave anything out. If everyone agrees on adding the recommended defensive code change in mag_l1b.py to avoid this in the future, I can implement that.

TL;DR

The v002 MAG burst CDFs for 2025-12-20 were built from a mis-labelled L0 file. imap_mag_l0_raw_20251220_v003.pkts contains day-30 burst packets under a day-20 filename. MAG L1A/L1B then faithfully stamped those packets with the dates their onboard MET actually implies, which is why the output epochs land on 2025-12-29/30. Nothing in imap_processing is broken — ground_software_version is identical for v001 and v002, and no MAG L1A/L1B code changed between them.

The fix for the existing v002 files is regeneration from the correct L0 (team should handle that). The fix for "can't happen again" needs two things:

1. SDC-side: find and fix whatever upstream step wrote day-30 packets under the day-20 filename on 2026-01-20 ~18:07 UTC, and audit other days' v002+ L0 files for similar mismatches.
2. imap_processing-side: add an epoch-window guard in mag_l1b that aborts with ValueError if any epoch falls outside [day_to_process − 30 min, day_to_process + 1 d + 30 min]. This won't fix the upstream mis-routing, but it will stop a bad L0 from silently producing a bad public CDF.

The issue, in one sentence

Epochs in imap_mag_l1[ab]_burst-mag[io]_20251220_v002.cdf start at 2025-12-29T23:59:58.667… instead of somewhere near 2025-12-20.

Root cause

imap_mag_l0_raw_20251220_v003.pkts is not a day-20 L0 file. It's a trimmed slice of day 30's L0. Specifically, packet-by-packet comparison shows:

imap_mag_l0_raw_20251220_v003.pkts
  ==  imap_mag_l0_raw_20251230_v002.pkts[900 : -900]

That is, day 30's 45,000-packet burst L0 with the first and last 900 packets removed. At 2 s per packet (PUS_SSUBTYPE = 1), that's 1,800 s = 30 min trimmed from each side — exactly the edge buffer webpoda adds around a day. So the file that was delivered as imap_mag_l0_raw_20251220_v003.pkts is day 30's L0 with its ±30-min buffers stripped and the filename rewritten to 20251220.

When the MAG pipeline reprocessed 20251220 on 2026-01-20, it picked up that wrong-day L0, decommutated its packets truthfully, converted each packet's SHCOARSE/FINETIME through SPICE, and stamped every output vector with the (day-30) timestamp those fields actually encode. The resulting CDFs are a correct rendering of a wrong input.

Recommended defensive code change (imap_processing)

Add a hard validation step at the top of mag_l1b:

• Where: imap_processing/mag/l1b/mag_l1b.py, inside mag_l1b(...), immediately before calling mag_l1b_processing (around line 77). day_to_process is already in scope there, plumbed from cli.py:1263.
• What: compute [day_to_process − 30 min, day_to_process + 1 d + 30 min] and raise ValueError if any input_dataset["epoch"] value falls outside it. Include in the error: logical source, first and last epoch, day_to_process, and the parent filenames.
• Why ValueError and not a warning: Alastair's "should never happen" phrasing, plus the fact that a misdated CDF is a correctness failure, argue for hard-failing. A warn-only guard would have logged this exact incident and still shipped the file.
• Why L1B and not L1A: mag_l1b already takes day_to_process as a parameter; mag_l1a only takes a packet file path, so adding the check there would require extending the signature. Since every public L1B product goes through mag_l1b, catching it here is sufficient to stop bad data from being published.

A mirror check in L1A (and L1C/L1D, which already take current_day) is a reasonable follow-up but not strictly necessary.

Recommended SDC-side actions

1. Identify which upstream step produced imap_mag_l0_raw_20251220_v003.pkts (ingested 2026-01-20 18:07:25 UTC). This is the origin of the mis-routing.
2. Audit other recent v002+ L0 re-deliveries for MAG (and plausibly other instruments) by decoding each file and comparing its PRI_COARSETM range against the filename date. The [900:-900]-of-a-neighbour shape is a specific signature worth scanning for.
3. Consider whether the re-delivery path should itself enforce a filename-vs-MET consistency check at write time.

Evidence

1. Parent files differ between v001 and v002; code does not

Global attributes read from both L1A MAGO CDFs via imap-data-access:

Attribute	L1A v001	L1A v002
Parents (L0)	imap_mag_l0_raw_20251220_v001.pkts	imap_mag_l0_raw_20251220_v003.pkts
SCLK kernel	imap_sclk_0072.tsc	imap_sclk_0097.tsc
ground_software_version	1.0.9.post15.dev0+90f72e4	1.0.9.post15.dev0+90f72e4 (identical)
Generation_date	20251223	20260120
epoch[0] (J2000 ns)	819459068095076497	820324867851763422
epoch.shape	11,520,000 (25 h = 24 h + 30-min buffers)	11,059,200 (24 h, buffers missing)

Epoch diff: 865,799,756,686,925 ns ≈ +10.02 days. Identical code, different L0 parent, 10-day shift in output. Same-code-different-input is the signature of an input problem, not a code problem.

2. The bad L0's MET values are 10 days later than the good one's

Decoded both 20251220 L0 files with imap_processing.mag.l0.decom_mag and a third file for comparison:

L0 file	burst packets	PRI_COARSETM (MET s) range
imap_mag_l0_raw_20251220_v001.pkts	45,000	503,883,004 → 503,973,002
imap_mag_l0_raw_20251220_v003.pkts	43,200	504,748,805 → 504,835,202
imap_mag_l0_raw_20251230_v002.pkts	45,000	504,747,005 → 504,837,002

The MET shift between v001 and v003 is +865,801 s ≈ +10.02 days — exactly matching the epoch shift in the output CDFs, and the v003 range is entirely contained within the day-30 file's range.

3. The bad L0 is a proper subset of day 30's L0

All 43,200 burst packets in 20251220_v003.pkts have a PRI_COARSETM that falls inside 20251230_v002.pkts's range:

Neighbour L0	burst range	overlap with 20251220_v003
imap_mag_l0_raw_20251230_v002.pkts (45,000)	504,747,005 → 504,837,002	43,200 of 43,200
imap_mag_l0_raw_20251229_v002.pkts (45,000)	504,660,604 → 504,750,601	899 (boundary only)
imap_mag_l0_raw_20251230_v001.pkts (28,749, partial)	504,747,005 → 504,804,501	27,849

Promoting this from range overlap to packet-level equality: after sorting both files by packet time, 20251220_v003 == 20251230_v002[900:-900] holds for every packet — sequence counters, PRI_COARSETM, PRI_FNTM, and vector payload bytes all match, first-to-last.

4. The MAG L1A/L1B code cannot introduce a 10-day jump

The relevant code is short and mechanical:

• imap_processing/mag/l0/decom_mag.py — decommutates packets, no day-range filtering; every packet in the input file is kept.
• imap_processing/mag/l1a/mag_l1a.py:147 — builds the packet start time directly from the packet header: TimeTuple(mag_l0.PRI_COARSETM, mag_l0.PRI_FNTM).
• imap_processing/mag/l1a/mag_l1a_data.py:84-95 — converts MET → J2000 TT ns via SPICE (met_to_ttj2000ns → sct_to_ttj2000s).
• imap_processing/mag/l1a/mag_l1a_data.py:388-427 — calculate_vector_time fills per-vector epochs with np.arange(start_time_ns, start_time_ns + 1/128 s × N, step). Stays strictly within one packet's span.
• imap_processing/mag/l1b/mag_l1b.py:139 — L1B does not recompute epochs; it passes them through with only a small additive calibration time_shift (fractional seconds).

None of these can turn a correct day-20 packet stream into a clean, contiguous day-29/30 day of data.

5. Ruling out the SCLK kernel change

The SCLK kernel did move from _0072 (v001 parent) to _0097 (v002 parent). Running the same MET through both kernels produces millisecond-level differences, not 10 days. A kernel re-anchoring at the 10-day scale would have broken many other instruments' products simultaneously, and it didn't.

6. Ruling out a code regression between v001 and v002

ground_software_version is identical. The build SHA 90f72e4e (PR #2533, Tim Plummer, 2025-12-22) only changed imap_processing/spice/pointing_frame.py. Between the v1.0.9 tag and 90f72e4e, the only MAG-touching commits on origin/dev are PR #2497 (L1D/L2 frame refactor) and PR #2519 (L1D/L2 CDF types). Neither touches L1A, L1B, or epoch handling.

[maxinelasp]

Thanks @sapols! One correction: I think the check against this error should happen in the output step for the CDF files. That way we can run it against all MAG levels and against any other instrument that can make this assumption. I'll continue investigation into the root cause of the bad file today.

[sapols]

Sure thing @maxinelasp! I'm not familiar with where that output step for the CDF files is, however, so could you point me there?
EDIT: Maxine pointed me there offline via Slack

[tmplummer]

@sapols, @maxinelasp Careful! Some instruments (ENAs + GLOWS) split data file boundaries on repointing number. Putting a check for all instruments that looks for data being outside the range [day_to_process − 30 min, day_to_process + 1 d + 30 min] will break those instruments. Furthermore, that range is, AFAIK, specific to ONLY MAG.

[sapols]

Oh! Thank you for chiming in, @tmplummer. I didn't know. So, that might be reason to confine this check to mag_l1b after all?

Although, our plan was to make the check off by default. I should clarify. We were thinking:

1. The starting point of the output writer is cli.py:post_processing()
2. We think the check could happen in utils.py:write_cdf(), which is called by post_processing()
3. This would be off by default, with a kwarg passed through post_processing() into write_cdf() to turn it on or off
   • (Only turn it on for all MAG levels at first, then later any instruments that can enforce this requirement)

So maybe the question is: is this truly specific to ONLY MAG, or can any other instrument enforce this requirement?

[tmplummer]

So maybe the question is: is this truly specific to ONLY MAG, or can any other instrument enforce this requirement?

The range specified ([day_to_process − 30 min, day_to_process + 1 d + 30 min]) is specific to MAG. Most of the in-situ instruments files contain data that is strictly bounded by the UTC day. ENAs + GLOWS separate data by pointing #.

I don't think that validation belongs in the utils.py:write_cdf() function. I would suggest that the appropriate place to add some sort of validation would be in a new method of the ProcessInstrument class that would be called in the post_processing() method. Something like _validate_datasets(). This would give flexibility to have checks that are unique to a single instrument or shared across some or all instruments.

Another alternative could be to only add this time range validation to L0 files.

Just as a note: Adding validation for the ENAs will necessarily add the repoint table as an upstream dependency for all ENA products. I'm not sure that we really want to add that.

[maxinelasp]

I agree with Tim that this should not be enabled for ENAs. Apologies, the subtlety that this would be off by default was discussed in slack but not added here. This specific check doesn't make a lot of sense for ENAs because of the pointing table product.

I agree with Tim's suggested implementation. Maybe we could make the range something that can be overwritten by instruments, so it's the 24 day by default and then MAG overwrites that to add the buffer?

[alastairtree]

Thanks for finding the cause. Was the source data file just ingested manually then and this is just human error?

Agree that a check somewhere before emitting a MAG CDF file and adding to all levels would be a most complete way to ensure it cannot happen again.

Also please can we get this day regenerated from scratch please so that we have some data for that day? Thanks.