[BUG] A fault confirmed right after a post-fault recording window gets no rosbag

[bburda]

Bug report

Steps to reproduce

1. Configure the fault manager with rosbag snapshots on and a post-fault window, e.g. snapshots.rosbag.duration_sec: 2.0 and snapshots.rosbag.duration_after_sec: 0.5.
2. Report and confirm a fault, and wait for its bag to finalise.
3. Report and confirm a second fault within a few milliseconds of that bag completing.
4. Call GetRosbag for the second fault code.

Expected behavior

The second fault gets a bag, or at minimum a record saying why it could not.

Actual behavior

No bag and no record. GetRosbag reports success=false indefinitely and the only trace is a warning in the log:

Bag file completed: .../fault_ROSBAG_TEST_001_...
New fault reported: ROSBAG_CLEANUP_TEST          (+1.4 ms)
WARN  Buffer is empty, cannot create bag file
WARN  Failed to create bag file for fault ROSBAG_CLEANUP_TEST

flush_to_bag returns an empty path when the ring buffer is empty and on_fault_confirmed then only warns and returns. No RosbagFileInfo row is written and there is no retry, so polling cannot recover a decision made in the first millisecond.

The buffer is empty at that moment by construction: during a post-fault window message_callback writes straight to the bag and does not buffer, and flush_to_bag has already drained the deque.

Environment

• ros2_medkit version: 0.6.0 (main)
• ROS 2 distro: reproduced on jazzy and humble
• OS: Ubuntu 24.04 / 22.04 (CI containers), also reproduced locally

Additional information

This matters most where the feature is supposed to earn its keep. Bursts of correlated faults are the case a black box exists for, and the second fault of a burst is the one that lands in the gap.

The code already accepts the principle for the neighbouring case: attach_to_active_recording handles a fault confirming inside an active post-fault window. Nothing covers the instant after it finalises.

Options, in increasing cost: write a row flagged as having no pre-fault data so the gap is at least visible; keep the recording open across the boundary; or open a post-fault-only bag.

Surfaced when a new package changed test scheduling. The integration tests have been given guards so the suite no longer depends on timing, but those guards only hide it from CI.

[bburda]

There is a second entry path into the same failure sink, with a different precondition. Recording it here rather than opening a separate issue, because it bears directly on which of the three options is chosen.

A cold start reaches Buffer is empty with no prior bag at all

test_external_app_fault_rollup fails at its rosbag assertion roughly 2 runs in 3 on an idle machine. The fault is always present; only the rosbag is missing. The two warnings are the ones described above, but Bag file completed: appears zero times in every failing run - there is no prior fault, no prior bag, and no post-fault window anywhere in the timeline.

887.276140  RosbagCapture started with 0 topic subscriptions
887.276355  FaultManager node started
887.776150  New fault reported: PLC_LEVEL_OVERFLOW
887.777603  Subscribed to '/plc_cell/level'          <- 1.5 ms AFTER the report
887.778159  fault confirmed, flushing buffer to bag
887.778219  WARN  Buffer is empty, cannot create bag file

Three composing facts:

1. RosbagCapture::start() runs from the FaultManagerNode constructor, while the node's graph cache is still empty, so get_topic_type() fails for every explicitly configured topic. started with 0 topic subscriptions is guaranteed on a cold start, not racy.
2. The first opportunity to subscribe is the 500 ms retry tick.
3. In this test the pre-publish window is also 500 ms and phase-locked to the same origin: the retry timer is armed 0.2 ms before report_fault becomes discoverable, and the caller is blocked in wait_for_service on exactly that service. Both deadlines start from one event and expire together.

To buffer even a single message the subscribe has to beat the report by more than one publish period (>50 ms here); the observed margin is around ±2 ms. That is why the rate is a stable ~2-in-3 rather than an occasional miss.

Measurements

11 runs, each preceded by a check for orphaned nodes, stale listeners and host load.

• Baseline: 7/7 fail, load 0.40-1.29, all reporting after 1 retries, all with an empty buffer and no prior bag.
• Probe with the pre-publish window widened 0.5 s to 2.0 s: 4/4 pass, load 1.38-1.63.

The probe passes at higher load than the baseline fails at, which rules out host load and the 30 s poll budget as causes. The widened window is a causality probe only - as noted in the issue description, that shape hides the defect rather than fixing it, so it is not proposed as a fix.

Bearing on the three options

Two of the three also close the cold-start path; one does not.

option	fixes post-window boundary	fixes cold start
write a row flagged as having no pre-fault data	yes	yes
open a post-fault-only bag	yes	yes
keep the recording open across the boundary	yes	no - at cold start there is no prior recording to hold open

If the third option is chosen alone, this failure mode survives. The common thread in both paths is that flush_to_bag returning an empty path produces no RosbagFileInfo row and no retry, so a poll can never recover a decision taken in the first millisecond - which is why a longer client-side timeout does not help in either case.