[o365-plugin] Office 365 audit events may be lost after collector restarts or interrupted collection runs

[mcjon3z]

Acknowledgements

• I have searched (https://github.com/utmstack/UTMStack/issues) for past instances of this issue
• I have verified that my UTMStack version is up-to-date

Describe the bug

The Office 365 collector appears to keep its collection time window in memory without persisting the last successfully completed collection position.

If the plugin, service, or host is restarted—or if a collection run is interrupted—the collector may initialize a new collection window based on the current runtime rather than resume from the last successfully processed point.

This can create gaps in Microsoft 365 audit-log collection. The collector also appears unable to distinguish a fully completed interval from an interval that was only partially processed.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The Office 365 collector should maintain persistent collection state for each configured tenant and subscription.

After a restart or interrupted collection run, it should:

1. Load the last successfully committed checkpoint.
2. Resume collection from that checkpoint.
3. Retry any interval that was not fully completed.
4. Advance the checkpoint only after all pages and content blobs for the interval have been successfully processed.
5. Avoid permanently skipping events generated while the collector was unavailable.
6. Minimize duplicate ingestion when retrying interrupted intervals.

Current Behavior

The collection window appears to be maintained only by the running collector process.

When the process is restarted, there does not appear to be a persistent per-tenant or per-subscription checkpoint from which collection can resume.

Potential failure cases include:

• Restarting the Office 365 plugin.
• Restarting the UTMStack service or host.
• Network interruption during collection.
• Authentication failure during collection.
• API throttling or temporary Microsoft service errors.
• Failure after some, but not all, content blobs in an interval have been processed.

In these cases, events generated during the interruption may not be retrieved, or a partially completed interval may not be retried safely.

Reproduction Steps

1. Configure the UTMStack Office 365 plugin and verify that it is collecting audit events.
2. Record the time of a successful collection run.
3. Stop the plugin or UTMStack service.
4. Generate Microsoft 365 audit activity while the collector is stopped.
5. Leave the collector stopped for longer than its normal collection interval.
6. Restart the plugin or service.
7. Allow collection to resume.
8. Search UTMStack for the audit events generated during the outage.
9. Compare the results with the Microsoft 365 unified audit log or Management Activity API.
10. Observe whether the events generated during the outage are absent.
11. Repeat the test by interrupting a run after only part of an interval has been processed.

Possible Solution

Persist collection state for each tenant and content subscription.

The state could include:

• The start and end time of the last fully completed interval.
• The last successfully committed collection timestamp.
• The status of an interval currently being processed.
• Optionally, processed content blob IDs to support safe retries.

Additional Information/Context

No response

UTMStack Version

v11

Operating System and version

Ubuntu 24.4

Hypervisor and Version | Server Vendor and Model

ESX

Browser and version

Chrome