[o365] - Report audit input degraded on persistent transport failures by ShourieG · Pull Request #20875 · elastic/integrations

ShourieG · 2026-08-24T13:02:06Z

Type of change

Enhancement

Proposed commit message

o365: report audit input degraded on persistent transport failures

The audit data stream's CEL program wraps every do_request in
try("_transport_error") so that a single unfetchable item does not
stall collection. That handler emits a dropped retry placeholder, which
the CEL input treats as a successful evaluation and reports Healthy. A
persistent failure such as an expired client secret failing OAuth 2.0
token acquisition was therefore swallowed: collection stopped while the
integration kept reporting Healthy.

Track consecutive transport failures in state. Below the new
transport_failure_threshold the existing recover-and-continue behaviour
is unchanged, so isolated transient failures still do not degrade the
input. Once the streak reaches the threshold the failure is treated as
persistent and returned as a single-object error, marking the input
degraded until a successful request resets the streak. The threshold is
a hidden variable defaulting to 5.

Regenerate the policy test snapshots for the rendered program and the
new variable.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates [o365] - Recover from transport-level fetch failures instead of stalling #20333

Screenshots

The audit data stream's CEL program wraps every do_request in try("_transport_error") so that a single unfetchable item does not stall collection. That handler emits a dropped retry placeholder, which the CEL input treats as a successful evaluation and reports Healthy. A persistent failure such as an expired client secret failing OAuth 2.0 token acquisition was therefore swallowed: collection stopped while the integration kept reporting Healthy. Track consecutive transport failures in state. Below the new transport_failure_threshold the existing recover-and-continue behaviour is unchanged, so isolated transient failures still do not degrade the input. Once the streak reaches the threshold the failure is treated as persistent and returned as a single-object error, marking the input degraded until a successful request resets the streak. The threshold is a hidden variable defaulting to 5. Regenerate the policy test snapshots for the rendered program and the new variable. Updates elastic/sdh-beats#7513 Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-08-24T13:03:19Z

Elastic Docs Style Checker (Vale)

Summary: 1 warning found

⚠️ Warnings (1): Fix when the suggestion improves clarity or correctness.

File	Line	Rule	Message
packages/o365/changelog.yml	8	Elastic.BritishSpellings	Use American English spelling 'behavior' instead of British English 'behaviour'.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

elastic-vault-github-plugin-prod · 2026-08-24T13:39:54Z

🚀 Benchmarks report

Package `o365` 👍(1) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`audit`	2096.44	1626.02	-470.42 (-22.44%)	💔

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-08-24T14:05:59Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2026-08-24T22:47:25Z

+            state.with(
+              {
+                "want_more": false,
+                "transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,


Suggested change

"transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,

"transport_error_streak": int(state.?transport_error_streak.orValue(0)) + 1,

Though, if the state is initialised with this field at zero, we don't need any of the optional type handling and this would be

Suggested change

"transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,

"transport_error_streak": int(state.transport_error_streak) + 1,

which I think would be clearer.

efd6 · 2026-08-24T22:47:38Z

+            // threshold the failure is persistent, so return a single-object
+            // error to mark the input degraded until collection recovers.
+            (
+              (int(state.?transport_error_streak.orValue(0.0)) + 1 >= int(state.base.transport_failure_threshold)) ?


Suggested change

(int(state.?transport_error_streak.orValue(0.0)) + 1 >= int(state.base.transport_failure_threshold)) ?

(int(state.?transport_error_streak.orValue(0)) + 1 >= int(state.base.transport_failure_threshold)) ?

efd6 · 2026-08-24T22:48:02Z

+                      }
+                    ),
+                    "want_more": false,
+                    "transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,


Suggested change

"transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,

"transport_error_streak": int(state.?transport_error_streak.orValue(0)) + 1,

efd6 · 2026-08-24T22:48:38Z

+                        "todo_content": tail(state.cursor.todo_content) + [state.cursor.todo_content[0]],
+                      }
+                    ),
+                    "want_more": false,


This line is inconsistent with the {"retry":true} object below.

@efd6, this is an existing behaviour from - #20333. It's the core of the backoff logic which was implemented. Due to how we influence cursor saves atm, this kind of structure was necessary, basically says "save my queue re-ordering, index nothing, then back off until the next poll."

With the current implementation of transport_error_streak we can technically change this to "want_more": true, but this will be a behaviour change to what the related PR introduced. Also now we might end up in states where we unnecessarily loop on "any transport error", until the threshold is reached. This will be self defeating to the backoff logic and result in quick bursts of errors.

~~That's fine, but the object below is pointless.~~

Ah, maybe not. If this is for the side-effect of publishing a cursor, then it makes sense. Is that what it is for?

@efd6 yes exactly, we need this for a cursor publish, and it results in this pattern unfortunately.

efd6 · 2026-08-24T22:49:21Z

                  {
-                    "todo_content": tail(state.cursor.todo_content) + [state.cursor.todo_content[0]],
+                    "want_more": false,
+                    "transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,


Suggested change

"transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,

"transport_error_streak": int(state.?transport_error_streak.orValue(0)) + 1,

Initialize transport_error_streak to zero in the state block so the counter is always present. The input applies the state block on every start and only overlays the persisted cursor, so the field is guaranteed even after an upgrade from a version that never set it. With the field always present the reads no longer need optional handling: int(state.?transport_error_streak.orValue(0.0)) becomes int(state.transport_error_streak) at all nine sites. Regenerate the policy test snapshots for the changed program. Co-authored-by: Cursor <cursoragent@cursor.com>

ShourieG · 2026-08-25T06:01:55Z

@efd6, I've resolved all comments besides one, which needs further discussion.

elastic-vault-github-plugin-prod · 2026-08-25T06:04:58Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-08-25T06:08:31Z

🟢 No issues across the latest commits f43916b.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.7 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-08-25T06:36:48Z

💚 Build Succeeded

Buildkite Build
Commit: f43916b

History

💚 Build #48429 succeeded f2ba578

cc @ShourieG

ShourieG self-assigned this Aug 24, 2026

ShourieG added enhancement New feature or request integration Label used for meta issues tracking each integration Integration:o365 Microsoft Office 365 Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Aug 24, 2026

updated changelog

f2ba578

ShourieG marked this pull request as ready for review August 24, 2026 14:05

ShourieG requested review from a team as code owners August 24, 2026 14:05

efd6 reviewed Aug 24, 2026

View reviewed changes

	"transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,
	"transport_error_streak": int(state.?transport_error_streak.orValue(0)) + 1,

	"transport_error_streak": int(state.?transport_error_streak.orValue(0.0)) + 1,
	"transport_error_streak": int(state.transport_error_streak) + 1,

	(int(state.?transport_error_streak.orValue(0.0)) + 1 >= int(state.base.transport_failure_threshold)) ?
	(int(state.?transport_error_streak.orValue(0)) + 1 >= int(state.base.transport_failure_threshold)) ?

Uh oh!

Conversation

ShourieG commented Aug 24, 2026

Type of change

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions Bot commented Aug 24, 2026

Elastic Docs Style Checker (Vale)

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package o365 👍(1) 💚(0) 💔(1)

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 24, 2026

Uh oh!

efd6 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ShourieG Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShourieG Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ShourieG commented Aug 25, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 25, 2026

Uh oh!

vera-review-bot Bot commented Aug 25, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 25, 2026

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elastic-vault-github-plugin-prod Bot commented Aug 24, 2026 •

edited

Loading

Package `o365` 👍(1) 💚(0) 💔(1)

efd6 Aug 25, 2026 •

edited

Loading

ShourieG Aug 25, 2026 •

edited

Loading