Skip to content

[BUGFIX] Complete an OTLP HTTP request on every state its client can end on - #4453

Open
thc1006 wants to merge 3 commits into
open-telemetry:mainfrom
thc1006:bugfix/otlp-http-terminal-events-4425
Open

[BUGFIX] Complete an OTLP HTTP request on every state its client can end on#4453
thc1006 wants to merge 3 commits into
open-telemetry:mainfrom
thc1006:bugfix/otlp-http-terminal-events-4425

Conversation

@thc1006

@thc1006 thc1006 commented Aug 19, 2026

Copy link
Copy Markdown
Member

Fixes #4425.

ResponseHandler::OnEvent decides whether a request is over from a switch that names seven states. The client can end a transfer on three more, and the switch below it recognises all three and logs them:

State Reported as over Logged
CreateFailed, ConnectFailed, SendFailed, SSLHandshakeFailed, TimedOut, NetworkError, Cancelled yes yes
Destroyed, ReadError, WriteError no yes

So a request that ends on one of those three arrives, is described, and is then dropped. It never reaches Unbind, its result callback never runs, and its session is never released. ForceFlush and Shutdown have nothing to wait on but their own deadlines, and the caller is left waiting for a callback that is not coming.

The three states join the same switch. Nothing else changes.

Why the exporter has to answer them

The bundled curl client does not currently produce any of the three, so this is reached through the installed HttpClient interface by a client the user supplies. That interface is a supported extension point, SessionState is part of it, and the header describes these as error reading response, error writing request and a session destroyed. An exporter cannot decide which of them a client is allowed to use.

Ordering does not change. stopping_ is a compare-exchange and the first writer wins, so one of these arriving after a response has already been reported still reports nothing. That is what the last case below holds.

Checks

Two cases, both in otlp_http_exporter_custom_client_test.cc, driven by the existing nosend client. The first walks the three states, delivering each as the only terminal event, and requires the result callback to have run exactly once with kFailure and ForceFlush with a 50 ms deadline to report success rather than wait it out. The second delivers a response first and then each of the three, and requires the callback count to stay at one.

Removing the three lines from the switch does not turn those cases red. It hangs the whole binary: without them the request never settles, so the OtlpHttpClient destructor waits for a session that is still outstanding and the run does not terminate. That is the same shape the defect has for a user, which is why it is worth saying rather than reporting a red assertion that does not exist.

Both cases run clean under ASan and under TSan, nine of nine with no sanitizer report.

The clang-tidy job counts warnings against a limit and main sits at it, so the scaffolding here had to stay out of that budget. It does: checked with CMAKE_CXX_CLANG_TIDY over the all-options-abiv1-preview cache file, the way the job runs it, the test file reports nothing and the one warning left in otlp_http_client.cc is the one main already has, moved down by the nine lines this adds.

This is an exporter-side correctness fix over the installed custom HttpClient interface. It does not change the curl implementation and does not pre-commit anything in the transport design discussed in #4448, which is where @lalitb noted it can proceed independently.

For significant contributions please make sure you have completed the following items:

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@thc1006
thc1006 marked this pull request as ready for review August 19, 2026 20:20
@thc1006
thc1006 requested a review from a team as a code owner August 19, 2026 20:20
…end on

The switch that decides whether a request is over named seven states. The
client can end a transfer on three more: Destroyed, ReadError and WriteError.
The switch below it recognises all three and logs them, so they arrive, are
described, and are then dropped.

A request ended that way never reaches Unbind, so its result callback never
runs and its session is never released. ForceFlush and Shutdown have nothing
to wait on but their own deadlines, and the caller is left waiting for a
callback that is not coming.

The bundled curl client does not currently produce those three states, so this
is reached through the installed HttpClient interface by a client the user
supplies. That interface is a supported extension point and the states are part
of it, which is why the exporter has to answer them rather than assume which
ones a client uses.

Ordering is unchanged. stopping_ is a compare-exchange and the first writer
wins, so one of these arriving after a response has already been reported still
reports nothing.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 force-pushed the bugfix/otlp-http-terminal-events-4425 branch from 5e754a9 to 31118d3 Compare August 23, 2026 23:29
@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.73%. Comparing base (1c2b007) to head (f5e17e2).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4453      +/-   ##
==========================================
+ Coverage   82.67%   82.73%   +0.06%     
==========================================
  Files         516      516              
  Lines       20197    20197              
==========================================
+ Hits        16696    16708      +12     
+ Misses       3501     3489      -12     
Files with missing lines Coverage Δ
exporters/otlp/src/otlp_http_client.cc 73.48% <ø> (+2.36%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Three warnings, all from scaffolding this branch added rather than from what
it is testing. The clang-tidy job counts warnings against a limit and main is
at that limit, so three new ones fail it.

The parameterised fixture is gone. It existed to run one body over three
states, and a loop with SCOPED_TRACE does that without a class that only this
translation unit can use, which is also how the case next to it is written.

The two shared_ptr parameters of the recording helper are const references.
The lambda inside still takes its own copy, so nothing about the lifetime
changes; what goes away is a copy per call at the boundary.

Checked the way CI checks it, with CMAKE_CXX_CLANG_TIDY over the
all-options-abiv1-preview cache file, which reports no warnings for the test
file and leaves the one in otlp_http_client.cc that main already has.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The switch comment and the two test comments described what the code used to do
before the fix. That reads as history beside code that no longer has it, and it
belongs in this PR and in the commit message.

What is left states the invariant: every state the client can end a request on
is named here, and the exchange below decides ordering.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] OTLP HTTP exporter never completes a request that ends in ReadError, WriteError or Destroyed

1 participant