fix(storage): do not retry permanent errors in async writer resume - #16340
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces OpenTelemetry tracing and metrics support to the asynchronous OpenObject class, adding a new OpenObjectTelemetry helper class and corresponding unit tests. It also updates the buffered and resumed writer connections to prevent resume attempts on permanent failures. The review feedback correctly identifies a build issue where GOOGLE_CLOUD_CPP_HAVE_OPENTELEMETRY is unconditionally defined as PUBLIC in CMake, which would break builds when OpenTelemetry is disabled. Additionally, the reviewer suggests correcting the header inclusion in open_object.cc to include the trace provider instead of the metrics provider, as the file only retrieves the current span and does not record metrics directly.
34f75fb to
4af6792
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #16340 +/- ##
========================================
Coverage 92.23% 92.24%
========================================
Files 2227 2227
Lines 209418 209543 +125
========================================
+ Hits 193166 193291 +125
Misses 16252 16252 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
9463c50 to
e071750
Compare
AsyncWriterConnectionResumed and AsyncWriterConnectionBuffered previously unconditionally invoked Resume() when stream operations returned an error, even if the error was a permanent failure such as FAILED_PRECONDITION. For appendable uploads (BidiWriteObject), issuing a new BidiWriteObject RPC on Resume() caused the client to request writer exclusivity again, resulting in concurrent writers endlessly stealing exclusivity back and forth from each other on FAILED_PRECONDITION errors. This change checks if the error status is a permanent failure (via AsyncRetryPolicy / AsyncStatusTraits) before attempting Resume(). If permanent, the upload immediately terminates with the error status.
e071750 to
5c3f0b7
Compare
Summary
This PR fixes a bug in Storage async writer connections (
AsyncWriterConnectionResumedandAsyncWriterConnectionBuffered) where permanent errors (such asFAILED_PRECONDITION) were being retried during streaming write operations.Cause
Previously, when a stream error occurred during
Write,Flush,Close,Finalize, orQuery,Resume(Status const& s)was called unconditionally without checking ifswas a permanent failure. For appendable uploads (BidiWriteObject), issuing a newBidiWriteObjectRPC onResume()requested writer exclusivity from the server, causing concurrent writers to continuously steal exclusivity back and forth from each other in an infinite loop upon receivingFAILED_PRECONDITION.Fix
Resume(Status const& s)inwriter_connection_resumed.ccandwriter_connection_buffered.ccto check ifsis a permanent failure (viaAsyncRetryPolicy/AsyncStatusTraits) before attemptingResume().sis a permanent failure (such asFAILED_PRECONDITION), the connection immediately terminates with statussviaSetError(...)without opening a new stream.WriteConnectionResumed.PermanentErrorNoResumeandWriteConnectionBuffered.PermanentErrorNoResumeto verify that permanent errors on active streams do not trigger resume attempts.