Skip to content

out_forward: drop keepalive sockets with failed or mismatched acks#5436

Open
JSap0914 wants to merge 1 commit into
fluent:masterfrom
JSap0914:fix-forward-ack-stale-socket-4057
Open

out_forward: drop keepalive sockets with failed or mismatched acks#5436
JSap0914 wants to merge 1 commit into
fluent:masterfrom
JSap0914:fix-forward-ack-stale-socket-4057

Conversation

@JSap0914

Copy link
Copy Markdown

Which issue(s) this PR fixes:
Fixes #4057

What this PR does / why we need it:

In Forwarder-Aggregator mode with keepalive true and flush_thread_count > 1, when the aggregator restarts/reconnects the forwarder starts logging

[warn]: ack in response and chunk id in sent data are different chunk_id="..." ack="...\n"

endlessly, and never recovers on its own until the pod is killed.

Root cause: when a chunk's ack times out (FAILED) or comes back with a chunk id that does not match the one tracked for its socket (CHUNKID_UNMATCHED), ack_check returned the keepalive socket to the reuse pool via ConnectionManager#close (→ SocketCache#checkin). Such a connection may still carry an in-flight/stale ack for a previous chunk. When the socket is checked out again for a new chunk, that stale ack is read first and compared against the new chunk's id, so it never matches. From then on every ack on that connection is offset by one chunk, producing an unbounded stream of mismatch warnings.

This PR adds ConnectionManager#discard, which revokes a keepalive socket (or closes a non-keepalive one) instead of checking it back in, and uses it from ack_check on the FAILED and CHUNKID_UNMATCHED paths. The tainted connection is torn down, so the next chunk uses a fresh connection with a clean ack stream and the warning storm stops. Successful acks keep reusing the socket exactly as before.

Docs Changes:
None.

Release Note:
out_forward: stop the endless "ack in response and chunk id in sent data are different" warning storm by discarding (instead of reusing) a keepalive socket whose ack failed or came back with a mismatched chunk id.


Tests (ruby 3.4.10):

  • test/plugin/out_forward/test_connection_manager.rb — new #discard cases: closes a non-keepalive socket, revokes a keepalive socket instead of checking it in, and confirms a discarded keepalive socket is not reused (a checked-in one is).
  • test/plugin/test_out_forward.rb — new ack_check cases: the socket is discarded on CHUNKID_UNMATCHED and FAILED, and reused on SUCCESS.
$ bundle exec ruby -Itest test/plugin/out_forward/test_connection_manager.rb
10 tests, 16 assertions, 0 failures, 0 errors

$ bundle exec ruby -Itest test/plugin/test_out_forward.rb
74 tests, 147 assertions, 0 failures, 0 errors, 3 omissions (Windows-only)

The new tests fail on master (the mismatched/failed ack socket is reused) and pass with this change.

When a chunk's ack times out or comes back with a chunk id that does not
match the one tracked for its socket, the keepalive connection may still
carry an in-flight/stale ack for a previous chunk. Returning that socket to
the reuse pool lets the stale ack be read against a chunk sent on the reused
socket, permanently offsetting every subsequent ack and flooding the log
with "ack in response and chunk id in sent data are different" warnings
until the forwarder is restarted (reported with keepalive +
flush_thread_count > 1).

Add ConnectionManager#discard, which revokes a keepalive socket (or closes a
non-keepalive one) instead of checking it back in, and use it from ack_check
on the FAILED and CHUNKID_UNMATCHED paths so the tainted connection is torn
down and the next chunk uses a fresh connection with a clean ack stream.
Successful acks still reuse the socket as before.

Fixes fluent#4057

Signed-off-by: JSup <hanjisang0914@gmail.com>
Comment on lines +94 to +95
sock.close_write rescue nil
sock.close rescue nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nits] These line now appears in

sock.close_write rescue nil
sock.close rescue nil

When you add a tiny private method (like a close_socket), it would remove duplicates.

@Watson1978 Watson1978 added this to the v1.20.0 milestone Jul 16, 2026
@Watson1978 Watson1978 added the backport to v1.19 We will backport this fix to the LTS branch label Jul 16, 2026
@Watson1978 Watson1978 requested a review from kenhys July 16, 2026 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport to v1.19 We will backport this fix to the LTS branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fluentd Forwarder pods getting flooded with warning messages when used in "Forwarder-Aggregator" mode.

2 participants