io_wait: run the fd-map consistency check only with EXTRA_DEBUG - #4221
Open
Lt-Flash wants to merge 1 commit into
Open
io_wait: run the fd-map consistency check only with EXTRA_DEBUG#4221Lt-Flash wants to merge 1 commit into
Lt-Flash wants to merge 1 commit into
Conversation
check_io_data() walks every entry of the reactor's fd map - max_fd_no of them, which is the reactor size, which follows open_files_limit - on every io_watch_add() and every io_watch_del(). A debugging aid that costs O(reactor size) per async registration or removal: on a node whose reactor holds 21k entries each call takes ~0.6 ms, and a module doing a few thousand async operations a second (cross-node cache pulls here, but any async DB or REST traffic alike) spends two cores in it - 13.6% of all CPU on a 16-core host under profile, the largest user-space symbol of the run. The check is now compiled only with EXTRA_DEBUG; the plain build keeps the error flag cleared and does nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
check_io_data()inio_wait.his a consistency pass over the reactor's fd map — every one of themax_fd_noentries — and it runs on everyio_watch_add()andio_watch_del(). The macro is defined unconditionally, so the plain build pays it too.That makes each async registration or removal O(reactor size), i.e. O(
open_files_limit). Measured on a 16-core host with a 21k-entry reactor (open_files_limit65536, reactor shrunk to 20971 by the pkg budget): ~0.6 ms per call. A module doing a few thousand async operations a second — here cross-node cache pulls through the async framework, but any async DB or REST traffic takes the same path — spent about two cores in it:io_watch_add/io_watch_delwere the largest user-space symbols of aperfprofile at 13.6% of all CPU, and the receive buffers of the SIP sockets overflowed (129UdpRcvbufErrorsover a run) because the workers were busy walking the map.Change
The check is compiled only with
EXTRA_DEBUG; the plain build keepscheck_errorcleared and does nothing. No behaviour change otherwise.Effect
Same workload, same host, before → after: async-heavy worker CPU down by the two cores, UDP receive drops 129 → 0, request p95 in the early part of the run 0.97 → 0.91 ms; the check still runs in
EXTRA_DEBUGbuilds.Built with gcc and clang under
-Werror.