Skip to content

io_wait: run the fd-map consistency check only with EXTRA_DEBUG - #4221

Open
Lt-Flash wants to merge 1 commit into
OpenSIPS:masterfrom
Lt-Flash:fix/io-wait-check-io-data
Open

io_wait: run the fd-map consistency check only with EXTRA_DEBUG#4221
Lt-Flash wants to merge 1 commit into
OpenSIPS:masterfrom
Lt-Flash:fix/io-wait-check-io-data

Conversation

@Lt-Flash

Copy link
Copy Markdown

Summary

check_io_data() in io_wait.h is a consistency pass over the reactor's fd map — every one of the max_fd_no entries — and it runs on every io_watch_add() and io_watch_del(). The macro is defined unconditionally, so the plain build pays it too.

That makes each async registration or removal O(reactor size), i.e. O(open_files_limit). Measured on a 16-core host with a 21k-entry reactor (open_files_limit 65536, reactor shrunk to 20971 by the pkg budget): ~0.6 ms per call. A module doing a few thousand async operations a second — here cross-node cache pulls through the async framework, but any async DB or REST traffic takes the same path — spent about two cores in it: io_watch_add/io_watch_del were the largest user-space symbols of a perf profile at 13.6% of all CPU, and the receive buffers of the SIP sockets overflowed (129 UdpRcvbufErrors over a run) because the workers were busy walking the map.

Change

The check is compiled only with EXTRA_DEBUG; the plain build keeps check_error cleared and does nothing. No behaviour change otherwise.

Effect

Same workload, same host, before → after: async-heavy worker CPU down by the two cores, UDP receive drops 129 → 0, request p95 in the early part of the run 0.97 → 0.91 ms; the check still runs in EXTRA_DEBUG builds.

Built with gcc and clang under -Werror.

check_io_data() walks every entry of the reactor's fd map - max_fd_no
of them, which is the reactor size, which follows open_files_limit - on
every io_watch_add() and every io_watch_del(). A debugging aid that
costs O(reactor size) per async registration or removal: on a node
whose reactor holds 21k entries each call takes ~0.6 ms, and a module
doing a few thousand async operations a second (cross-node cache pulls
here, but any async DB or REST traffic alike) spends two cores in it -
13.6% of all CPU on a 16-core host under profile, the largest user-space
symbol of the run.

The check is now compiled only with EXTRA_DEBUG; the plain build keeps
the error flag cleared and does nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant