Fix bulk data download stalling over WAN / high-latency links (#31) - #61
Open
hmshb wants to merge 1 commit into
Open
Fix bulk data download stalling over WAN / high-latency links (#31)#61hmshb wants to merge 1 commit into
hmshb wants to merge 1 commit into
Conversation
…inks readWithBuffer() returns the full reply on a LAN but truncates to the first ~2 chunks (often surfacing as "0 logs" or only old records) when the device is reached across the internet / a slow link. Long-standing pain point — see caobo171#31. - Request chunks sequentially (send one, await its full arrival, then request the next) instead of firing every sendChunkRequest up front. Firing all at once works on a LAN but over a high-latency link the device's send buffer fills faster than the link drains and the later chunks never arrive, truncating the download. One chunk in flight lets the slow link keep up (same approach pyzk uses). - Respect the constructor `timeout` for the inter-packet wait instead of the fixed TIMEOUTS.CHUNK_TCP (falls back to it when unset). - New optional `maxChunk` arg on ZKLib/ZKLibTCP (default unchanged). Lowering it (e.g. 8184) makes each chunk individually more robust on poor links. Fully backward compatible. Verified against a live ZKTeco K50 behind PPPoE: stalled at ~3,200 records with defaults, pulled the full ~9,900 in ~18s with maxChunk=8184.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
readWithBuffer()(and thereforegetAttendances()) works fine on a LAN but truncates the download to roughly the first ~2 chunks when the device is reached across the internet or any slow / high-latency link. It usually shows up asFetched 0 logsor only stale records, with recent entries (at the tail of the history) never arriving. This is the issue reported in #31 ("get attendance timeout with large amounts of data").I traced it on a live ZKTeco K50 behind PPPoE, polled from a cloud server: connect, voice test and record-count all worked; only the bulk download stalled, consistently at the same byte boundary regardless of timeout.
Root causes
All chunk requests were fired up front (
for (i=0..numberChunks) sendChunkRequest(...)). On a LAN the device keeps up. Over a high-latency link its send buffer fills faster than the link drains, the remaining chunks never arrive, and the stream wedges — truncating the download.The inter-packet timeout (
TIMEOUTS.CHUNK_TCP) ignored the constructortimeout, so slow links couldn't be given more room.Changes (backward compatible)
pyzk.)timeoutfor the inter-packet wait, falling back toTIMEOUTS.CHUNK_TCPwhen unset.maxChunkargument onZKLib/ZKLibTCP(default unchanged). Lowering it (e.g.8184) makes each chunk individually more robust on poor links.Result
On the real K50 over the WAN link: stalled at ~3,200 records (months-old data) with defaults → pulled the full ~9,900 records including same-day punches in ~18s with
maxChunk=8184.No API breakage — omit
maxChunkand behaviour is identical. Verified against a live device;node -cclean. Happy to adjust naming/placement to fit your conventions.