feat(deliverymq): bound retry task receives with backoff and DLQ#997
Open
LendritIbrahimi wants to merge 1 commit into
Open
feat(deliverymq): bound retry task receives with backoff and DLQ#997LendritIbrahimi wants to merge 1 commit into
LendritIbrahimi wants to merge 1 commit into
Conversation
Failed retry tasks now back off exponentially using rsmq's receive count, instead of reappearing every 30s forever. Tasks received more than RETRY_MAX_RECEIVE_COUNT times are moved to the deliverymq-retry-dlq queue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes the issue where a single lost attempt log produced an unbounded 30s warning loop that couldn't be turned off by destination disable or pod restarts. (#663)
The implementation follows the suggestion in the linked issue, failed retry tasks now back off exponentially using rsmq's receive count, instead of reappearing every 30s forever. Tasks received more than RETRY_MAX_RECEIVE_COUNT times are moved to the deliverymq-retry-dlq queue.
The move to the DLQ is send-then-delete, so a failure between the two steps can only duplicate a task, never lose one, and landing there emits an error log with the full task payload for operator visibility. RETRY_MAX_RECEIVE_COUNT defaults to 5 and can be set to 0 to disable the DLQ path entirely.