Only restore a retried parent when the retry succeeds - #13474
Open
masaori335 wants to merge 1 commit into
Open
Conversation
A parent that accepts connections but never sends a response was marked down and then restored to the pool by the very next retry probe, even when that probe also failed, so it kept receiving traffic indefinitely while healthy peers took none of the load. handle_response_from_parent() called markParentUp() on entry, before the switch on current.state, so a parent selected as a retry candidate was restored before ATS knew the retry's outcome. markParentUp() zeroes failedAt and failCount; the markParentDown() that follows a failed probe then takes its result->retry branch, which leaves new_fail_count at 0, so available is never re-cleared. The parent therefore returned to the pool with a cleared failure count after every retry_time window, no matter how long it stayed degraded. The CONNECTION_ALIVE branch already calls markParentUp() for a retry that actually succeeded, which is the correct restore point. Removing the entry call restores the behavior that predates apache#8546: that commit backed out the retry-limiting work and replaced a retryComplete() call -- which only decremented an in-flight retry counter and never touched availability -- with markParentUp(), silently turning "the retry finished" into "the parent is healthy". Two autests pin both outcomes of the retry path. A parent that is still silent when probed stays down; a parent that has recovered is restored. Each was negative-controlled against its own call site: reverting this change fails the first, and removing the CONNECTION_ALIVE call fails only the second.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes parent retry availability handling in ATS’s HTTP transaction path so a parent marked down is only restored when a retry probe actually succeeds, preventing a degraded-but-connecting parent from re-entering the pool after every retry_time window.
Changes:
- Remove the unconditional
markParentUp()call at entry toHttpTransact::handle_response_from_parent()for retried parents, so restore happens only on success (CONNECTION_ALIVE). - Add two AuTest replay scenarios that pin both retry outcomes: failed retry keeps the parent down; successful retry restores it.
- Document the corrected
retry_timesemantics in the admin guide.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
src/proxy/http/HttpTransact.cc |
Stops restoring a retried parent before the retry outcome is known; restore remains on the success path. |
tests/gold_tests/parent_proxy/parent_retry_availability.test.py |
Adds a gold test that runs both retry-outcome replays. |
tests/gold_tests/parent_proxy/replays/parent_retry_failure_stays_down.replay.yaml |
New replay asserting a failed retry probe does not restore a degraded parent. |
tests/gold_tests/parent_proxy/replays/parent_retry_success_restores.replay.yaml |
New replay asserting a successful retry probe restores the parent to the pool. |
doc/admin-guide/files/records.yaml.en.rst |
Clarifies retry_time behavior: restore only on successful retry, otherwise remain unavailable until next window. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A parent that accepts connections but never sends a response was marked down and then restored to the pool by the very next retry probe, even when that probe also failed, so it kept receiving traffic indefinitely while healthy peers took none of the load.
handle_response_from_parent() called markParentUp() on entry, before the switch on current.state, so a parent selected as a retry candidate was restored before ATS knew the retry's outcome. markParentUp() zeroes failedAt and failCount; the markParentDown() that follows a failed probe then takes its result->retry branch, which leaves new_fail_count at 0, so available is never re-cleared. The parent therefore returned to the pool with a cleared failure count after every retry_time window, no matter how long it stayed degraded.
The CONNECTION_ALIVE branch already calls markParentUp() for a retry that actually succeeded, which is the correct restore point. Removing the entry call restores the behavior that predates #8546: that commit backed out the retry-limiting work and replaced a retryComplete() call -- which only decremented an in-flight retry counter and never touched availability -- with markParentUp(), silently turning "the retry finished" into "the parent is healthy".
Two autests pin both outcomes of the retry path. A parent that is still silent when probed stays down; a parent that has recovered is restored. Each was negative-controlled against its own call site: reverting this change fails the first, and removing the CONNECTION_ALIVE call fails only the second.