RATIS-2599. Add design for listener delegation from followers#1513
RATIS-2599. Add design for listener delegation from followers#1513spacemonkd wants to merge 1 commit into
Conversation
|
@szetszwo could you take a look at the high level design document? |
szetszwo
left a comment
szetszwo
left a comment
There was a problem hiding this comment.
@spacemonkd , thanks for working on this! Please see the comments inlined.
| │ │──appendEntries──► Listener1 │ │──heartbeat-only──► Listener1 | ||
| │ │──appendEntries──► Listener2 │ │──heartbeat-only──► Listener2 |
There was a problem hiding this comment.
It is good to have heartbeat directly from the leader 👍
|
|
||
| ### Key Invariants | ||
|
|
||
| 1. **Leader authority:** The leader makes all delegation decisions. Followers do not self-elect to serve listeners. |
There was a problem hiding this comment.
It is a good start to have Leader making all delegation decisions.
Later on, we may let the Listener to select which follower to get appendEntries from.
BTW, use short lines for doc. It is easier to comment and make changes.
1. **Leader authority:**
The leader makes all delegation decisions.
Followers do not self-elect to serve listeners.| 3. **Term safety:** Listeners validate that `leaderTerm >= currentTerm` in delegated entries. A deposed leader's follower will have a stale term, so listeners reject stale entries automatically. | ||
|
|
||
| 4. **Log consistency:** The standard Raft log consistency check (`previousLog` matching) still runs on the listener side preventing any divergence regardless of the source. | ||
|
|
||
| 5. **Full catch-up support:** The follower replicator can serve listeners from any `nextIndex` using its local log — not limited to near-real-time forwarding. |
There was a problem hiding this comment.
Is it the same as before? If yes, let's emphasize it as the "same as before".
| message ListenerProgressProto { | ||
| RaftPeerIdProto listenerId = 1; | ||
| uint64 matchIndex = 2; | ||
| uint64 nextIndex = 3; | ||
| uint64 commitIndex = 4; | ||
| } |
There was a problem hiding this comment.
This may not be needed since the Leader heartbeat to the listeners. The listeners can reply to the Leader directly.
| | `follower-lag-threshold` | `100` | Max entries a follower can lag behind leader and still be eligible | | ||
| | `reassignment-interval` | `30s` | How often the leader re-evaluates assignments | | ||
|
|
||
| ### 3. Leader-side: Selection and Assignment |
There was a problem hiding this comment.
We should just change setConf to let the admin setting. They could set it according to the topology. Automatic selection may not be useful.
2 participants
What changes were proposed in this pull request?
Add design for listener delegation from followers
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/RATIS-2599
How was this patch tested?
N/A