Possible fixes: Leader Election Issue (CHEF-15608) #9400
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Allow a member in `Suspect` health to participate in the `electorate`. 2. Log failure of `check_quorum` as `warn!` when `check_quorum` fails. 3. Reverse the order of `insert_mlw_` when processing `Ping`. 4. Added `insert_mlw_` when processing `PingReq` or `Ack` to mark the originator as alive. Signed-off-by: Abhijit Gadgil <[email protected]>
👷 Deploy Preview for chef-habitat processing.
|
Signed-off-by: Matt Wrock <[email protected]>
Signed-off-by: Matt Wrock <[email protected]>
If we receive member list in `Ping` or `Ack` from a `Confirmed` member, likely that member was in a network partition and came back, this causes the member in network partition to overwhelm the ring with `Confirmed` supervisors that takes a while to converge. This would also mean for a node emerging out of network partition, it has to receive at-least two messages (Ping and/or Ack) from another member before that member's membership list can be accepted and stored. This might be a small extra time to converge *after* network partition, but is better than the entire ring diverging completely. Signed-off-by: Abhijit Gadgil <[email protected]>
Signed-off-by: Matt Wrock <[email protected]>
1. Always mark ourselves as Alive 2. Mark Sender as Alive (regardless of Incarnation) 3. Ignore memberlist if Sender is currently not `Alive` or `Suspect` 4. If we mark Sender as alive and state is updated , purge it's rumours Signed-off-by: Abhijit Gadgil <[email protected]>
Signed-off-by: Abhijit Gadgil <[email protected]>
When a member becomes `Alive` after being `Suspect` and/or `Confirmed`, we have added a cool-off interval during which rumors are not sent to the member. This allows that member from receiving incorrect information (especially if coming out of a partition) Signed-off-by: Abhijit Gadgil <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Suspecthealth to participate in theelectorate.check_quorumaswarn!whencheck_quorumfails.insert_mlw_when processingPing.insert_mlw_when processingPingReqorAckto mark the originator as alive.