Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

raft: Fix infinite election loop #1310

Merged
merged 2 commits into from
Aug 4, 2016
Merged

raft: Fix infinite election loop #1310

merged 2 commits into from
Aug 4, 2016

Commits on Aug 4, 2016

  1. raft: Fix infinite election loop 

    It looks like the election loops on startup are caused by something
proposing a value before all entries in the log have become committed.
This would cause ApplyStoreActions to try and get a lock on the raft
store, while processInternalRaftRequest is holding the lock (since it's
called by the memory store with the lock held). This situation is a
deadlock. It blocks the loop in Node.Run and prevents further
participation in the cluster.

To try and solve the issue, don't signal that the local node has become
the leader until it has committed all the uncommitted entries from
before. Also, block proposals until it is caught up.

Signed-off-by: Aaron Lehmann <[email protected]>
    @aaronlehmann
    aaronlehmann committed Aug 4, 2016
    Configuration menu
    Copy the full SHA
    64ffb46 View commit details
    Browse the repository at this point in the history

  2. raft: Don't forward or accept proposals 

    Only the leader should make proposals, so we can guarantee that no two
proposals will conflict. This commit prevents a follower from sending a
proposal, or the leader from accepting one. We already have code to
prevent this, but there are certain corner cases where a leader could
become a follower while it's proposing a value.

Signed-off-by: Aaron Lehmann <[email protected]>
    @aaronlehmann
    aaronlehmann committed Aug 4, 2016
    Configuration menu
    Copy the full SHA
    7e70556 View commit details
    Browse the repository at this point in the history