Casual Cluster Leader not electing

[fme_mrc]

i have scenario where i have setup 3 node casual cluster let's call them a,b,c.

Now i'm adding 3 more nodes d,e,f with initial discovery type a,b,c.

Now when i remove a,b,c nodes together. d,e,f nodes are unable to elect a leader.

fyi, Initial discovery is 3 and Minimum runtime nodes 3

[david_allen]

This is not yet detailed enough to diagnose or solve. There is a variety of things that could be happening here.

When the new nodes join the cluster, they must bind, get cluster state, and catch up. In order for the new set to elect a new leader, they have to have consensus on where they are. Usually to figure this out you'd want to look at the debug.log files for all 6 nodes. You'd first be looking for some evidence that the new 3 joined and caught up properly. Then when the previous 3 exited, you'd look for leader re-election.

But strange about this scenario is that when the first 3 exit, none of your initial set is left. What motivates this scenario?

[andrew_bowman]

Note that a majority quorum is required for the cluster to maintain write operation (and also to have a cluster leader).

If all nodes in the cluster joined and caught up successfully, you would have had a 6-node cluster. If you brought down the first 3 nodes at the same time, you now have 3/6 cluster members, you've lost quorum, so no write access and no leader.

If you instead do a gradual removal of members such that the cluster can scale down, then you can tolerate dropping down to a 3-node cluster.

Quorum is required for commits, adding, and removing members from a cluster, and member changes will alter the number of online nodes required for quorum.

Raft operations are sequential, so when there are a majority of nodes able to reach consensus on voting in or out cluster members, that implies that they are caught up on all previous commit operations.