Unable to add additional core node to existing cluster

[davehagler]

I have a causal cluster with 3 core members and 1 replica running Neo4j enterprise 3.4.4. The database has around 10G of data. I am trying to add more core members. Following the documentation, I have a new core server configured exactly as the other cores, with the same initial_discovery_members. When the new core starts up it gets stuck waiting for members and never gets past this point.

INFO [c.n.c.d.SslHazelcastCoreTopologyService] Discovering cluster with initial members
INFO [c.n.c.d.SslHazelcastCoreTopologyService] Attempting to connect to the other cluster members before continuing...
INFO [o.n.c.c.c.l.s.SegmentedRaftLog] log started with recovered state State{prevIndex=-1, prevTerm=-1, appendIndex=-1}
INFO [o.n.c.c.c.m.RaftMembershipManager] Membership state before recovery: RaftMembershipState{committed=null, appended=null, ordinal=-1}
INFO [o.n.c.c.c.m.RaftMembershipManager] Recovering from: -1 to: -1
INFO [o.n.c.c.c.m.RaftMembershipManager] Membership state after recovery: RaftMembershipState{committed=null, appended=null, ordinal=-1}
INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: []
INFO [o.n.c.n.Server] raft-server: bound to 0.0.0.0:7000
INFO [o.n.c.i.ClusterBinder] Waiting for 3 members. Currently discovered 0 members: {}.
INFO [o.n.c.i.ClusterBinder] Waiting for 3 members. Currently discovered 0 members: {}.
INFO [o.n.c.i.ClusterBinder] Waiting for 3 members. Currently discovered 0 members: {}.
INFO [o.n.c.i.ClusterBinder] Waiting for 3 members. Currently discovered 0 members: {}.

[michael_hunger]

Does the network visibility between the machines work?
Looks as if it is just bound to the public interface not to a specific IP?

[davehagler]

Yes, I verified via telnet that it can reach the other core members on port 5000. It's using discovery type DNS. I left this part of the log out. The DnsHostnameResolver found the correct member IPs. I also tried using discovery type LIST and explicitly listing the other core members but same result.

INFO [o.n.c.d.DnsHostnameResolver] Resolved initial host 'myservers:5000' to [10.194.251.128:5000, 10.194.249.75:5000, 10.194.250.254:5000]
INFO [c.n.c.d.SslHazelcastCoreTopologyService] My connection info: [
Discovery: listen=0.0.0.0:5000, advertised=myserver1:5000,
Transaction: listen=0.0.0.0:6000, advertised=myserver1:6000,
Raft: listen=0.0.0.0:7000, advertised=myserver1:7000,
Client Connector Addresses: bolt://myserver1:7687,http://myserver1:7474,https://myserver1:7473
]
INFO [c.n.c.d.SslHazelcastCoreTopologyService] Discovering cluster with initial members: [myservers:5000]