Not able to start neo4j server

[anitha_thota]

Hi Team,
We have two neo4j servers which are in cluster. When I am trying to restart the server, its showing Server is running at so and so port. But when I am trying to open it from browser, its not opening. After some time its getting shut down with attached error.
can you please let me know what can be the issue.

Regards,
Anitha

[anitha_thota]

below is the error we are getting.

019-11-14 10:17:00.187+0000 WARN  Unknown config option: dbms.security.ldap.authentication.user_search_attribute
2019-11-14 10:17:00.206+0000 INFO  ======== Neo4j 3.4.9 ========
2019-11-14 10:17:00.232+0000 INFO  Starting...
2019-11-14 10:17:01.452+0000 INFO  Initiating metrics...
2019-11-14 10:17:01.520+0000 INFO  My connection info: [
	Discovery:   listen=10.254.36.80:5000, advertised=10.254.36.80:5000,
	Transaction: listen=0.0.0.0:6000, advertised=10.254.36.80:6000, 
	Raft:        listen=0.0.0.0:7000, advertised=10.254.36.80:7000, 
	Client Connector Addresses: bolt://10.254.36.80:7687,http://10.254.36.80:7474,https://10.254.36.80:7473
]
2019-11-14 10:17:01.520+0000 INFO  Discovering cluster with initial members: [10.254.36.17:5000, 10.254.36.80:5000]
2019-11-14 10:17:01.520+0000 INFO  Attempting to connect to the other cluster members before continuing...
Exception in thread "neo4j.FileWatcher-1" java.nio.file.ClosedWatchServiceException
	at sun.nio.fs.AbstractWatchService.checkOpen(AbstractWatchService.java:80)
	at sun.nio.fs.AbstractWatchService.checkKey(AbstractWatchService.java:92)
	at sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:119)
	at org.neo4j.io.fs.watcher.DefaultFileSystemWatcher.startWatching(DefaultFileSystemWatcher.java:76)
	at org.neo4j.io.fs.watcher.RestartableFileSystemWatcher.startWatching(RestartableFileSystemWatcher.java:94)
	at org.neo4j.kernel.impl.util.watcher.DefaultFileSystemWatcherService$FileSystemEventWatcher.run(DefaultFileSystemWatcherService.java:92)
	at java.lang.Thread.run(Thread.java:748)
2019-11-14 10:18:41.105+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1b560eb0' was successfully initialized, but failed to start. Please see the attached cause exception "Out of order transaction. Expected that 6782114 < 6047567". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1b560eb0' was successfully initialized, but failed to start. Please see the attached cause exception "Out of order transaction. Expected that 6782114 < 6047567".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1b560eb0' was successfully initialized, but failed to start. Please see the attached cause exception "Out of order transaction. Expected that 6782114 < 6047567".
	at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:220)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:111)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:79)
	at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1b560eb0' was successfully initialized, but failed to start. Please see the attached cause exception "Out of order transaction. Expected that 6782114 < 6047567".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:212)
	... 3 more

[david_allen]

It looks like a possibility here is that the database members disagree about the transaction history here. The reason you're not starting is that the transaction history has to be consistent in order to apply a history from one member to others.

There's two potential causes that I can imagine here, do any of these sound like they apply?

1. The cluster members have different databases or differing versions of the same database
2. You had a "split brain" for a while, where you had two different independent clusters whose history diverged