Error Failed to start Neo4j after power outage

[atesterguy]

After our ESXI server shutdown due to a power outage (a UPS will be added VERY soon after this incident) my neo4j server wont start again.

Here is the error message I receive when I try to start the server.
Any help would be greatly appreciated.

2020-05-07 16:13:37.625+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@1f67761b' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1587666667320, randomId=-1079962435470005166, storeVersion=3471765337752883975, upgradeTime=1587666667320, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586700350222, randomId=-2629912999185239023, storeVersion=3471765337752883975, upgradeTime=1586700350222, upgradeTxId=1}". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@1f67761b' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1587666667320, randomId=-1079962435470005166, storeVersion=3471765337752883975, upgradeTime=1587666667320, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586700350222, randomId=-2629912999185239023, storeVersion=3471765337752883975, upgradeTime=1586700350222, upgradeTxId=1}".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@1f67761b' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1587666667320, randomId=-1079962435470005166, storeVersion=3471765337752883975, upgradeTime=1587666667320, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586700350222, randomId=-2629912999185239023, storeVersion=3471765337752883975, upgradeTime=1586700350222, upgradeTxId=1}".
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:164)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:114)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:89)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@1f67761b' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1587666667320, randomId=-1079962435470005166, storeVersion=3471765337752883975, upgradeTime=1587666667320, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586700350222, randomId=-2629912999185239023, storeVersion=3471765337752883975, upgradeTime=1586700350222, upgradeTxId=1}".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:157)
... 3 more
Caused by: java.lang.RuntimeException: Error starting database server at /var/lib/neo4j/data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:167)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:145)
at org.neo4j.server.database.CommunityGraphFactory.newDatabaseManagementService(CommunityGraphFactory.java:36)
at org.neo4j.server.database.LifecycleManagingDatabaseService.start(LifecycleManagingDatabaseService.java:88)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.dbms.database.DefaultSystemGraphInitializer@545e57d7' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1587666667320, randomId=-1079962435470005166, storeVersion=3471765337752883975, upgradeTime=1587666667320, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586700350222, randomId=-2629912999185239023, storeVersion=3471765337752883975, upgradeTime=1586700350222, upgradeTxId=1}".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:158)
... 9 more
Caused by: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
at org.neo4j.kernel.availability.DatabaseAvailabilityGuard.assertDatabaseAvailable(DatabaseAvailabilityGuard.java:172)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginKernelTransaction(GraphDatabaseFacade.java:185)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:165)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:119)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:113)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:101)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:96)
at org.neo4j.dbms.database.DefaultSystemGraphInitializer.isSystemGraphEmpty(DefaultSystemGraphInitializer.java:100)
at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:76)
at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:69)
at org.neo4j.dbms.database.SystemGraphInitializer.start(SystemGraphInitializer.java:30)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
... 11 more
Caused by: java.lang.RuntimeException: Mismatching store id. Store StoreId: StoreId{creationTime=1587666667320, randomId=-1079962435470005166, storeVersion=3471765337752883975, upgradeTime=1587666667320, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586700350222, randomId=-2629912999185239023, storeVersion=3471765337752883975, upgradeTime=1586700350222, upgradeTxId=1}
at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:386)
at org.neo4j.kernel.database.Database.start(Database.java:377)
at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:132)
at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:36)
at org.neo4j.dbms.database.AbstractDatabaseManager.forEachDatabase(AbstractDatabaseManager.java:165)
at org.neo4j.dbms.database.AbstractDatabaseManager.startAllDatabases(AbstractDatabaseManager.java:112)
at org.neo4j.dbms.database.AbstractDatabaseManager.start(AbstractDatabaseManager.java:106)

[stefan_armbrust]

First take a backup copy of your graph.db folder into a safe location.

Remove the transaction log files (in 3.x you'll find them in the graph.db folder) and try to start the db up.

Which neo4j version are you using?

[atesterguy]

Hi im using version 4.0.4
And it seems like this idea did not work.
I added the dbms.recovery.fail_on_missing_files=false so it will create a new file but now the error says:

ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@343fddd9' was successfully initialized, but failed to start. Please see the attached cause exception "Index clean jobs can't be added after collector start. | GB+Tree[file:/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1/index-1, layout:GenericLayout[version:0.5, identifier:5643935348791705601, fixedSize:false], generation:6/8]". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@343fddd9' was successfully initialized, but failed to start. Please see the attached cause exception "Index clean jobs can't be added after collector start. | GB+Tree[file:/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1/index-1, layout:GenericLayout[version:0.5, identifier:5643935348791705601, fixedSize:false], generation:6/8]".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@343fddd9' was successfully initialized, but failed to start. Please see the attached cause exception "Index clean jobs can't be added after collector start. | GB+Tree[file:/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1/index-1, layout:GenericLayout[version:0.5, identifier:5643935348791705601, fixedSize:false], generation:6/8]".
        at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:164)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:114)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:89)
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@343fddd9' was successfully initialized, but failed to start. Please see the attached cause exception "Index clean jobs can't be added after collector start. | GB+Tree[file:/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1/index-1, layout:GenericLayout[version:0.5, identifier:5643935348791705601, fixedSize:false], generation:6/8]".
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:157)
        ... 3 more
Caused by: java.lang.RuntimeException: Error starting database server at /var/lib/neo4j/data/databases
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:167)
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:145)
        at org.neo4j.server.database.CommunityGraphFactory.newDatabaseManagementService(CommunityGraphFactory.java:36)
        at org.neo4j.server.database.LifecycleManagingDatabaseService.start(LifecycleManagingDatabaseService.java:88)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
        ... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.dbms.database.DefaultSystemGraphInitializer@545e57d7' was successfully initialized, but failed to start. Please see the attached cause exception "Index clean jobs can't be added after collector start. | GB+Tree[file:/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1/index-1, layout:GenericLayout[version:0.5, identifier:5643935348791705601, fixedSize:false], generation:6/8]".
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:158)
        ... 9 more
Caused by: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
        at org.neo4j.kernel.availability.DatabaseAvailabilityGuard.assertDatabaseAvailable(DatabaseAvailabilityGuard.java:172)
        at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginKernelTransaction(GraphDatabaseFacade.java:185)
        at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:165)
        at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:119)
        at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:113)
        at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:101)
        at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:96)
        at org.neo4j.dbms.database.DefaultSystemGraphInitializer.isSystemGraphEmpty(DefaultSystemGraphInitializer.java:100)
        at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:76)
        at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:69)
        at org.neo4j.dbms.database.SystemGraphInitializer.start(SystemGraphInitializer.java:30)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
        ... 11 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.api.index.IndexingService@732f6050' failed to initialize. Please see the attached cause exception "Index clean jobs can't be added after collector start. | GB+Tree[file:/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1/index-1, layout:GenericLayout[version:0.5, identifier:5643935348791705601, fixedSize:false], generation:6/8]".
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:426)
        at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
        at org.neo4j.kernel.recovery.TransactionLogsRecovery.start(TransactionLogsRecovery.java:192)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.kernel.recovery.Recovery.performRecovery(Recovery.java:364)
        at org.neo4j.kernel.database.Database.start(Database.java:380)
        at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
        at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:132)
        at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:36)
        at org.neo4j.dbms.database.AbstractDatabaseManager.forEachDatabase(AbstractDatabaseManager.java:165)
        at org.neo4j.dbms.database.AbstractDatabaseManager.startAllDatabases(AbstractDatabaseManager.java:112)
        at org.neo4j.dbms.database.AbstractDatabaseManager.start(AbstractDatabaseManager.java:106)
        ... 12 more
Caused by: java.lang.IllegalStateException: Index clean jobs can't be added after collector start. | GB+Tree[file:/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1/index-1, layout:GenericLayout[version:0.5, identifier:5643935348791705601, fixedSize:false], generation:6/8]
        at org.neo4j.util.Preconditions.checkState(Preconditions.java:173)
        at org.neo4j.index.internal.gbptree.GroupingRecoveryCleanupWorkCollector.add(GroupingRecoveryCleanupWorkCollector.java:60)
        at org.neo4j.index.internal.gbptree.GBPTree.createCleanupJob(GBPTree.java:1336)
        at org.neo4j.index.internal.gbptree.GBPTree.<init>(GBPTree.java:541)
        at org.neo4j.kernel.impl.index.schema.NativeIndex.instantiateTree(NativeIndex.java:69)
        at org.neo4j.kernel.impl.index.schema.GenericNativeIndexAccessor.<init>(GenericNativeIndexAccessor.java:60)
        at org.neo4j.kernel.impl.index.schema.GenericNativeIndexProvider.newIndexAccessor(GenericNativeIndexProvider.java:185)
        at org.neo4j.kernel.impl.index.schema.GenericNativeIndexProvider.newIndexAccessor(GenericNativeIndexProvider.java:105)
        at org.neo4j.kernel.impl.index.schema.NativeIndexProvider.getOnlineAccessor(NativeIndexProvider.java:104)
        at org.neo4j.kernel.impl.index.schema.GenericNativeIndexProvider.getOnlineAccessor(GenericNativeIndexProvider.java:105)
        at org.neo4j.kernel.impl.api.index.IndexProxyCreator.onlineAccessorFromProvider(IndexProxyCreator.java:159)
        at org.neo4j.kernel.impl.api.index.IndexProxyCreator.createOnlineIndexProxy(IndexProxyCreator.java:106)
        at org.neo4j.kernel.impl.api.index.IndexingService.lambda$init$1(IndexingService.java:259)
        at org.neo4j.kernel.impl.api.index.IndexMapReference.modify(IndexMapReference.java:53)
        at org.neo4j.kernel.impl.api.index.IndexingService.init(IndexingService.java:241)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:405)

[stefan_armbrust · Accepted solution]

Given you have a backup copy, try to recursively remove the schema subfolder. Don't know for sure but this might force a full index rebuild.

[atesterguy]

Thank you very much !
I have deleted the index in the system folder and it seemed to have fixed the issue.