Neo4j v4.0 `Mismatching store id`

[wdrosko]

I'm running Neo4j v4.0. Today I decided to reload a database with the admin import tool (since this is how we are loading our graph) and it successfully runs. The tool imports all of the nodes and relationships without fail. Additionally, there is nothing recorded about any errors in the logs related to the import.

I'm using the default graph names that v4 uses: neo4j and system

When I try to start the database it fails to start and I've seen a couple of people post about this regarding v4 but haven't seen any solution that helps me nail down the issue:

2020-03-17 22:49:52.260+0000 ERROR [o.n.g.f.DatabaseManagementServiceFactory] Error starting database server at /usr/local/neo4j/data/databases Component 'org.neo4j.dbms.database.DefaultSystemGraphInitializer@16c8b7bd' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1584484410778, randomId=8700672283562579661, storeVersion=3471765337752883975, upgradeTime=1584484410778, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1582566686283, randomId=8599113573121817080, storeVersion=3471765337752883975, upgradeTime=1582566686283, upgradeTxId=1}".
org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.dbms.database.DefaultSystemGraphInitializer@16c8b7bd' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1584484410778, randomId=8700672283562579661, storeVersion=3471765337752883975, upgradeTime=1584484410778, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1582566686283, randomId=8599113573121817080, storeVersion=3471765337752883975, upgradeTime=1582566686283, upgradeTxId=1}".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:158)
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:145)
	at org.neo4j.server.database.CommunityGraphFactory.newDatabaseManagementService(CommunityGraphFactory.java:36)
	at org.neo4j.server.database.LifecycleManagingDatabaseService.start(LifecycleManagingDatabaseService.java:88)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:156)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:114)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:89)
	at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35)
Caused by: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
	at org.neo4j.kernel.availability.DatabaseAvailabilityGuard.assertDatabaseAvailable(DatabaseAvailabilityGuard.java:172)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginKernelTransaction(GraphDatabaseFacade.java:185)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:165)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:119)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:113)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:101)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:96)
	at org.neo4j.dbms.database.DefaultSystemGraphInitializer.isSystemGraphEmpty(DefaultSystemGraphInitializer.java:100)
	at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:76)
	at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:69)
	at org.neo4j.dbms.database.SystemGraphInitializer.start(SystemGraphInitializer.java:30)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
	... 11 more
Caused by: java.lang.RuntimeException: Mismatching store id. Store StoreId: StoreId{creationTime=1584484410778, randomId=8700672283562579661, storeVersion=3471765337752883975, upgradeTime=1584484410778, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1582566686283, randomId=8599113573121817080, storeVersion=3471765337752883975, upgradeTime=1582566686283, upgradeTxId=1}
	at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:386)
	at org.neo4j.kernel.database.Database.start(Database.java:377)
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:132)
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:36)
	at org.neo4j.dbms.database.AbstractDatabaseManager.forEachDatabase(AbstractDatabaseManager.java:165)
	at org.neo4j.dbms.database.AbstractDatabaseManager.startAllDatabases(AbstractDatabaseManager.java:112)
	at org.neo4j.dbms.database.AbstractDatabaseManager.start(AbstractDatabaseManager.java:106)
	... 12 more
2020-03-17 22:49:52.261+0000 INFO [o.n.g.f.DatabaseManagementServiceFactory] Shutdown started

[andrew_bowman]

Hello, which version of Neo4j are you using? 4.0.2 was just released.

[wdrosko]

Hey @andrew.bowman thanks for taking the time, this was originally with v4.0.1

However, I did try a fresh install of v4.0.2 and now I'm having an entirely different issue that is in regards to the browser connection (although I think there may be overlap between this and what I was getting before). So this is now with v4.0.2

Some things to note:

• I am running on an ec2 instance which I can only connect to through my organization's vpn. I've never had an issue with this before after uncommenting the connections 0.0.0.0 line in the .neo4j.conf file and have been able to access and work with my instance and neo4j up until yesterday. All of my respective ports are open in my security group -- like I said I haven't had an issue before now.
• I've had no disruption in my connection to my instances, nor to jupyter running on my instances

I decided to try and just start and load up a fresh session of v4.0.2 without running the import tool just to see if things run smoothly. I'm using the neo4j.conf file that I've attached here.

Now for the fun part:
Upon starting the Neo4j browser (and clearing my browser history) I get the authentication screen to which I enter the defaults. Then upon submitting I get a still connecting prompt followed by the following screen:

After hanging for a bit, it then looks like the session has started and authentication works (I've never had it hang for so long) before. Then to check whether the connection is legit I do a simple MATCH query. This hangs and then I get this error:
WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket readyState is: 3

Running neo4j status tells me that it's running.

I have a v3.5 setup too and as a test I tried running it after failing to get v4.0.2 working. It runs the admin import works perfectly as does the browser. So I'm not entirely sure what's going on.

I am going to try and re-download and use v4.0.1 to see if things work as normal.

Log and config files:
neo4j.log.txt (331 Bytes) neo4j.conf.txt (15.3 KB) debug.log.txt (70.1 KB)

[wdrosko]

It looks like this is related to my issue with the browser: SessionExpired: WebSocket connection failure

I have tried the suggested config settings and yet it is still hanging with the login.

[malik]

I'm experiencing the same issue.

I followed these instructions but on an Ubuntu 18.04 to migrate the store from 3.5.14 to 4.0.3. I've effectively copied the contents of 3.5.14 data folder to data foldedr of 4.0.3. Migration went well according to logs. However, startup is failing.

Directories in use:
  home:         /var/lib/neo4j
  config:       /etc/neo4j
  logs:         /var/log/neo4j
  plugins:      /var/lib/neo4j/plugins
  import:       /var/lib/neo4j/import
  data:         /var/lib/neo4j/data
  certificates: /var/lib/neo4j/certificates
  run:          /var/run/neo4j
Starting Neo4j.
2020-04-06 08:24:50.957+0000 INFO  ======== Neo4j 4.0.3 ========
2020-04-06 08:24:50.964+0000 INFO  Starting...
2020-04-06 08:25:05.423+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@2673ba1f' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1586155486029, randomId=8142000683778803301, storeVersion=3471765337752883975, upgradeTime=1586155486029, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586155296154, randomId=-806197780697231748, storeVersion=3471765337752883975, upgradeTime=1586155296154, upgradeTxId=1}". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@2673ba1f' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1586155486029, randomId=8142000683778803301, storeVersion=3471765337752883975, upgradeTime=1586155486029, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586155296154, randomId=-806197780697231748, storeVersion=3471765337752883975, upgradeTime=1586155296154, upgradeTxId=1}".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@2673ba1f' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1586155486029, randomId=8142000683778803301, storeVersion=3471765337752883975, upgradeTime=1586155486029, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586155296154, randomId=-806197780697231748, storeVersion=3471765337752883975, upgradeTime=1586155296154, upgradeTxId=1}".
	at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:164)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:114)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:89)
	at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabaseService@2673ba1f' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1586155486029, randomId=8142000683778803301, storeVersion=3471765337752883975, upgradeTime=1586155486029, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586155296154, randomId=-806197780697231748, storeVersion=3471765337752883975, upgradeTime=1586155296154, upgradeTxId=1}".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:157)
	... 3 more
Caused by: java.lang.RuntimeException: Error starting database server at /var/lib/neo4j/data/databases
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:167)
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:145)
	at org.neo4j.server.database.CommunityGraphFactory.newDatabaseManagementService(CommunityGraphFactory.java:36)
	at org.neo4j.server.database.LifecycleManagingDatabaseService.start(LifecycleManagingDatabaseService.java:88)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
	... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.dbms.database.DefaultSystemGraphInitializer@769a58e5' was successfully initialized, but failed to start. Please see the attached cause exception "Mismatching store id. Store StoreId: StoreId{creationTime=1586155486029, randomId=8142000683778803301, storeVersion=3471765337752883975, upgradeTime=1586155486029, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586155296154, randomId=-806197780697231748, storeVersion=3471765337752883975, upgradeTime=1586155296154, upgradeTxId=1}".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:465)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:158)
	... 9 more
Caused by: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
	at org.neo4j.kernel.availability.DatabaseAvailabilityGuard.assertDatabaseAvailable(DatabaseAvailabilityGuard.java:172)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginKernelTransaction(GraphDatabaseFacade.java:185)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:165)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:119)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:113)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:101)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:96)
	at org.neo4j.dbms.database.DefaultSystemGraphInitializer.isSystemGraphEmpty(DefaultSystemGraphInitializer.java:100)
	at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:76)
	at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:69)
	at org.neo4j.dbms.database.SystemGraphInitializer.start(SystemGraphInitializer.java:30)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
	... 11 more
Caused by: java.lang.RuntimeException: Mismatching store id. Store StoreId: StoreId{creationTime=1586155486029, randomId=8142000683778803301, storeVersion=3471765337752883975, upgradeTime=1586155486029, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586155296154, randomId=-806197780697231748, storeVersion=3471765337752883975, upgradeTime=1586155296154, upgradeTxId=1}
	at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:386)
	at org.neo4j.kernel.database.Database.start(Database.java:377)
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:132)
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:36)
	at org.neo4j.dbms.database.AbstractDatabaseManager.forEachDatabase(AbstractDatabaseManager.java:165)
	at org.neo4j.dbms.database.AbstractDatabaseManager.startAllDatabases(AbstractDatabaseManager.java:112)
	at org.neo4j.dbms.database.AbstractDatabaseManager.start(AbstractDatabaseManager.java:106)
	... 12 more
2020-04-06 08:25:05.431+0000 INFO  Neo4j Server shutdown initiated by request

[andrew_bowman]

Is this a cluster deployment, or single instance?

If this is a cluster, then we'd recommend running neo4j-admin unbind on all cluster instances, and make sure the same data is seeded on all members of the cluster.

I am a bit concerned with this part:

I've effectively copied the contents of 3.5.14 data folder to data foldedr of 4.0.3.

We don't recommend file copying for this, you should instead use neo4j-admin backup/restore or neo4j-admin dump/load.

[malik]

Good day Andrew,

Thanks for reaching back. As far as your comments are concerned:

1. It's not a cluster. It's a single-instance community edition 4.0.3 installed through apt-get.
2. I was worried about moving folders instead of using neo4j-admin load (as I used to do to migrate from 3.x to 3.x) but that's what was advised in the migration video from my initial post. I figured it was necessary to do it this way because of database structure and graph.db illegal folder name. Is there a better way of migrating 3.5.14 store to 4.0.3?

[malik]

Friends, any ideas as to how to migrate from 3.5.14 to 4.0.3 other than the way it was shown in this video?

So far I'm still stuck with the 'Mismatching store id' error.

Thank you in advance.

[wdrosko]

If you follow the migration guide step by step you should be fine. https://neo4j.com/docs/migration-guide/current/

I ran into the mismatching IDs after purging the migrated data and reloading it. Currently running 3.5 until there is a clear answer.

[malik]

Well, it still suggests I copy/paste data folders but apparently it's not recommended. I'm really confused.

---

1. Copy the files used for encryption from the old installation to the new one.
2. Copy the data directory from the old installation to the new one. This step is not applicable if you have dbms.directories.data pointing to a directory outside of NEO4J_HOME .

[dazhi_jiao · Accepted solution]

@wdrosko Is the data directory for your instance in a symlinked folder? What I found is that when the data directory is a symlink, I will get the "Mismatching store id" error.

[wdrosko]

dazhi.jiao:

@wdrosko Is the data directory for your instance in a symlinked folder? What I found is that when the data directory is a symlink, I will get the "Mismatching store id" error.

This was precisely the issue -- just went back and tried to upgrade again (finally). So very strange. thanks for the info

[chenziyang]

There is no file name with symlinked in my import directory. Only CSV files have connectors, such as administrative_ area. CSV, but my error log is the same as that of the landlord. Does the file name with symlinked have any effect?

[malik]

Friends,

I'm still running into these issues when trying to upgrade to 4.0.5. It, previously, already failed with 4.0.3 where this issue was presumably fixed:

Caused by: java.lang.RuntimeException: Mismatching store id. Store StoreId: StoreId{creationTime=1592553480206, randomId=6493240928352535721, storeVersion=3471765337752883975, upgradeTime=1592553480206, upgradeTxId=1}. Transaction log StoreId: StoreId{creationTime=1586155296154, randomId=-806197780697231748, storeVersion=3471765337752883975, upgradeTime=1586155296154, upgradeTxId=1}
	at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:388)
	at org.neo4j.kernel.database.Database.start(Database.java:377)
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:132)
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:36)
	at org.neo4j.dbms.database.AbstractDatabaseManager.forEachDatabase(AbstractDatabaseManager.java:165)
	at org.neo4j.dbms.database.AbstractDatabaseManager.startAllDatabases(AbstractDatabaseManager.java:112)
	at org.neo4j.dbms.database.AbstractDatabaseManager.start(AbstractDatabaseManager.java:106)

Any clues? I'd like to migrate once and for all but it proves to be impossible.
Thank you in advance.