Neo4J Error while importing example database Musicbrainz

[wunderbardan]

I try to import Musicbrainz database from the example datasets https://neo4j.com/developer/example-data/. I downloaded the 4.5GB big file archive and dezipped the whole database to a new graph.db folder. When I now try to start the database after sucessfully opened it I get the following error:

java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /home/neo4j/data/databases/graph.db
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:212)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:125)
	at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:137)
	at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:130)
	at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:107)
	at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:199)
	at org.neo4j.shell.kernel.GraphDatabaseShellServer.instantiateGraphDb(GraphDatabaseShellServer.java:228)
	at org.neo4j.shell.kernel.GraphDatabaseShellServer.<init>(GraphDatabaseShellServer.java:75)
	at org.neo4j.shell.StartClient.getGraphDatabaseShellServer(StartClient.java:311)
	at org.neo4j.shell.StartClient.tryStartLocalServerAndClient(StartClient.java:294)
	at org.neo4j.shell.StartClient.startLocal(StartClient.java:282)
	at org.neo4j.shell.StartClient.start(StartClient.java:213)
	at org.neo4j.shell.StartClient.main(StartClient.java:147)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.index.labelscan.NativeLabelScanStore@549949be' was successfully initialized, but failed to start. Please see the attached cause exception "The node label update contained unsorted label ids [17, 9]".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)
	at org.neo4j.kernel.lifecycle.LifeSupport.bringToState(LifeSupport.java:315)
	at org.neo4j.kernel.lifecycle.LifeSupport.add(LifeSupport.java:248)
	at org.neo4j.kernel.lifecycle.Lifespan.add(Lifespan.java:41)
	at org.neo4j.kernel.impl.storemigration.participant.NativeLabelScanStoreMigrator.migrate(NativeLabelScanStoreMigrator.java:83)
	at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateToIsolatedDirectory(StoreUpgrader.java:224)
	at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateStore(StoreUpgrader.java:144)
	at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:122)
	at org.neo4j.kernel.impl.storemigration.DatabaseMigrator.migrate(DatabaseMigrator.java:100)
	at org.neo4j.kernel.NeoStoreDataSource.upgradeStore(NeoStoreDataSource.java:564)
	at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:419)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:208)
	... 12 more
Caused by: java.lang.IllegalArgumentException: The node label update contained unsorted label ids [17, 9]
	at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.extractChange(NativeLabelScanWriter.java:210)
	at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.flushPendingChanges(NativeLabelScanWriter.java:175)
	at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.write(NativeLabelScanWriter.java:144)
	at org.neo4j.kernel.impl.api.scan.FullLabelStream.visit(FullLabelStream.java:61)
	at org.neo4j.kernel.impl.storemigration.participant.NativeLabelScanStoreMigrator$MonitoredFullLabelStream.visit(NativeLabelScanStoreMigrator.java:178)
	at org.neo4j.kernel.impl.storemigration.participant.NativeLabelScanStoreMigrator$MonitoredFullLabelStream.visit(NativeLabelScanStoreMigrator.java:164)
	at org.neo4j.kernel.impl.transaction.state.storeview.StoreViewNodeStoreScan.process(StoreViewNodeStoreScan.java:90)
	at org.neo4j.kernel.impl.transaction.state.storeview.NodeStoreScan.run(NodeStoreScan.java:74)
	at org.neo4j.kernel.impl.api.scan.FullLabelStream.applyTo(FullLabelStream.java:54)
	at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanStore.start(NativeLabelScanStore.java:420)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
	... 28 more
	Suppressed: java.lang.IllegalArgumentException: The node label update contained unsorted label ids [17, 9]
		at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.extractChange(NativeLabelScanWriter.java:210)
		at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.flushPendingChanges(NativeLabelScanWriter.java:175)
		at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.close(NativeLabelScanWriter.java:270)
		at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanStore.start(NativeLabelScanStore.java:421)
		... 29 more

My Neo4j version is 3.4.5.
Thank you!

[michael_hunger]

That dataset is for an older version, I'll upload one for 3.4.x.

Here you go: example-data.neo4j.org/3.4-datasets/musicbrainz.tgz

[wunderbardan]

The new dataset works. Thank you. And I execute call apoc.meta.graph(). And now I have waited for 443 minutes, the procedure is still running. My PC has 8G RAM and core i7. I am not sure how long should I wait. Is there any accelerating tool to execute apoc procedure for large datasets?
Thank you!

[michael_hunger]

It should finish in seconds. Can you try call db.schema() meanwhile?

[wunderbardan]

I observed my memory. For call apoc.meta.graph(), memory would be increased up to 50%, cpu would be occupied over 100%.

By call db.schema() the graph can be illustrated after seconds and wrote that there are over 13000 relationships. The graph would be incrementally updated. The cpu would be ocupied from over 100% to over 400%. And it seems like another endless procedure.

By call apoc.meta.data() the process finished in 3 minutes and displayed first 1000 rows. When processed: cpu: 10%~20%, memory: over 40%. After displaying the result, cpu would be relatively low and memory is constant 40.2%. After exporting the data file into csv, I read there are over 2000 rows.

Maybe the illustration over 10000 relationships is too difficult. Now I want to know the principle of sampling datasets. And it would be beneficial for my research.
Thank you!

[nii_akrong]

Hello Michael,
I'm getting a similar error trying to use a graph.db folder from neo4j 2.3.9 in a 3.4.x instance. I shut down the database cleanly before i copied the directory. I also set the allow upgrade flag in the neo4j.conf file in the 3.4 db to true. Any suggestions please?

[baik_narrativeg]

3.4 dataset works for me too. I checked the number of Artist nodes and it is about 790,000. If I check the musicbrainz stats site, the number of Artists is 1440,000 as of January 2019. Is it because the 3.4 dataset was a copy of old musicbrainz? If yes, how can I generate the copy of graph db of the latest musicbrainz? Thanks much in advance.

[michael_hunger]

Yes it's from an older copy. The process to import it was documented here:

Neo4j Graph Database Platform – 5 Nov 13

Musicbrainz in Neo4j - Part 1 - Neo4j Graph Database Platform

MusicBrainz is an “open content music database that captures information about artists. It was founded in response to the restrictions placed on the CDDB.

[baik_narrativeg]

Thank you for the link. I followed all steps, but the final neo4j database does not load properly onto neo4j 3.4.9 version. All info on the link assume neo4j 2.0. Do I have to do something differently to work for neo4j latest version?

[baik_narrativeg]

Hi Michael,
Thank you. The batch importer in the link should work with neo4j 3.4.x too? The link shows the steps how to dump into neo4j 2.0, and I tried with neo4j 3.4.9, it didn't work. When starting service neo4j, the db was not loaded properly and bolt was not enabled. I don't know it should work with the latest version of neo4j or some steps should be adjusted. Thank you in advance!

[michael_hunger]

Can you share the exact error / your debug.log?

[nii_akrong]

Sure thing Michael.

ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@5b970f7' was successfully initialized, but failed to start. Please see the attached cause exception "The node label update contained unsorted label ids [3, 2]". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@5b970f7' was successfully initialized, but failed to start. Please see the attached cause exception "The node label update contained unsorted label ids [3, 2]".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@5b970f7' was successfully initialized, but failed to start. Please see the attached cause exception "The node label update contained unsorted label ids [3, 2]".
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@5b970f7' was successfully initialized, but failed to start. Please see the attached cause exception "The node label update contained unsorted label ids [3, 2]".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory, /opt/neo4j-community-3.5.11/data/databases
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:232)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:148)
at org.neo4j.server.database.CommunityGraphFactory.newGraphDatabase(CommunityGraphFactory.java:41)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:90)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.index.labelscan.NativeLabelScanStore@36b0fcd5' was successfully initialized, but failed to start. Please see the attached cause exception "The node label update contained unsorted label ids [3, 2]".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
at org.neo4j.kernel.lifecycle.LifeSupport.bringToState(LifeSupport.java:322)
at org.neo4j.kernel.lifecycle.LifeSupport.addNewComponent(LifeSupport.java:238)
at org.neo4j.kernel.lifecycle.LifeSupport.add(LifeSupport.java:214)
at org.neo4j.kernel.lifecycle.Lifespan.add(Lifespan.java:41)
at org.neo4j.kernel.impl.storemigration.participant.NativeLabelScanStoreMigrator.migrate(NativeLabelScanStoreMigrator.java:84)
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateToIsolatedDirectory(StoreUpgrader.java:223)
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateStore(StoreUpgrader.java:144)
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:122)
at org.neo4j.kernel.impl.storemigration.DatabaseMigrator.migrate(DatabaseMigrator.java:101)
at org.neo4j.kernel.NeoStoreDataSource.upgradeStore(NeoStoreDataSource.java:505)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:353)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:116)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:227)
... 9 more
Caused by: java.lang.IllegalArgumentException: The node label update contained unsorted label ids [3, 2]
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.extractChange(NativeLabelScanWriter.java:264)
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.flushPendingChanges(NativeLabelScanWriter.java:229)
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.write(NativeLabelScanWriter.java:198)
at org.neo4j.kernel.impl.api.scan.FullLabelStream.visit(FullLabelStream.java:61)
at org.neo4j.kernel.impl.storemigration.participant.NativeLabelScanStoreMigrator$MonitoredFullLabelStream.visit(NativeLabelScanStoreMigrator.java:176)
at org.neo4j.kernel.impl.storemigration.participant.NativeLabelScanStoreMigrator$MonitoredFullLabelStream.visit(NativeLabelScanStoreMigrator.java:162)
at org.neo4j.kernel.impl.transaction.state.storeview.StoreViewNodeStoreScan.process(StoreViewNodeStoreScan.java:70)
at org.neo4j.kernel.impl.transaction.state.storeview.StoreViewNodeStoreScan.process(StoreViewNodeStoreScan.java:34)
at org.neo4j.kernel.impl.transaction.state.storeview.PropertyAwareEntityStoreScan.run(PropertyAwareEntityStoreScan.java:117)
at org.neo4j.kernel.impl.api.scan.FullLabelStream.applyTo(FullLabelStream.java:54)
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanStore.start(NativeLabelScanStore.java:437)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 26 more
Suppressed: java.lang.IllegalArgumentException: The node label update contained unsorted label ids [3, 2]
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.extractChange(NativeLabelScanWriter.java:264)
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.flushPendingChanges(NativeLabelScanWriter.java:229)
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanWriter.close(NativeLabelScanWriter.java:334)
at org.neo4j.kernel.impl.index.labelscan.NativeLabelScanStore.start(NativeLabelScanStore.java:438)```

[michael_hunger]

I think for a cross version upgrade you might have to go via some intermediate versions. Like 3.0 and 3.2

you can also try to use neo4j-admin import --mode=database --from=/path/to/graph.db

[nii_akrong]

Thanks for the pointers. I'll try them out and let you know how it turns out.

[nii_akrong]

Thanks again for the help.
Unfortunately, I'm still having issues. Using the admin import tool with 3.0 gave me this error when i started the database neo4j-community-3.0.0/data/databases/graph.db/neostore' has a store version 'v0.A.6' that we cannot upgrade from.

3.2 looked promising because when i started up the database after the import, the logs showed the message "Successfully updated database". But after a while the "The node label update contained unsorted label ids [3, 2]" error was thrown again.

[simon_boucher_p]

HI,
Since I have read the Neo4j and Musicbrainz article from Paul Tremberth, I was trilled to see the results.
Unfortunately, for a non-programer person like me, going through the necessary steps does not mean you get the expected results.
Anybody else can point me to another knowledge graph or associative network of music artist relations such as musicroamer.com?
Thanks!
Simon