Migrate data from 3.5.1 Community to 4.0.0 Community

[keithave]

Howdy,

Not sure if this is the right place to post or if this has been answered elsewhere, please point me in the right direction.

I've been using Neo4j 3.5.1 Community and would like to migrate my data into Neo4j Community 4.0.0.
If there is clear documentation for migrating from 3.5 to 4.0 Community, please point me in that direction.

Hurdles I've cleared thus far...

1. updated java jdk from 1.8 to jdk 11 (and edited env var)
2. downloaded community-4.0.0
3. edited neo4j.conf to...

# Enable this to be able to upgrade a store from an older version.
dbms.allow_upgrade=true

4. copied 3.5.1 graph.db file into C:\Users\owner\Documents\neo4j-community-400\data\databases
5. ran "neo4j console" from command prompt and received the following in cmd line...

C:\Users\owner\Documents\neo4j-community-400\bin>neo4j console
2020-02-07 21:31:04.218+0000 INFO ======== Neo4j 4.0.0 ======== 2020-02-07 21:31:04.225+0000 INFO Starting... SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 2020-02-07 21:31:14.032+0000 INFO Bolt enabled on localhost:7687. 2020-02-07 21:31:14.034+0000 INFO Started. 2020-02-07 21:31:14.963+0000 INFO Remote interface available at http://localhost:7474/

Roadblocks I'm encountering...
So 4.0 'appears' to be running (despite whatever that is about SLF4J), but when I go to http://localhost:7474/browser/, get past all the new login info I do see that the browser recognizes graph.db - default as the db to use, BUT none of my Node Labels, Relationship Types, or Property Keys show in browser AND if I run a simple query (and even if I run the query without the :Beer label) against this graph.db I get the following...

Database 'graph.db' is unavailable.

What am I missing? What else do I need to do to migrate my old data into Community-4.0.0?

[david_allen]

Here is the documentation:

https://neo4j.com/docs/migration-guide/4.0/upgrade-single-instance/

From a brief scan of what you've done, a thing you may be missing is setting dbms.default_database as described in step 1.a.ii "Prepare to upgrade"

[keithave]

Thanks for your quick response David.
Ok, so I've edited the neo4j.conf file a couple times now with the following...

# The name of the default database
dbms.default_database=graph.db

editing, saving, restarting neo4j did not work.

Then I tried giving a direct path to the file...

# Paths of directories in the installation.
dbms.directories.data=C:\Users\owner\Documents\neo4j-community-400\data\databases\graph.db

Neither of these seem to work, either together or independent of each other.

For what it's worth, my log file shares this re: migration...

2020-02-07 21:30:36.183+0000 INFO [o.n.k.i.s.DatabaseMigrator] [graph.db] Starting upgrade of database
2020-02-07 21:30:36.245+0000 INFO [o.n.k.i.s.DatabaseMigrator] [graph.db] Migrating Store files (1/6):
2020-02-07 21:30:36.572+0000 WARN [o.n.k.d.Database] [graph.db] Exception occurred while starting the database. Trying to stop already started components. Failure doing migration
org.neo4j.kernel.impl.storemigration.StoreUpgrader$UnableToUpgradeException: Failure doing migration

So it's not gotten past 1 of 6.

[david_allen]

graph.db is not a valid database name in neo4j 4.0. That's a folder name in 3.5. Version 3.5 didn't have the concept of named graphs, so basically that was just a directory name.

https://neo4j.com/docs/operations-manual/current/manage-databases/configuration/

Try moving it to "graph" or the default which comes with Neo4j 4.0, which is "neo4j"

Naming rules for databases are as follows:

• Length must be between 3 and 63 characters.
• The first character of a name must be an ASCII alphabetic character.
• Subsequent characters must be ASCII alphabetic or numeric characters, dots or dashes; [a..z][0..9].-
• Names are case-insensitive and normalized to lowercase.
• Names that begin with an underscore and with the prefix system are reserved for internal use.

[keithave]

Thanks for your patience David,

Here's what I've done but does not work. Btw, it's unclear which 'data' dir I'm to drop folders in 'data' or 'databases'. Here's what I have...

I took the 3.5 graph.db and renamed it neo4j. I dropped this folder here...
C:\Users\owner\Documents\neo4j-community-400\data\databases\

I edited the neo4j.conf file to read...

# Paths of directories in the installation.
dbms.directories.data=C:/Users/owner/Documents/neo4j-community-400/data/databases/neo4j

(please note I've changed the direction of the slash / to appease the conf file)

neo4j console starts, I get this...

2020-02-07 23:00:29.409+0000 INFO  ======== Neo4j 4.0.0 ========
2020-02-07 23:00:29.415+0000 INFO  Starting...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2020-02-07 23:00:38.745+0000 INFO  Completed migration of 1 user into system graph.
2020-02-07 23:00:40.274+0000 INFO  Called db.clearQueryCaches(): Query cache already empty.
2020-02-07 23:00:40.656+0000 INFO  Bolt enabled on localhost:7687.
2020-02-07 23:00:40.657+0000 INFO  Started.
2020-02-07 23:00:41.399+0000 INFO  Remote interface available at http://localhost:7474/

but when I open this in the browser, I see the neo4j - default, but nothing is in it, though I can now run a query without the aforementioned error.

[david_allen]

I think you've got your data directory 2 levels deeper (data/databases) than it should be. You're pointing your data directory straight to your database folder, but that isn't right because the folder you're pointing to would end up containing many databases (since 4.0 can support many databases).

To see the correct structure, a thing worth trying would be to start 4.0 completely empty without your migration dataset, from a new default root. In this case, you'll end up with an empty neo4j database created for you. By seeing where that's placed on disk, you can know to put your older graph folder under a different name at the same relative path.

[keithave]

Thanks so much for your patience David, I'm sorry to put you through this, but there seem to be others with a somewhat similar issue. Granted 2. and 3. seem to be able to point to the old data, they just can't get the migrating to complete all the 1-6 steps.

1. Unable to Upgrade to v4.0.0 from v3.0.1 (Community Edition)
2. Migration to 4.0 stops
3. Migrating Store files (to 4.0)
   ...
   Hmm, I see what you're saying about the '2 level deeper' but it seems that no matter where I start with my data dir, these other neo4j & systems dirs get build regardless.
   ...

How bout this, I've gone back to the beginning, Chapter 4. Upgrade a single instance, and following the Upgrade directions directly, I get the following result…

Upgrade

1. Install Neo4j 4.0.0 using one of the following methods, specific to your technology:
   a. If using a tarball or zipfile for installation:
   i. Untar or unzip Neo4j 4.0.0
   ii. Transfer the new neo4j.conf that you prepared in the Apply configuration changes step in Chapter 2, Prepare to upgrade.
   iii. Set dbms.allow_upgrade=true in neo4j.conf of the 4.0.0 installation. Neo4j will fail to start without this configuration.
   iv. Copy the files used for encryption from the old installation to the new one.
   v. Copy the data directory from the old installation to the new one. This step is not applicable if you have dbms.directories.data pointing to a directory outside of NEO4J_HOME .

I have done all the above, and for 1.a.v., I am not pointing to a directory outside of NEO4J_HOME . I have copied my 3.5 data directory into the neo4j-community-400 folder, my home dir now has this structure…

bin
certificates
conf
data
........|_databases
.........................|_graph (I have removed .db from the file name and dir contains many neostore.* files and dirs like certificates, data, dbms, import, index, logs, plugins, schema)
.........................|_store_lock
........|_dbms
........|_auth
import
lib
logs
plugins
run

Without altering anything in neo4j.conf, other than 1.a.iii., I run cmd line neo4j console and now my data dir has been given a transactions dir, and more importantly, my data > databases dir (which only contained my graph dir) has been given a neo4j & systems dir. I would not expect this to happen since I replaced the 4.0 data dir with my 3.5 data dir.

When I open localhost:7474/browser, I see the neo4j and systems dbs, but not my graph db…

If there’s something from the above directions I’m doing incorrectly, please let me know, otherwise I am at a loss and it would seem that the directions above need to be updated because they are lacking a piece(s) of this puzzle.

[keithave]

Perhaps a How-to video on YouTube?

[david_allen]

YouTube video on migrating 3.5 -> 4.0!

[keithave]

Holy Smokes! David, you rock!
Ok here's my neo4j.conf file... neo4j.txt (15.8 KB)
As you can see, it's mostly unedited, only having un-commented the section allow_upgrade=true.
I'll take a look at your video and let you know how it goes.

[david_allen]

Your issue is this line:

#dbms.default_database=neo4j

Uncomment it, and change it to "graph" which is what you named your folder. Neo4j is starting up with the "default default" which is neo4j, which doesn't exist, so you get a blank database.

[keithave · Accepted solution]

Thanks David, Sorry I attached a version of the neo4j.conf file that had only had the Upgrade line un-commented. I did uncomment this in subsequent endeavors.

Ok, tears of joy here, but not without more trial and tribulation, such is life. Here's the story...

I set up everything as you described in your video, but ran into this error (first 1/2 of pic below) when I tried to start with neo4j console...

Solution

stackoverflow.com

Unable to start server [3.5.3]

neo4j

asked by Valentine on 09:32AM - 20 Feb 19 UTC

This PowerShell? issue is likely unrelated to my Migration issue.
...
...

After employing the fix from stackoverflow, I ran neo4j console again, but saw no Migration Steps 1-6. So I turned to the debug.log file (which proved quite helpful).

In the log file I received this warning...

2020-02-11 21:58:05.558+0000 WARN [o.n.k.d.Database] [graph] Exception occurred while starting the database. Trying to stop already started components. The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.

org.neo4j.kernel.impl.storemigration.StoreUpgrader$DatabaseNotCleanlyShutDownException: The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.

So I Googled neo4j clean shutdowns and came across this info...

Solution

stackoverflow.com

How to Cleanly shutdown database in neo4j

neo4j

asked by Rosti on 04:10AM - 30 Aug 17 UTC

I did as the solution suggested and deleted 2 neostore.transaction.db.* files.
For the record, I had been using a database that runs on my local, hence the 'dirty' shutdown (since there doesn't seem to be a way to run neo4j stop when running neo4j console) So I ran neo4j stop against a test database thinking these neostore.transaction files would be gone. They were not, so still left wondering what a cleanshut down is and how to avoid this pitfall in the future.
...
...
I ran neo4j console again and received Migration 1-6 Joy! as seen here...

However, my labels and relationships were still not showing in localhost:7474/browser. So I turned to the log file again and found another warning that suggested I add a line to my conf file...

2020-02-11 22:11:27.552+0000 WARN [o.n.k.d.Database] [graph] Exception occurred while starting the database. Trying to stop already started components. Component 'org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck@7f12094d' failed to initialize. Please see the attached cause exception "Transaction logs are missing and recovery is not possible. To force the database to start anyway, you can specify 'dbms.recovery.fail_on_missing_files=false'. This will create new transaction log and will update database metadata accordingly. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.".
org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck@7f12094d' failed to initialize. Please see the attached cause exception "Transaction logs are missing and recovery is not possible. To force the database to start anyway, you can specify 'dbms.recovery.fail_on_missing_files=false'. This will create new transaction log and will update database metadata accordingly. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.".

I added dbms.recovery.fail_on_missing_files=false to the conf file, ran neo4j console, and my database migration APPEARS to be successful, meaning I can run cypher queries and see labels, relationships, and properties from my 3.5 data in the new neo4j 4.0.
...
...
That said, I'm hoping maybe you can confirm that what appears to be a successful migration actually is? Thank you so much for sticking with me and for providing the very helpful video. I seemed to have found other 'anomalies' that couldn't possibly be accounted for.

Many Thanks!

Keith

[oleg_neo4j]

Hi David,
I had an issue migrating from 3.5 to 4.0 and this thread helped me, but it was the video that did it. My old database folder had a name like: xxxx.graphdb and the period was causing problems. After I pointed to my old database(with the period in the folder name) in the .conf file, it went through the six step process of upgrading successfully, but then in the browser none of my nodes were visible, and I couldn't figure out why.... until I saw the video and took the period out
The period is still mentioned here as part of a valid db name: https://neo4j.com/docs/operations-manual/current/manage-databases/configuration/
and there's nothing about a valid name mentioned in the upgrade instructions of the Operations Manual(https://neo4j.com/docs/operations-manual/current/upgrade/deployment-upgrading/). Could information about what makes a valid name be added/fixed in both locations? Thanks a lot for the video! 🙂

[faaizshah]

Thanks for the video!

[kiran_ghanwat]

Hi David,

Thanks for Video.
By referring this Video I followed steps to migrate from 3.5.22 to 4.0.2 version.

But I am getting below error. Please help me in this.

[david_allen]

Hi @kiran.ghanwat the error is clearly indicated in your screenshot. The trouble is that you must use Java 11 to run Neo4j 4, and your JDK here is too old and not supported for Neo4j 4 (you are using java version 1.7.0_76)

Upgrade your JVM and you should be in good shape.

[kiran_ghanwat]

Thanks David.

I fixed issue related to JVM. I am using Neo4j desktop, within desktop, for practice purpose I created V3.5.22, added few nodes. Then I created another DB with V 4.0.2.

As per your video I copied old files to new DB and renamed to neo4j. Updated conf file as well.

while starting console I am getting below errors:

-===== in security log ====
2021-05-17 13:35:21.976+0000 INFO {plugin-com.neo4j.plugin.jwt.auth.JwtAuthPlugin} Log in attempted for user 'neo4j'.
2021-05-17 13:35:21.977+0000 ERROR {plugin-com.neo4j.plugin.jwt.auth.JwtAuthPlugin} Invalid serialized unsecured/JWS/JWE object: Missing part delimiters
2021-05-17 13:35:21.981+0000 INFO [neo4j]: logged in

===== debug.log ======
Database neo4j is requested to transition from EnterpriseDatabaseState{databaseId=DatabaseId{f682c3c4[neo4j] }, operatorState=INITIAL, failed=false} to EnterpriseDatabaseState{databaseId=DatabaseId{f682c3c4[neo4j] }, operatorState=STARTED, failed=false}
2021-05-17 13:24:08.201+0000 INFO [c.n.d.d.EnterpriseMultiDatabaseManager] Creating 'neo4j' database.
2021-05-17 13:24:08.203+0000 INFO [o.n.k.i.f.StatementLocksFactorySelector] No services implementing StatementLocksFactory found. Using SimpleStatementLocksFactory
2021-05-17 13:24:08.207+0000 INFO [c.n.d.d.EnterpriseMultiDatabaseManager] Starting 'neo4j' database.
2021-05-17 13:24:08.231+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [neo4j] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store C:\Users\w49575.Neo4jDesktop\relate-data\dbmss\dbms-74e1167c-8f64-4e8c-a92d-fe84891194d8\data\databases\neo4j
2021-05-17 13:24:08.233+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [neo4j] Selected format 'RecordFormat:StandardV4_0[SF4.0.0]' for existing store C:\Users\w49575.Neo4jDesktop\relate-data\dbmss\dbms-74e1167c-8f64-4e8c-a92d-fe84891194d8\data\databases\neo4j with format 'RecordFormat:StandardV3_4[v0.A.9]'
2021-05-17 13:24:08.291+0000 INFO [o.n.k.i.s.DatabaseMigrator] [neo4j] Starting upgrade of database
2021-05-17 13:24:08.319+0000 WARN [o.n.k.d.Database] [neo4j] Exception occurred while starting the database. Trying to stop already started components. The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
org.neo4j.kernel.impl.storemigration.StoreUpgrader$DatabaseNotCleanlyShutDownException: The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.assertCleanlyShutDownByCheckPoint(StoreUpgrader.java:287)
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrate(StoreUpgrader.java:195)
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:146)

==== in browser =====

Please help me in this upgrade issue.

[kiran_ghanwat]

One more query @david.allen , this Video and procedure mentioned is applicable for community only or enterprise edition ?

[david_allen]

@keithave the instructions you've got make it look like you're doing everything right -- except I can't see your config and that's probably where it's not working. In particular -- if neo4j doesn't know to look for "graph" where you placed it, then what you'll get is exactly what you see -- it will create a default "neo4j" database instead.

Post the snippet of your neo4j.conf file and I'll try to have a look. The YouTube video is a great idea, I'll look into it.

[gagan_suri]

@david.allen I have a custom procedure written in Java which uses org.neo4j.kernel.internal.GraphDatabaseAPI especially findNodes method from 3.5. But looks like in 4.0 this GraphDatabaseAPI doesn't have findNodes method. Which class and which method should I be using for custom procedures in Neo4j 4.x. Thank you,Gagan