Unable to Import Neo4j Community 3.5.1 Database to AuraDB Free Version

[kave]

Greetings,

We are considering moving our Neo4j Community 3.5.1 database from VMs to AuraDB, and in so doing, we have created a db on the AuraDB Free version to test whether we can actually get our current data into AuraDB. At present, we cannot.

I've used the following help page to no avail:

Neo4j Aura

Importing Your Neo4j 3.5.x Database into Aura

With the release of Neo4j 4 on Aura, you may wish to take advantage of the new features using an existing database. If you have already been using 3.5 on Aura, please see this article for the steps...

I have followed these import options detailed here, but neither work:

Option 1) Create a .dump file
Yes I stopped the db, yes I created the requested directory and yes our db meets the free reqs = nodes < 50,000 rels < 175000.
This is the cmd used:

bin\neo4j-admin dump --database=graph.db --to=backups\graph.db'graphdb'.dump

Option 2) push-to-cloud
Yes I stopped the db, yes I have the required jar (neo4j-push-to-cloud-3.5.23.jar) in the lib dir.
This is the cmd used:
bin\neo4j-admin push-to-cloud --bolt-uri neo4j+s://cbbb4918.databases.neo4j.io --database kndb --overwrite. I received the error seen in the attached pic: pushtocloud-errormessage.PNG.

Please let me know if I'm doing something wrong or what more I can provide to explain my processes.

Many Thanks,

Keith

[tony_chiboucas]

Hello kave,

TL;DR: Outdated article. First, upgrade Desktop to 4.1 (it's at least a day's work).

Aparently none of that was true. Updated response below.

[tony_chiboucas]

From Desktop 4.4.x, you can load an old dump, and then copy it to a new empty db, reindex, then push-to-cloud with a command, or push-to-cloud a web walkthrough.

Let me know if this helps:

Neo4j Aura

Push to cloud fails to load older version of Neo4j dump file to AuraDB

Under some circumstances, it is possible that your database Dump produced from some old version of the Neo4j database fails to load successfully due to a defect in the underlying database archive p...

[kave]

Hi Tony, Thank you so much for your willingness to help, however you've invoked Neo4j Desktop, which I absolutely despise with its 'create projects' and other new-to-me-junk.

I just need a soup to nuts solution now that I've got my database from Community 3.5.1 successfully migrated to Enterprise 3.5.30, and I have created an Enterprise 3.5.30 .dump file, and I am trying to load said dump file into Enterprise 4.1.11, but it's not working.

If you have any information for this type of migration (not including desktop), I would be grateful.

[tony_chiboucas]

@kave, those instructions are not unique to Desktop. I'd suggest first attempting the walkthrough here: Neo4j Aura

You can instead do it purely through commands, but you'll probably need to add the push-to-cloud jar (link in that article).

[kave]

Thanks Tony,
All of these commands are assuming I can create a functioning .dump file in Enterprise 4.0.* and for reasons unknown, I cannot.

The following are directions provided to me by Neo4j tech support, but step 3. Copy only yields a size 3KB directory with none of my data in it. So I'm unable to move beyond step 3.

1. [neo4j-enterprise-4.1.11]> bin/neo4j stop

2. [neo4j-enterprise-4.1.11]> bin/neo4j-admin load --from=import/movie35.dump --database=movie35

3. [neo4j-enterprise-4.1.11]> bin/neo4j-admin copy --from-database=movie35 --to-database=bar

4. [neo4j-enterprise-4.1.11]> echo "dbms.allow_upgrade=true" >> conf/neo4j.conf

5. [neo4j-enterprise-4.1.11]> bin/neo4j start

6. // create database bar in neo4j browser create database bar

7. [neo4j-enterprise-4.1.11]> bin/neo4j stop

8. [neo4j-enterprise-4.1.11]> bin/neo4j-admin dump --database=bar --to=import/bar.dump

9. import the bar.dump to AuraDB in your AuraDB console.neo4j.io

Here are the specific cmds I'm using with cmd line results:

C:\neo4j-ent-4111>bin\neo4j-admin load --from=import/graph3530.dump --database=graphdb
Done: 178 files, 369.0MiB processed.

C:\neo4j-ent-4111>bin\neo4j-admin copy --from-database=graphdb --to-database=bar
Failure when checking for recovery state: '%s'.Error reading transaction logs, recovery not possible. To force the database to start anyway, you can specify 'unsupported.dbms.tx_log.fail_on_corrupted_log_files=false'. This will try to recover as much as possible and then truncate the corrupt part of the transaction log. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.

[yes I uncommented dbms.allow_upgrade=true and added the unsupported bit to my neo4j.conf file]

C:\neo4j-ent-4111>bin\neo4j start
Neo4j service started

[when I go to Browser and Create Database bar, NOW there are files in this database bar, but it's just a blank slate, it has none of my data... this bar database is 3KB in size... I should be able to see my data in Browser]

C:\neo4j-ent-4111>bin\neo4j stop
Neo4j service stopped

C:\neo4j-ent-4111>bin\neo4j-admin dump --database=bar --to=backups/bar.dump
Done: 67 files, 800.7KiB processed.

Having done all of the above, the result is a 3KB Enterprise 4.1.11 .dump file with none of my data so there's no reason for me to try to upload it to Auradb.

It would seem the error is in the Copy command not working? Maybe? I don't know and am grateful for anyone who can shed light on this issue.

[kave · Accepted solution]

I was finally able to migrate 3.5.1 to AuraDB Free version, but not by the help of any AuraDB Support professionals.



I am thoroughly disappointed in the fact that 3, yes 3 separate AuraDB Support professionals said the steps above would work. They do not. The short of the long is instead of LOADING the .dump file into 4., one needs to BACKUP the 3.5. database and COPY that into 4.*. This is the needle in the haystack that eventually helped: Tutorial: Back up and copy a database in a standalone instance - Upgrade and Migration Guide


Here are the steps for migrating Neo4j Community 3.5.1 to Enterprise 4.0.12 to Free AuraDB 4.4.3:

Download Neo4j Enterprise 3.5.30

Create Clean Start Stop of Community 3.5.1

cmd> cd C:\neo4j-community-351
cmd> bin\neo4j install-service
cmd> bin\neo4j start (neo4j start and needs a few min before viewable in browser)
[confirmed data viewable in browser]
cmd> bin\neo4j stop

Create .dump File in Community 3.5.1

C:\neo4j-community-351> bin\neo4j-admin dump --database=graph.db --to=C:\neo4j-community-351\backups\graph351.dump
cmd> bin\neo4j uninstall-service
[helps to use the neo4j service of each respective server]

Load 3.5.1 .dump into Neo4j Enterprise 3.5.30

change neo4j.conf to allow upgrade
open new cmd prompt
cmd path to ent 3.5.30 on my local
C:\neo4j-ent-3530> bin\neo4j-admin load --from=import\graph351.dump --database=graphdb
changed db title to graph.db
cmd> bin\neo4j install-service
cmd> bin\neo4j start (give a few mins before browser)

For Free AuraDB

DELETE nodes for < 50,000
Match (n:Label) Detach Delete (n)
Call db.indexes()
DROP INDEX ON :Label (Property)
DROP CONSTRAINT ON (n:Label) ASSERT n.Property IS UNIQUE
Don’t Stop the Server!! Keep it running for the neo4j-admin backup cmd…

Back up Enterprise 3.5.30

Back Up Tutorial 3.5 to 4.X [this provided the necessary cmds]

Neo4j Graph Database Platform

Tutorial: Back up and copy a database in a standalone instance - Upgrade and...

This tutorial provides a detailed example of how to back up a 3.5 database and use the `neo4j-admin copy` command to migrate it to a 4.x Neo4j standalone instance.

cmd> bin\neo4j-admin backup --backup-dir C:\neo4j-ent-3530\backups --name=graph3530backup
cmd> bin\neo4j stop
cmd> bin\neo4j uninstall-service

Download Neo4j Enterprise 4.0.12

Not sure if it’s required, but I’m using PowerShell as admin in Enterprise 4.X servers.
Need to change Environmental Variable to Java 11
PS C:\WINDOWS\system32> cd C:\neo4j-ent-4012
PS C:\neo4j-ent-4012> bin\neo4j install-service

Copy Database in Neo4j Enterprise 4.0.12

PS C:\neo4j-ent-4012> bin\neo4j-admin copy --from-path=C:\neo4j-ent-3530\backups\graph3530backup --to-database=graphdb

In Browser, Create New Database

It’s not enough just to copy data into the databases directory, you have to CREATE the database in Neo4j Browser
PS C:\neo4j-ent-4012> bin\neo4j start
Browser: system$ CREATE DATABASE graphdb;
system$ Show Databases;
system$ :use graphdb [NO SEMI-COLON!]

Create .dump File for AuraDB

PS C:\neo4j-ent-4012> bin\neo4j stop
PS C:\neo4j-ent-4012> bin\neo4j-admin dump --database=graphdb --to=backups\graphdb.dump

Import .dump File into Free AuraDB

After you’ve created your free AuraDB, select the db title then select Import Database, import the .dump file.