Can't map postgres to Neo4j graph using Neo4j ETL tool?

[steves]

Hey there 🙂

I am trying to map my Cloudquery database (postgres) to Neo4j using ETL Tool.

Project created with the following version:

Version

4.4.5

Neoj4 Desktop version 1.4.15.

I am on OSX Monterey 12.5 (21G72), Apple M1 Pro.

For the postgres (where cloudquery is pouring the data) I am using postgres:13.7-alpine3.16 docker image.

Selected RDBMS instance (connection succeeded) and my project. Within the main.log I get:

[2022-07-28 14:27:13.192] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.rdbms.Support, jdbc:postgresql://localhost:5432/postgres?ssl=false, postgres, pass'
[2022-07-28 14:27:13.844] [info] Process [87889] exit with code '1', signal 'null'
[2022-07-28 14:27:43.264] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:27:43.372] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:28:23.273] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:28:23.382] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:03.278] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:29:03.386] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:39.909] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.rdbms.Support, jdbc:postgresql://localhost:5432/postgres?ssl=false, postgres, pass'
[2022-07-28 14:29:40.771] [info] Process [88036] exit with code '0', signal 'null'
[2022-07-28 14:29:43.282] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:29:43.338] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:44.217] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.NeoIntegrationCli, generate-metadata-mapping, --rdbms:url, jdbc:postgresql://localhost:5432/postgres?ssl=false, --rdbms:password, pass, --rdbms:user, postgres, --output-mapping-file, /var/folders/nl/301vjtr53s92b4wr66tchrjr0000gn/T/postgresql_postgres_mapping.json'
[2022-07-28 14:30:04.214] [info] Process [88039] exit with code '0', signal 'null'
[2022-07-28 14:30:04.216] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.util.FileUtils, readfile, /var/folders/nl/301vjtr53s92b4wr66tchrjr0000gn/T/postgresql_postgres_mapping.json'
[2022-07-28 14:30:04.616] [info] Process [88058] exit with code '143', signal 'null'
[2022-07-28 14:30:23.288] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:30:23.596] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:31:03.292] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:31:03.344] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:31:43.296] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:31:43.459] [info] Online check response: 200 version: 1.4.15

It seems I do get the mapping file but the mapping process takes ages without finishing.

Please advise what should I do to make it work?

Can I upload manually the mapping file? How can I force it to do the mapping? Is it ok it takes so long?

[david_allen]

@michael_hunger if you have a chance to look at this when you're back please do; I spoke with @steves by discord.  Doesn't seem to be a clear error here so some debugging questions to gather more information might be a way to proceed.

[michael_hunger]

It is an issue with the posgres JDBC driver that the fetching the metadata for the mapping takes so long, it's not the case for other databases.
It's a know issue that they don't want to fix. 

So you'd need to have patience unfortunately. It might be faster if you have a local copy of the database and fetch the mapping from there.

You cannot upload the mapping file directly, but if you use the command line version of the etl-tool, you can pass in the mapping file as a parameter:

https://neo4j.com/labs/etl-tool/1.5.0/#_export_command

https://neo4j.com/labs/etl-tool/1.5.0/neo4j-etl/#_online_batch_import_via_java_bolt_driver_for_incre...

[steves]

Thanks a lot @michael_hunger !

Would you be so kind and post an A-Z tutorial or how-to to convert RDBMS to Neo4j graph? Maybe using Hop?

I will be happy to see an example. If it's possible it will help me and others.