Node import got stuck while migrating mariaDB objects to neo4j

[phalvi]

While importing CSV data to neo4j, it got stuck in the node degrees. Waited for more than 2 hrs but no luck. Please find the below content which was printed while running the
MigrateMariaObjectsToNeo4j.py. No errors in neo4j.log also. Any debugging steps/alternative approach to solve this issue should be appriciated.

Os : centos 7.7
Neo4J versison: 3.2.12

Available resources:
Total machine memory: 23.39 GB
Free machine memory: 9.83 GB
Max heap memory : 5.65 GB
Processors: 1
Configured max memory: 3.76 GB

Nodes, started 2020-01-29 06:29:37.813+0000
[>:6.97 MB/s----------|NODE:7.|PROPERTIES------------------------------------|LA|v:8.07 MB/s-] 196K ∆10.9K
Done in 17s 340ms
Prepare node index, started 2020-01-29 06:29:55.312+0000
[DETECT:8.37 MB------------------------------------------------------------------------------] 200K ∆ 200K
Done in 688ms
Relationships, started 2020-01-29 06:29:56.159+0000
[>:22.69 MB/s------|TY|PREPARE------------------------------------------------------------|||] 191K ∆81.9K
Done in 3s 981ms
Node Degrees, started 2020-01-29 06:30:00.329+0000
**[>:904.00 B/s--------------------------------------------------------------------------------] 0 ∆ 0
[*>:320.00 B/s--------------------------------------------------------------------------------] 0 ∆ 0
[*>:308.00 B/s--------------------------------------------------------------------------------] 0 ∆ 0
[*>:216.00 B/s--------------------------------------------------------------------------------] 0 ∆ 0

[stefan_armbrust]

What is your exact command line call your using for neo4j-admin import?

[phalvi]

Yes stefan, I am using neo4j-admin import and below is the command.
neo4j-admin import --nodes /tmp/node_export_ne.csv --nodes /tmp/node_export_ep.csv --nodes /tmp/node_export_service.csv --nodes /tmp/node_export_connection.csv --nodes /tmp/node_export_subnet.csv --nodes /tmp/node_export_customer.csv --relationships /tmp/node_export_subnet_ne_rel.csv --mode=csv

I am able to ran the same command in other linux server, there it's able to convert neo4j objects properly. But in my VM, it's not happening . I had checked the neo4j.log and debug.log for any error/warning but it seems to be proper. Is there any additional parameter/setps to be enabled to find the root cause for it ?

[stefan_armbrust]

you can set an env variable NEO4J_DEBUG=true to get more diagnosis. Can you check if CPU is full utilized and what IO is doing while getting stuck?

[phalvi]

Thanks for replying stefan.

I have set the NEO4J_DEBUG variable to true but there was no extra information in debug.log and below is the CPU & IO stats:

avg-cpu: %user %nice %system %iowait %steal %idle
96.33 0.00 3.67 0.00 0.00 0.00

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 1.00 0.00 5.33 0 16
sdb 0.00 0.00 0.00 0 0
scd0 0.00 0.00 0.00 0 0

avg-cpu: %user %nice %system %iowait %steal %idle
96.01 0.00 3.99 0.00 0.00 0.00

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 3.99 0.00 29.24 0 88
sdb 0.00 0.00 0.00 0 0
scd0 0.00 0.00 0.00 0 0

avg-cpu: %user %nice %system %iowait %steal %idle
96.00 0.00 4.00 0.00 0.00 0.00

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 3.00 0.00 24.00 0 72
sdb 0.00 0.00 0.00 0 0
scd0 0.00 0.00 0.00 0 0

[phalvi]

Hi Stefan,
I could see that there was only one cpu in my VM, is that a problem ?

[stefan_armbrust]

I guess stuff is just taking up way more time. Import tool is optimized to use all available CPU power. Why not run the import on the more powerful server and move then the generated db folder over?

[phalvi]

We are using very less number of data to migrate which shouldn't consume more cpu usage. However, I have increased cpu count in my VM and it is working fine. Thanks for replying.