How to accelerate writing

[shao-shuai]

I have 200,000,000 relationships to write into my database. The data is a nested list like b = [[1, 2],[3, 4], [5, 6]]. Right I am using Neo4j Bolt Driver for Python to write data. But the writing is very slow, it would take 5 days to finish all the writing.

I am using the following query:

query='''WITH $names AS nested
UNWIND nested AS x
MERGE (w:Patent {name: x[0]})
MERGE (n:Patent {name: x[1]})
MERGE (w)-[r:NIHAO]-(n)
'''

Is there anyone who can help me with the writing? Many thanks!

[Thomas_Silkjaer]

Is it a new database, or an existing database?

If it's a new, it's more efficient to use the neo4j-admin import tool.

If it's an existing, it can be beneficial to do the imports in parallel using apoc.periodic.iterate – but only if you can do it in a way where you are sure to avoid deadlock situations.

Also, if you are sure that the Patent's exist, it will be more beneficial to match them, rather than merge them.

[shao-shuai]

Thanks Thomas, it's a new database. I tried neo4j-admin import, it is reaally fast. The thing is, if I use csv, I have like 1600 csvs to import, I'll try to find a way to make some Concatenation.

Also thanks for the MATCH suggestion, definitely I'll give it a try.

[conker84 · Accepted solution]

Did you create a constraint on Patent.name?

CREATE CONSTRAINT ON (p:Patent) ASSERT p.name IS UNIQUE

[conker84]

Moreover don't try to send the 2*10^8 relationships to the UNWIND all in once, try split them in batches of 10/20k

[shao-shuai]

Thanks Conker for let me know UNIQUE, I'll have a try.

[shao-shuai]

Thank you Conker again, it's super fast!