Very slow cypher queries to create relationships

[chauhan_ishan]

Hey guys,

I am experimenting with the community edition of neo47 using the browser application. We may move onto the enterprise one but there's a roadblock that I am seeing. Here's my problem:

I imported and creates 1.5 million+ nodes with 20 metadata properties using APOC. Now, I need to create relationships between approx 250k of these service nodes. I am using APOC to do that and the data is in python dictionary(json) format in the code. The queries are extremely slow, it took 4-5 hours just to get 20k relationships build out. Here's my query:

query = '''
CALL apoc.periodic.iterate("UNWIND $relationship_list as individual_service RETURN individual_service",
"MATCH (parent_service:Service_Asset_ID), (child_service:Service_Asset_ID) WHERE parent_service.id=individual_service.parent AND child_service.id=individual_service.child AND parent_service.source=child_service.source MERGE (parent_service)-[:SERVICED_BY]->(child_service) ",
{batchSize:1000, iterateList:true, parallel:true, params: {relationship_list:$relationship_list}})
'''
I have tried to follow up on some of the best practices but to no avail. I have also indexed the id and source fields. Any suggestions ?

Neo4j version: 4.1.3
I am using python driver

Thanks,
Ishan

[TonyOD27 · Accepted solution]

Hi Ishan,

I had a similar issue. What solved this problem for me was using the neo4j-admin import tool to create both, nodes and relationships, at the same time. Here are the docs: https://neo4j.com/docs/operations-manual/current/tools/import/

Creating millions of nodes was not a problem for me using APOC or LOAD CSV. However, the APOC functions were taking more than 6 hours to create the relationships between a subset of 300,000 nodes.

If you're already working in Python, then creating the csv files for the neo4j-admin import tool should not be too difficult. Pandas was pretty much all I needed for this and the neo4j-admin import tool took about 30 minutes to create all of my nodes and relationships.

I hope this helps. Best of luck.

-Tony