Create large amount of relationships [not enough memory]

[nadavbe]

I got this large dataset (~1M nodes) of actions, each is attributed to some user id ("id") and phase_index that indicates a chronological order.
I want to create a relationship within these nodes, in order to draw a path for each id, based on the order of the index.
When trying to create this in a simple way, I get a not enough memory error.
And idea on how to iterate on this, so that each id will build its entire path?

[jggomez]

Hi, How are you doing those relationships? are you using UNWIND??

Thanks

[nadavbe]

This is what I currently use:
MATCH (a1:Actions)
WITH a1
match (a2:Actions{id:a1.id, phase_index:a1.phase_index+1})
merge (a2)<-[:JOURNEY_PHASE]-(a1)

[jggomez · Accepted solution]

Hi, You could try periodic commits.

You can read the following information:
https://neo4j.com/docs/labs/apoc/current/graph-updates/periodic-execution/

Thanks

[nadavbe]

I'm pretty new to this. Using desktop and seeing some examples online, this is what I came up with, though I'm really not sure what the last return is for. Anyways, it's running for a long time (and no conclusion yet). Any comments on my implementation? Thanks!

call apoc.periodic.commit(
"MATCH (a1:Product)
WITH a1 limit $limit
match (a2:Product{id:a1.id, phase_index:a1.phase_index+1})
merge (a1)-[:JOURNEY_PHASE]->(a2)
return count(*)",
{limit:100000})

[Cobra]

Hello @nadavbe

I think 100000 as limit is too big, you should use 10000 or 1000

call apoc.periodic.commit("
MATCH (a1:Product)
WITH a1 limit $limit
MATCH (a2:Product{id:a1.id, phase_index:a1.phase_index+1})
MERGE (a1)-[:JOURNEY_PHASE]->(a2)
RETURN count(*)
", {limit:1000})

Regards,
Cobra

[nadavbe]

Hi @Cobra,
Actually, I let it run for a while and query the nodes while running.
It seemed to be stuck on some number and never advance, for a reason I can't understand.
I then tried to periodic iterate instead and it worked pretty quickly.
To me it looks like both methods should provide the same results, but for some reason only iterate worked for me.
But at least I solved it 🙂