More Resources on Updating Large Graphs Efficiently

[carib]

Just noticed that the Graph Academy is going through a redesign and overhaul. Would it be possible to also add more courses on updating nodes on large graphs with a focus on efficiency and performance? Say in the millions, or billions, of nodes and relationships. I know that there's a course on ingesting data, but it would be helpful to offer another course that goes more in-depth where users want to inject fresh data on a running graph database so that they can perform daily updates with the driver of their choice (Python, Java, etc). This course can cover topics on things like deleting old and stale nodes and relationships effective and then re-create new relationships between new nodes, or between existing and updated nodes.

That is my suggestion.

[michael_hunger]

Thanks a lot for that suggestions, it's a great topic, and yes we should have more material on that.

In general as a pointer it's best done with apoc.periodic.iterate to determine which nodes to update/delete in the "driving" statement and then perform the update in the "update" statement.

call apoc.periodic.iterate(
'MATCH (n:Node) WHERE n.lastUpdate < datetime() - duration("P1D") RETURN id(n) as id',
'MATCH (n) WHERE id(n) = id SET n.counter = n.counter +1',
{batchSize:50000, parallel:true})

You can call that procedure also from any driver

For updates with outside data I recommend sending batches of data with UWIND as parameters

Medium – 8 Jul 20

5 Tips & Tricks for Fast Batched Updates of Graph Structures with Neo4j...

This post was originally published on my blog, but I think that a wider audience can benefit from it.

Reading time: 6 min read