"forces all dependent data to be materialized in main memory before proceeding" - Upload CSV file with 30K nodes

[bmalbusca]

Hello, I'm trying to process a big file with data from multiple monitoring stations. I've found a solution, but the same thing that Neo4j's warning tells me "the execution plan for this query contains the Eager operator, which forces all dependent data to be materialized in main memory before proceeding", I am also not satisfied with this query. Any help is going to be great!

LOAD CSV WITH HEADERS FROM "file:///data.csv" as obj
MATCH (n:Station) WHERE (obj.SNIRH CONTAINS n.SNIRH) OR (n.EDIA=obj.EDIA)
WITH n, obj
CREATE (data:Data)
SET data += obj
MERGE (data)-[r:COLLECTED]->(n)
RETURN count(r) as Relations, count(data) as Data

[Joel · Accepted solution]

Have you tried using PERIODIC COMMIT? I try to use it whenever loading anything more than a trivial dataset. I see some notes in there about Eager operator as well.

Neo4j Graph Database Platform

Importing CSV Data into Neo4j - Developer Guides

This article demonstrates different approaches to importing CSV data into Neo4j and solutions to potential issues that might arise during the process.