CSV loading, out of memory problem

[jupiter]

hello,
I'm trying to import data using this code:

:auto USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
MERGE (followed:User {twitter_id: followers.followed})
MERGE (follower:User {twitter_id: followers.follower})
MERGE(follower)-[:FOLLOW]->(followed)

But seems that my RAM is not enough to execute the query, I tried to split it into two queries and I just tried to execute this one:

:auto USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
MERGE (:User {twitter_id: followers.followed})
MERGE (:User {twitter_id: followers.follower})

but I always get the same problem, how should I write the query to solve? increasing the commit frequency doesn't help.
I have to say that the first two MERGE will fail a lot of times because there are a lot of duplicated entries in the file, I don't know if it is important.

thanks

[sameerG]

Follow the below mentioned sequence and see if memory error goes away

1. Issue command $neo4j-home > bin/neo4j-admin memrec --memory=50%of RAM in g --database=Your_db_name
2. Look at the output parameters.
3. Stop Neo4j server
4. Copy parameters to neo4j.conf file and apply the changes.
5. Start the server
6. Modify Query (:auto USING PERIODIC COMMIT 200 or 300)
7. See if it still going out of memory
8. If yes then you can run PROFILE on your queries to see if they use EAGER loading
   and either modify queries or run multiple passes on the same file, so it does not do this.

[intouch_vivek]

Hi Jupiter,

Welcome to the community!!
Kindly lets know the available RAM in your system and also the dataset size you are trying to ingest.
As per the query you have mentioned it should not have any problem in ingest unless and until you are very less available memory. Also I hope you have not modified memory configuration setting in the config file.

Also for the time being try below

USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
MERGE (followed:User {twitter_id: followers.followed})

USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
MERGE (follower:User {twitter_id: followers.follower})

USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
Match (followed:User {twitter_id: followers.followed})
Match (follower:User {twitter_id: followers.follower})
MERGE(follower)-[:FOLLOW]->(followed)

[jupiter]

intouch.vivek:

USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
MERGE (followed:User {twitter_id: followers.followed})

USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
MERGE (follower:User {twitter_id: followers.follower})

USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS FROM "file:///followers.csv" AS followers
Match (followed:User {twitter_id: followers.followed})
Match (follower:User {twitter_id: followers.follower})
MERGE(follower)-[:FOLLOW]->(followed)

This solved for me, thanks!