Loading multiple CSV files in Neo4j with each row of each CSV being a node

[kasthuri]

I need to upload multiple CSV files in Neo4j with each row of each CSV file being a node. How to effectively do this? I cannot concatenate the CSVs since each row is a label and the rows are the same across multiple CSVs.

It looks like the LOAD CSV command either take a single CSV and generates nodes for each row or takes a bunch of CSVs and makes a node for each CSV. Thanks!

[ameyasoft]

LOAD CSV reads only one file at a time. You have to run the import with each file.

Other option is copy the data from one file and copy into the first file as second column.

[kasthuri]

Thanks. Can I automate the import with each file? Like writing LOAD CSV filename multiple times in a file and uploading the query file somewhere so each LOAD CSV query is executed? If so, how do I do it?

[mckenzma]

One way I have handled importing ~1000 files is to create another CSV file with the file names/urls. I use LOAD CSV to import the rows from the CSV that contains all the file names and create nodes for each file. Then iterate over the file nodes to import rows from each file.

[kasthuri]

Ok, I will try this. Is there a query that you could provide for such creation? Like a pseudo-code with Cypher statements. Sorry, I am pretty new to Neo4j. Thanks!

[mckenzma]

Hi,
Here is an example of what I have been using:

LOAD CSV WITH HEADERS FROM "url" AS row
WITH row
WITH row, row.URL AS fileUrl
MERGE (file:File: {url: fileUrl})
ON CREATE SET file.url = fileUrl,
              file.folder = row.Folder,
              file.name = row.File,
              file.createdOn = timestamp()

Note that I am currently creating File Nodes in my graph to store the url and then iterating over the file nodes to import and connect data.

MATCH (file:File)
WHERE NOT (file)-[:CONTAINS]->(:Row)
WITH collect(file.url) AS fileURLs
UNWIND fileURLs AS fileURL
CALL apoc.periodic.iterate(
'
CALL apoc.load.csv($url,{header:true,quoteChar:"\u0000"}) YIELD map AS row
RETURN row
','
CREATE (fileRow:Row {createdOn: date()})
SET fileRow += row,
fileRow.url = $url,
fileRow.createdOn = date()
',
{batchSize:10000,parallel:false,params:{url:fileURL}}) YIELD batches, total
RETURN batches, total

After this iterate through and connect file nodes to row nodes. Then once all of that is in the graph I just start working with the raw data within the graph.

I hope this helps.

[sameer_gijare14]

May be you can write a custom procedure that takes input folder where you keep all CSV files.You can loop over all file handles ,then transform data as per your target Data model and finally load it into Your graph DBMS.
Many thanks
Mr Sameer S Gijare

[tangxingbang]

I use python ,neo4j python driver , python os , python csv. Then let python uploads files to database one by one.

Neo4j Graph Database Platform

Using Neo4j from Python - Developer Guides

This guide provides an overview of how to connecting to Neo4j from Python.

https://docs.python.org/3/library/csv.html
https://docs.python.org/3/library/os.html