Cannot load data from cypher script

[deven_pandey]

Below is my content of cyper file.

WITH "file:///cite1.json" AS url
CALL apoc.load.json(url) YIELD value
UNWIND value.citation as q with q where q.venue.id is not null
MERGE (citation:citations {id:q.id}) ON CREATE
SET citation.id = q.id, citation.title = q.title, citation.year = q.year, citation.n_citation = q.n_citation, citation.page_start = q.page_start, citation.page_end = q.page_end, citation.doc_type = q.doc_type, citation.publisher = q.publisher, citation.volume = q.volume, citation.issue = q.issue, citation.doi = q.doi

MERGE (venue:venue {id:q.venue.id}) ON CREATE SET venue.id = q.venue.id, venue.raw = q.venue.raw, venue.type = q.venue.type
MERGE (venue)-[:FOR]->(citation)

FOREACH (refName IN q.references | MERGE (ref:References {name:refName}) MERGE (citation)-[:REFERRED_BY]->(ref))

FOREACH (a IN q.authors |
MERGE (author:authors {id:a.id}) ON CREATE
SET author.name = a.name, author.id = a.id
MERGE (citation)<-[:AUTHORED_BY]-(author)
MERGE (citation)-[:WROTE_BY]->(author))

FOREACH (f IN q.fos |
MERGE (fos:fos {name:f.name}) ON CREATE
SET fos.name = f.name, fos.w = f.w
MERGE (citation)<-[:FOS_DETAILS]-(fos)
)

When I run

cat /tmp/load_data.cypher | /usr/bin/cypher-shell -u usename -p password

nothing happens. no error but data is not loaded

[dana_canzano]

is your cypher statement terminated by a ; ?

also you should be aware that if environment variables NZ_USER and NZ_PASSWORD are defined, representing the username and password to connect with, then you no longer need to user -u username -p password

[deven_pandey]

When I use semicolon at the end I get below error. I used semicolon after the final right parentheses. Same query works when logged into cypher-shell

invalid input 'p': expected 'n/n' or 's/s'

[dana_canzano]

What version of Neo4j? though I suspect this shouldnt be of issue but I am unable to replicate using Neo4j 4.1.0 and on a ubuntu os.

I added a ; as the final character to the cypher statement and thus the last 3 line looks like

 tail -3 test
SET fos.name = f.name, fos.w = f.w
MERGE (citation)<-[:FOS_DETAILS]-(fos)
);

and my file has 22 lines, as evidence

wc -l test
22 test

and yet

 cat test | ./cypher-shell
neo4j@neo4j-lg:~/single/instance1/neo4j-enterprise-4.1.0/bin$

i.e. no error is logged?

[deven_pandey]

Yes no error logged but it doesn't even inserts data

[dana_canzano]

ok.. now im confused.
but previously you indicated you recieved an error

When I use semicolon at the end I get below error. I used semicolon after the final right parentheses. Same query works when logged into cypher-shell

invalid input 'p': expected 'n/n' or 's/s'

and your last update now indicates

Yes no error logged but it doesn't even inserts data

so the error is no more? ???

[deven_pandey]

Ok let me elborate.

this is the script.

#!/bin/bash

cd /tmp
ls -1 cite* > list_file

for i in cat list_file
do
cat $i > tmp_file
sed -e "s/\r//g" tmp_file > $i
#sed -e "s/^M//" tmp_file > $i
echo "$i"

echo "WITH "file:///$i" AS url CALL apoc.load.json(url) YIELD value UNWIND value.citation as q with q where q.venue.id is not null MERGE (citation:citations {id:q.id}) ON CREATE SET citation.id = q.id, citation.title = q.title, citation.year = q.year, citation.n_citation = q.n_citation, citation.page_start = q.page_start, citation.page_end = q.page_end, citation.doc_type = q.doc_type, citation.publisher = q.publisher, citation.volume = q.volume, citation.issue = q.issue, citation.doi = q.doi MERGE (venue:venue {id:q.venue.id}) ON CREATE SET venue.id = q.venue.id, venue.raw = q.venue.raw, venue.type = q.venue.type MERGE (venue)-[:FOR]->(citation) FOREACH (refName IN q.references | MERGE (ref:References {name:refName}) MERGE (citation)-[:REFERRED_BY]->(ref)) FOREACH (a IN q.authors | MERGE (author:authors {id:a.id}) ON CREATE SET author.name = a.name, author.id = a.id, author.org = a.org MERGE (citation)<-[:AUTHORED_BY]-(author) MERGE (citation)-[:WROTE_BY]->(author)) FOREACH (f IN q.fos | MERGE (fos:fos {name:f.name}) ON CREATE SET fos.name = f.name, fos.w = f.w MERGE (citation)<-[:FOS_DETAILS]-(fos));" > /tmp/load_data.cypher

#cat /tmp/load_data.cypher
cat /tmp/load_data.cypher | /usr/bin/cypher-shell

Now I don't get an error but it doesn't load data as well

devendra@cs-web-drupal:/var/lib/neo4j/data/tmp$ ./load_data.sh
cite2.json
cite2.json processed

Shows processed instantly. However on cypher shell running this query takes some time

[dana_canzano]

What version of Neo4j ? Again maybe it doesnt matter ? maybe it does? ?? ?

also, your cypher is performing a MERGE which is effectively a create or replace. if its replacing then you would see no new nodes/relationships created

[deven_pandey]

While running manually it shows some relationship. How do I ensure that is is actually inserting processing or replacing data. Is there a way? sorry I am new to Neo4j

[deven_pandey]

Here's the result running manually.

devendra@neo4j> WITH "file:///cite4.json" AS url
CALL apoc.load.json(url) YIELD value
UNWIND value.citation as q with q where q.venue.id is not null
MERGE (citation:citations {id:q.id}) ON CREATE
SET citation.id = q.id, citation.title = q.title, citation.year = q.year, citation.n_citation = q.n_citation, citation.page_start = q.page_start, citation.page_end = q.page_end, citation.doc_type = q.doc_type, citation.publisher = q.publisher, citation.volume = q.volume, citation.issue = q.issue, citation.doi = q.doi

            MERGE (venue:venue {id:q.venue.id}) ON CREATE SET venue.id = q.venue.id, venue.raw = q.venue.raw, venue.type = q.venue.type
            MERGE (venue)-[:FOR]->(citation)

            FOREACH (refName IN q.references | MERGE (ref:References {name:refName}) MERGE (citation)-[:REFERRED_BY]->(ref))

            FOREACH (a IN q.authors |
            MERGE (author:authors {id:a.id}) ON CREATE
            SET author.name = a.name, author.id = a.id
            MERGE (citation)<-[:AUTHORED_BY]-(author)
            MERGE (citation)-[:WROTE_BY]->(author))


            FOREACH (f IN q.fos |
            MERGE (fos:fos {name:f.name}) ON CREATE
            SET fos.name = f.name, fos.w = f.w
            MERGE (citation)<-[:FOS_DETAILS]-(fos)
            );

0 rows available after 124104 ms, consumed after another 0 ms
Created 1902 relationships

[deven_pandey]

Version info

Connected to Neo4j 4.2.0

[clem]

What I do when I have trouble with an import (or anything at all complex) with Cypher, is make a version that is simple as possible and gradually build up from there. (You can also use PROFILE to see where the data "disappears".)

e.g. start with

WITH "file:///cite1.json" AS url
CALL apoc.load.json(url) YIELD value
RETURN value.citation, value.venue.id

and see if you are getting what you expect. (If you had a typo in a field name, Cypher doesn't complain...)

and slowly add statements. Also, do this on a clone of your DB, which you can delete if it doesn't work, so you don't pollute your existing data.

It's a bit hard to eyeball this much Cypher code and see what's wrong.

You might want to follow this thread on Debugging Cypher:

Debug cypher query Cypher

Is there a way to proper debug a cypher query? I've been busting my head over an integer value vs string property which I could not detect as it just did not give any error, not even a hint. And also the query was embedded in apoc.do.case, hence, no real visibility on what happens in there. Explain and profile are not quite what I would call "debug" = being able to pause, as to check the variable value and the result of the function, step by step Cutting and shortening the query into pieces a…

[deven_pandey]

My same query works from within cypher-shell only problem is it doesn't work when I put same query in a file and try to run that using cypher script.

[clem]

What if you have a very simple query in your file, like RETURN 'Hello world'. Does it work?

We need to separate the script vs. executing from a file.