NiFi goes Neo

[martin_ebert]

Hi community,
I've already tried Kettle and Neo. That went without a hitch.
Now I want to use NiFi and Neo together and encounter problems:

Neo processors are not included in the standard installation of NiFi, but there is a "nifi-graph-bundle" in the official NiFi Github Repo (https://github.com/apache/nifi/tree/master/nifi-nar-bundles).:

Following the instructions I come across some errors.

Since I haven't worked with Maven and Java yet, I don't know if it's the configuration.

Error message after executing "mvn -T 2.0C clean install" in the nifi root directory:

Can someone help me and even provide me with the required nar file? Initially I wanted to reach the graph database via JDBC, but of course the corresponding Execute SQL processor doesn't understand Cypher.

[martin_ebert]

First of all, the graph bundle is coming but unfortunately not with nifi 1.9.2 but with 1.10.0 RC.

I also found out what it takes to successfully create the package: Execute

mvn clean install -DskipTests

to compile tests, but skip running them.

So is there currently no way to work with Neo4j on nifi 1.9.2?

[jdsbanares]

Hi @martin.ebert,

Thank you for this! Would you know if the graph bundle is compatible with Nifi 1.9.2?

[larsen_raja]

martin.ebert:

DskipTests

Does this mvn clean install take a long time? It's going on for several hours in my laptop and often fails with connection reset error.

[martin_ebert · Accepted solution]

NiFi 1.10.0 is out now.

#Bad News
The graph bundle is not included in the original nifi 1.10.0 but you can create the bundle by yourself.

#Good News
The processor works perfectly and only with URL, username and password.

You can choose:

The Controller Service is set up in seconds:

Processor is really simple:

Most importantly, it works:

[larsen_raja]

Which of the below nar files should be copied over to

\nifi-1.10\nifi-assembly\target\nifi-1.10.0-bin\nifi-1.10.0\lib

[larsen_raja]

Got this done. After setting up nifi per the build instructions given in Github, just copy the below nar files to

nifi-graph-client-service-api-nar-1.10.0.nar
nifi-graph-nar-1.10.0.nar
nifi-neo4j-cypher-service-nar-1.10.0.nar
nifi-other-graph-services-nar-1.10.0.nar

\nifi-1.10\nifi-assembly\target\nifi-1.10.0-bin\nifi-1.10.0\lib

[tom1]

Thanks for the guide.

I'm trying the same setup and can successfully query Neo4j using Cypher in the ExecuteGraphQuery when returning one node. However, when I query for multiple nodes I get an empty array back. For example, following this tutorial: https://neo4j.com/graphgist/first-steps-with-cypher

Running this query:

CREATE (you:Person {name:"You"})
RETURN you

Result:

[{"name":"You"}]

But a Cypher query with multiple nodes, gives an empty JSON array:

Query:

MATCH  (you:Person {name:"You"})
CREATE (you)-[like:LIKE]->(neo:Database {name:"Neo4j" })
RETURN you,like,neo

Result:
[{"you":{},"like":{},"neo":{}}]

Does anyone know how to solve this? we're using the graph bundle from 1.10

[yyyguy]

Hey @tom1,

What database version of Neo4j are you using? I have found an issue with particular versions of the Neo4j database not working properly with my NiFi implementation.

-yyyguy

[tom1]

Hi @yyyguy
We're using a Neo4j Docker image with version 3.5.15 and can connect fine:

[yyyguy]

That's good. Have you confirmed that the Neo4JCypherClientService controller service is Enabled?

I apologize for asking such basic questions. It is just that I have had this controller service sitting in an "Enabling" state without me realizing it.

-yyyguy

[tom1]

Yes, definitely enabled and we can run the Cypher queries fine:

[yyyguy]

Great. If you do a simple query like:

MATCH (n:Person)
RETURN n.name

it should return multiple results, however you are seeing no results returned? Is that correct?

-yyyguy

[tom1]

yyyguy:

MATCH (n:Person) RETURN n.name

Yes, that query runs fine giving this:

However, a query that returns multiple nodes in the Neo4j browser:

Gives an empty JSON array in the Flowfile content:

[yyyguy]

@tom1 can you run the query that I provided. I would like to see what the results are without the "you" filter. Also, can you provide a screenshot of the results in the LogAttribute processor on the success branch?

-yyyguy

[tom1]

Sure:

Gives this in the 'Success' branch:

Flowfiles all contain this:

[yyyguy]

Ok, so in the next query can you please try this? Note: This assumes that you already have a Database node with name = "Neo4j"

MATCH (you:Person {name:"You"})
MERGE (you)-[:LIKE]->(neo:Database {name:"Neo4j" })
RETURN you.name, neo.name

-yyyguy

[tom1]

Now this query:

yyyguy:

MATCH (you:Person {name:"You"})
MERGE (you)-[:LIKE]->(neo:Database {name:"Neo4j" })
RETURN you.name, neo.name

OK this works:

[yyyguy]

@tom1, it looks like if you do not have any properties associated with a relationship [like:LIKE], the query will fail.

Do you have any relationship properties that you plan to include in your database design? If not, do you think that is workable?

-yyyguy

[yyyguy]

BTW, it looks like you have created 3 relationships between the You node and the Neo4J node. MERGE is a better option perhaps.

-yyyguy

[tom1]

OK great - thank you for your help. We'll see if we can rework our schema and then try our query again

[bhaumik0369]

1.how did you passed the flowfile through the ExecuteGraphQuery processor
2.when i pass a query in process it gives "WRITER server connection not established error"

[martin_ebert]

Did anyone manage to ingest the FlowFile content? No matter if Parquet or json format.

I could well imagine that this could work with Load JSON. I can save the FlowFile content as an attribute if needed and reuse it directly in the Cypher statement.

I just want to save the intermediate step (save data locally and write it to Neo4j)

[gaving]

Don't suppose you got anywhere with this one?

Currently doing it ourselves with ExtractText and an attribute, but feels like a bit of a hack and would rather use the JSON in the flowfile content!

[kevinneo4j]

hi

Thanks @yyyguy and @martin.ebert for information above. I have Apache Nifi 1.12 working against Neo4j 3.5.15.

However, when using Apache Nifi 1.12.0 I have problems connecting to the Neo4J version 4.0.4. The Neo4JCypherClientService throws a "org.neo4j.driver.v1.exceptions.ServiceUnavailableException. Connection to the database terminated"

You mentioned that you did have problems connecting with different versions of Neo4j. I was wondering if you had a chance to look at this and if there was a fix on the way.

Regards,
Kevin

[martin_ebert]

Hi Kevin,
you talk about NiFi 1.11.x. 1.12 is not released yet. Anyways. I noticed the exact same error message. The neo4j nar files are not 4.0 compatible.

[martin_ebert]

Yes 3.x is working with NiFi 1.9.2.

[karim_el]

Hello Martin,

Do you have any news regarding the compatibility of recent versions of neo4j and Nifi ? I tested the following versions :

Nifi 1.11.x + Neo 4.2.2 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.1.5 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.0.11 => ServiceUnavailableException
Nifi 1.11.x + Neo 3.5.26 => OK

Nifi 1.12.x + Neo 4.2.2 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.1.5 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.0.11 => ServiceUnavailableException
Nifi 1.11.x + Neo 3.5.26 => OK

[conker84]

@karim.el I think that the best option to connect NiFi to the last Neo4j versions is via JDBC driver.