Unable to call user-defined stored procedure from Neo4j Streams Sink Configuration

[2018csb1056]

Hello Graph Community!

Recently I have been trying to create a Neo4j data sink by subscribing it to a topic. I am using Neo4j streams to accomplish this task.

The topic is subscribed properly and all the messages are passed as event object to the cypher query which I have written in the .conf file.

While CREATE cypher query in the configuration is working as expected, I want to write little complicated logic for node creation from stream data and for this I have written a custom procedure.

I am able to load the procedure and ensure that it is functioning properly, but the procedure is not working when data from event object is passed into it. I went through the logs and there were no sign of any failure.

To give more insights here is the configuration which is working -

NEO4J_kafka_bootstrap_servers: "ld-kafka-svc:9092"
NEO4J_streams_sink_enabled: "true"
NEO4J_streams_sink_topic_cypher_tree: "CALL ni.insert_node({name: 'xyz'}) YIELD node RETURN node"

While something like this is not working -

NEO4J_kafka_bootstrap_servers: "ld-kafka-svc:9092"
NEO4J_streams_sink_enabled: "true"
NEO4J_streams_sink_topic_cypher_tree: "CALL ni.insert_node({name: event.name}) YIELD node RETURN node"

This is the signature of the stored procedure -

public Stream<NodeResult> insertNode(@Name("data_dict")Map<String, Object> data)

[david_allen]

I think we would need to see some debug.log output to know what's happening here. In the log file, neo4j-streams can only report what the error of the query was. But in this case, it appears you are calling a custom stored procedure. This raises the possibility that an error is inside of the stored procedure.

By default, neo4j-streams will do some logging when things fail like this. I would recommend starting by looking into debug.log to see what the database is saying when it fails in this way; this information would be needed to go further.

[2018csb1056]

Hey @david.allen!

I have already gone through the logs and there is no error (or warning or info) as such.

Also inside the procedure I am not doing anything special. In fact, procedure does the task of logging Hello World to indicate that it has been called. I am pretty sure I am not making any mistake in writing procedure as such.

Also if there were some problem with the procedure it should not have worked while passing a string explicitly.

[david_allen]

then the next step is to play with the error reporting settings in neo4j-streams and see what you can discover

Neo4j Graph Database Platform

Neo4j Streams - Sink: Kafka → Neo4j - Neo4j Streams Docs

fundamentally, it's necessary to get some kind of error data out of the query that (appears) to be failing, since you can't really debug without it. If the procedure says "Hello world" just before it streams the result back out then this should occur within reportable logs.

Also consider turning neo4j query logging on.

What would happen if you skipped the custom stored procedure and instead used the sink query:

CREATE (t:Test { name: event.name })

Would this succeed? This would help control for where the error might occur.