Neo4j apoc.periodic.repeat doesn't work with streams.publish

[kavitakjava]

Does anyone ever used Neo4j apoc.periodic.repeat function to publish message to kafka? Here is what I have now:

I have a working query to send message to my kafka topic:

CALL streams.publish("my-neo4j-topic", "Hello World from Neo4j!")

However, when I used apoc repeat function:

CALL apoc.periodic.repeat('kafka-only',
    'CALL streams.publish("my-neo4j-topic", "Hello World from Neo4j!")', 5)

Nothing happens..

I have checked the log/debug log and can't get any hint. Can anyone help me?

This is my configuration:

kafka.zookeeper.connect=localhost:2181
kafka.bootstrap.servers=localhost:9092
streams.procedures.enabled=true
streams.source.enabled=true
streams.source.schema.polling.interval=10000
dbms.security.procedures.unrestricted=apoc.*

Please provide the following information if you ran into a more serious issue:

• neo4j version: neo4j-community-3.4.17
• desktop version : MacOS 10.12.6
• which plugins / extensions / procedures do you use : apoc-3.4.0.7-all.jar and neo4j-streams-3.4.4.jar
• neo4j.log and debug.log : No exceptions and no logs

[david_allen]

Can you be more specific on nothing happens? You don't see the messages going to Kafka, or you don't get any results from APOC?

What happens if you call streams.publish() by itself without the APOC?

[kavitakjava]

When I use APOC procedure then I don't see the messages getting publish to Kafka topic.
If i call streams.publish() without APOC in that case message is getting published in Kafka topic.

[michael_hunger]

Could you try to reproduce with a 3.5.12 setup and appropriate more current apoc + streams versions?

[kavitakjava]

Yes, I tried to reproduce with 3.5.12 version and it has same issue.
streams.publish without apoc producer is publishing message in Kafka topic.

CALL streams.publish("my-neo4j-topic", "Hello World from Neo4j!")

Below query with apoc job management procedure is not publishing message in Kafka topic.

CALL apoc.periodic.repeat('kafka-only',
    'CALL streams.publish("my-neo4j-topic", "Hello World from Neo4j!")', 5)

[conker84]

Thanks @kavitakjava please try with the following jar:

Dropbox

neo4j-streams-3.5.4.jar

Shared with Dropbox

I look forward to your feedback.
Thanks a lot
Andrea

[kavitakjava]

Hi Andrea @conker84,

The jar which you provided worked with neo4j-community-3.5.13 and i can see messages published every 5 seconds in the Kafka topic !!

The jar from https://github.com/neo4j-contrib/neo4j-streams/releases/tag/3.5.4 did not worked.

Seems like you have some fixes in ur jar.
Can this be available on github?

Your fix worked! Thank you so much for your help!

Best,
Kavita

[conker84]

Yes I fixed the bug in that jar. I'm preparing a PR so it'll be available with the next release.
Thanks a lot for pointing this out!

[kavitakjava]

Awesome!
I'm glad that my use case helped to brought up this issue !

Will this fix available for previous releases e.g. 3.4.17 so that we might now have to upgrade the version in production?

[conker84]

@kavitakjava sorry for the delay, do you think it's an option upgrade Neo4j to v 3.5.13?
Please lemme know otherwise we have to find an alternative to support you.
Thanks a lot

[kavitakjava]

Hi @conker84,

Has this issue fixed got in the latest release? We can upgrade to the latest release in prod.
Thank you!

Best,
Kavita

[conker84]

HI @kavitakjava we did a new release yesterday that contains the fix:

GitHub

neo4j-contrib/neo4j-streams

Neo4j Kafka Integrations, Docs =>. Contribute to neo4j-contrib/neo4j-streams development by creating an account on GitHub.

[kavitakjava]

conker84:

neo4j-contrib/neo4j-streams

Awesome!!
Thank you so much @conker84