Spark UDF calling neo4j

[idayan]

Hi,

On my app I have data of 5 million users. I load it into Neo4j as a graph.

Also, on Spark I processed the data, and for each user I need to query Neo4j.

I did it using a spark UDF, that http calls Neo4j server.

It took too long and get connection errors.

What is the better way to do 5M queries to neo4j?

[david_allen]

The better way to do 5 million queries is to not do 5 million queries to neo4j. :).

The better way would be to use something like the neo4j spark connector. Use 1 cypher query to pull all of the data you need from Neo4j into a single DataFrame, and then use standard Spark SQL to join that resulting dataframe to the data that you have.

That's 1 big query pulling 5 million results, which you can then further partition and join in spark.

GitHub

neo4j-contrib/neo4j-spark-connector

These are the beginnings / experiments of a Connector from Neo4j to Apache Spark using the new binary protocol for Neo4j, Bolt. - neo4j-contrib/neo4j-spark-connector

[idayan]

Thanks for you response.

Now I need to figure how to craft the query that ask 5M questions with getting out of memory 🙂 . Will open new topic if needed.