GDS and Causal Cluster Lack of Integration

[stu_v_kerr]

Now that we have been using the Neo4J graph database for a few years and have integrated and using the GDS library we are now moving to a causal cluster. But Neo4J has not integrated the GDS library to run on a cluster. What? GDS algorithms are allegedly parallelized, but they are unable to run on a cluster? I remember that being the entire point of using Hadoop and Spark - eliminate the transport of data across the network. Yet here we are again? Starting to look at TIgerGraph.

When will this integration issue be fixed?

[alicia_frame1]

Neo4j causal clusters are built for fault tolerance and high availability (they're ACID compliant); they're not built for scale out (and don't use Hadoop or Spark). If you're looking for a big data scale out solution, that would be Neo4j fabric. The parallelization of the algorithms is intended to leverage multiple CPUs; if you have an enterprise license you can simply set the concurrency parameter and see speed ups for your algorithm execution time.

GDS does not run on core members of a causal cluster because the algorithms are extremely memory hungry and operate on a longer time scale than simple queries. In a cluster, this causes problems with leader election, and leads to instability.

If you're looking to "integrate" GDS with a causal cluster, you can run GDS on a read replica (and consume the results in a separate program, or use kafka to apply writes to the leader) or you can detach a single instance from your cluster with the same data.

A thorough discussion of why GDS shouldn't run on a causal cluster, and your options, is available here: Installation - Neo4j Graph Data Science

If you're primarily interested in availability, we'll be introducing warm backups/read replicas to GDS with Neo4j 4.3.

If what you're actually after is scale out, then we recommend either using a bigger box (we have customers running GDS on tens of billions of nodes in production) or leveraging Fabric. We document how to use GDS with fabric here.

[taffyb]

I appreciate that this topic is a little dated now. So I was wondering if there has been any change/progress on this topic. As I understand it a limitation of the Neo4j sharding approach (please correct me if I am wrong) is that it is not possible to create cross shard relationships. @alicia_frame1 am I correct that the proposed approach (link above) requires that the entire graph being analysed is in a single shard? How is this scaling?

[alicia_frame1]

Hi @taffyb - I think you're conflating a few different topics here. Let me break them down and see if I can help.

For GDS & causal clusters: we introduced a cluster compatible deployment option for GDS in 2.0 - https://neo4j.com/docs/graph-data-science/current/production-deployment/causal-cluster/

From the perspective of the core database and fabric, we use logical sharding which means that by design, there shouldn't be too many cross shard edges. However, you can match entities across shards in order to allow cross shard joins. This developer guide gives a great overview: https://neo4j.com/developer/neo4j-fabric-sharding/ 

For running graph algorithms on sharded data, we support the ability to run an algorithm on an individual shard - https://neo4j.com/docs/graph-data-science/current/production-deployment/fabric/

As I'm sure you understand, many graph algorithms don't distribute very well (eg. https://www.cs.bgu.ac.il/~elkinm/book.pdf). That's why libraries like Spark's GraphX/GraphFrames only ever offered a few algorithms. Knowing this limitation, our focus has been on developing compression techniques and highly performant algorithms that can operate in a scale up context. We've successfully benchmarked our algorithms on graphs with hundreds of billions of nodes and relationships using widely available cloud compute instances.