All pair shortest algorithm is taking very long time on very large data. Is there any way to optimise

[kalyan_b_aninda]

Hello , I am trying to experiment All pair shortest path algorithm on 3 million nodes and writing it into csv file . The algorithm is taking so long time. My instance memory is 374 GB with 48 cores. I am using neo4j version 3.5.15 and gds version 1.1. My query is given bellow

call apoc.export.csv.query("CALL gds.alpha.allShortestPaths.stream({nodeProjection: 'CUSTOMER', relationshipProjection: {SENDMONEY: {type: 'SENDMONEY'}}})YIELD sourceNodeId, targetNodeId, distance WITH sourceNodeId, targetNodeId, distance WHERE gds.util.isFinite(distance) = true MATCH (source:CUSTOMER) WHERE id(source) = sourceNodeId MATCH (target:CUSTOMER) WHERE id(target) = targetNodeId WITH source, target, distance WHERE source <> target RETURN source.WALLETID AS source, target.WALLETID AS target, distance as hop ORDER BY hop DESC, source ASC, target ASC","sendmoney_shortnet.csv",{batchsize:10000});

the algorithm is now running for 12 hrs but my log file is showing this

Feb 06 03:22:23 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 03:22:23.819+0000 INFO  [algo-1] MSBFSAllShortestPaths 21%
Feb 06 03:22:23 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 03:22:23.819+0000 INFO  [algo-4] MSBFSAllShortestPaths 78%
Feb 06 03:27:35 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 03:27:35.299+0000 INFO  [algo-1] MSBFSAllShortestPaths 21%
Feb 06 03:32:26 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 03:32:26.201+0000 INFO  [algo-4] MSBFSAllShortestPaths 78%
Feb 06 03:37:37 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 03:37:37.708+0000 INFO  [algo-1] MSBFSAllShortestPaths 21%
Feb 06 03:47:16 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 03:47:16.321+0000 INFO  [algo-3] MSBFSAllShortestPaths 38%
Feb 06 03:57:15 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 03:57:15.642+0000 INFO  [algo-3] MSBFSAllShortestPaths 38%
Feb 06 04:12:04 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 04:12:04.108+0000 INFO  [algo-4] MSBFSAllShortestPaths 78%
Feb 06 04:17:13 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 04:17:13.998+0000 INFO  [algo-4] MSBFSAllShortestPaths 78%
Feb 06 04:36:51 ip-10-10-1-181.ap-southeast-1.compute.internal neo4j[4678]: 2021-02-06 04:36:51.699+0000 INFO  [algo-1] MSBFSAllShortestPaths 21%

didnt find any progress after that.

is the algorithm stuck by any mean. Kindly help me to figure out the problem