Greetings.
I want to transfer my question from StackOverflow over here as well as I'm seeking for understanding how apoc.cypher.mapParallel / apoc.cypher.mapParallel2 functions actually work.
Full setup is decribed in linked question, I'll keep it short here:
When I'm checking how's my parallelized query (apoc.cypher.mapParallel in the core) is executing with help of dbms.listQueries() I see which iterable items are being processed at the moment.
And there are a few problematic regularities:
-
It doesn't matter which values are set in "batchSize", "concurrency", "partitions" (for apoc.cypher.mapParallel2) parameters - amount of iterated items always stays the same, performance isn't affected in any way. All examples in StackOverflow question and down here are tested with batchSize=10 and concurrency=10.
-
However, the longer is the iterable list - the more items from that list are being processed at once. For example: same query on the same server iterates by 1 item if list length = 10 items, by 3 items for list length ~ 900, and by 10 items for list length ~ 3700 items.
-
Exactly same queries on servers with different hardware are being iterated over different number of items. For example: server with 80 GB RAM processing 10 items at once (~3700 items in iterable list), slower backup server processing 5 items for same query and dataset.
-
But despite that, slower backup server doing the job only 15-30% longer than main server.
Is there a way to override that execution plan and manually choose how many items should be in the batch?
