GraphAware issue: (Maybe) Failed to invoke procedure `apoc.periodic.iterate`: Caused by: java.util.concurrent.RejectedExecutionException:

[david_meza01]

I am running through the Grtaphaware NLP example
After working through websocket issues I now get this error when running this command:

CALL apoc.periodic.iterate( "MATCH (l:Lesson) RETURN l",
 "CALL ga.nlp.annotate({text: l.lesson, id: id(l)}) YIELD result
 MERGE (l)-[:HAS_ANNOTATED_TEXT]->(result)", {batchSize:1, iterateList:True}) 

Neo.ClientError.Procedure.ProcedureCallFailed: 
Failed to invoke procedure `apoc.periodic.iterate`: 
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a5af280 
rejected from java.util.concurrent.ThreadPoolExecutor@4154d930[Shutting down, pool size = 1, active threads = 1, queued tasks = 49, completed tasks = 194]

My initial heap size is 1G with 4G max.

neo4j version 3.5.3 Enterprise, with appropriate apoc and graphaware plugins
desktop version Neo4j Desktop 1.1.21

[ryan_boyd]

david.meza01:

RejectedExecutionException

Welcome to the community @david.meza01! Would be great if you could introduce yourself over here (https://community.neo4j.com/c/general/introduce-yourself).

This actually looks like something is limiting access to launch another thread. @michael.hunger - can you take a look?

[michael_hunger]

Hi David,

can you try a bigger batchSize e.g. 100 ?

How many CPUs do you have ?

Looks a bit like resource exhaustion.

Does it work if you call it regularly for a single lesson?

[david_meza01]

I have not tried a higher batch size. The Graphaware tutorial states to leave it at 1.
I am running this on an MAC Book pro with 4 cores and 16GB of RAM.
It will run on the first 300 or so lessons and then fail.

[njmmeibergen]

I am running into the exact same problem, with the same macbook specifications. When performing only on the first 200 everything works fine. However, when I try again on the next 200 (by using 'limit 200'), neo4j still shuts down. It seems that it doesn't matter whether to perform all in one go or run in sets of 200. It will always shut down. Seems that apoc.periodic.iterate keeps something in memory?

@david.meza01 Have you been able to solve this issue?

@michael.hunger Any idea on this, given that I run it regularly for a single lesson and it still shuts down after enough have been run?

[david_meza01]

For this dataset, I needed to set my Heap size to 6GB, then it worked. However I ended up running it on a server on AWS instead of my laptop.

[njmmeibergen]

Hi David,

Thanks for getting back to me. I just used 6GB but that did not change a lot. I'm currently experimenting on Azure. Did increasing the Heap size eventually improved the amount of annotation you were able to perform?

[david_meza01]

Yes, the increased Heap size allowed me to run the process. One of the things the people from GraphAware told me was to make sure my text was cleaned and ended with a period. Are you using the Stanford NLP in your pipeline?

[dhughes1]

What cleaning steps did they recommend, other than termination with a period? I am having the same issue. While monitoring the db I see two cycles of Heap usage peaks and then on the third peak it always crashes. I was monitoring with GraphLab’s Neo4j App.

BTW, what was your solution to the Websocket issue, we are seeing that error as well.

[mfkilgore]

@dhughes1would you happen to be running on a mac? I have seen the same problem myself and a common theme appears to be the OS and the call to apoc.periodic.iterate possibly independent of the GraphAware plugin but I have not had the time to confirm that yet.