Neo4j 3.5.17 unstable waiting forever for a lock

[mendoza1024]

We have an application with 10M nodes and recently we started observing that the neo4j server became unresponsive, after a while it's observed a few thread running indefinitely and no more request/queries can execute. I dump the process with jstack and found the below stacktrace all over multiple threads. I'm still experimenting to figure out what triggers this issue but makes neo4j totally unresponsive.

Thread 26056: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
 - org.neo4j.kernel.impl.locking.community.RWLock.waitUninterruptedly(long) @bci=34, line=460 (Interpreted frame)
 - org.neo4j.kernel.impl.locking.community.RWLock.acquireWriteLock(org.neo4j.storageengine.api.lock.LockTracer, java.lang.Object) @bci=148, line=411 (Compiled frame)
 - org.neo4j.kernel.impl.locking.community.LockManagerImpl.getWriteLock(org.neo4j.storageengine.api.lock.LockTracer, org.neo4j.kernel.impl.locking.community.LockResource, java.lang.Object) @bci=11, line=68 (Compiled frame)
 - org.neo4j.kernel.impl.locking.community.CommunityLockClient.acquireExclusive(org.neo4j.storageengine.api.lock.LockTracer, org.neo4j.storageengine.api.lock.ResourceType, long[]) @bci=90, line=132 (Compiled frame)
 - org.neo4j.kernel.impl.newapi.Operations.acquireExclusiveRelationshipLock(long) @bci=55, line=1247 (Compiled frame)
 - org.neo4j.kernel.impl.newapi.Operations.relationshipSetProperty(long, int, org.neo4j.values.storable.Value) @bci=2, line=635 (Compiled frame)
 - org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext$RelationshipOperations.setProperty(long, int, org.neo4j.values.storable.Value) @bci=11, line=729 (Compiled frame)
 - org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$setProperty$1.apply$mcV$sp() @bci=19, line=307 (Compiled frame)
 - org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$setProperty$1.apply() @bci=1, line=307 (Compiled frame)
 - org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$setProperty$1.apply() @bci=1, line=307 (Compiled frame)
 - org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslationSupport$class.translateException(org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslationSupport, scala.Function0) @bci=1, line=33 (Compiled frame)
 - org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext.translateException(scala.Function0) @bci=2, line=42 (Compiled frame)
 - org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations.setProperty(long, int, org.neo4j.values.storable.Value) @bci=16, line=307 (Compiled frame)
 - org.neo4j.cypher.internal.runtime.interpreted.UpdateCountingQueryContext$CountingOps.setProperty(long, int, org.neo4j.values.storable.Value) @bci=28, line=193 (Compiled frame)
 - org.neo4j.cypher.internal.runtime.interpreted.pipes.AbstractSetPropertyOperation.setProperty(org.neo4j.cypher.internal.runtime.interpreted.ExecutionContext, org.neo4j.cypher.internal.runtime.interpreted.pipes.QueryState, org.neo4j.cypher.internal.runtime.Operations, long, org.neo4j.cypher.internal.runtime.interpreted.pipes.LazyPropertyKey, org.neo4j.cypher.internal.runtime.interpreted.commands.expressions.Expression) @bci=124, line=101 (Compiled frame)
 - org.neo4j.cypher.internal.runtime.interpreted.pipes.SetEntityPropertyOperation.set(org.neo4j.cypher.internal.runtime.interpreted.ExecutionContext, org.neo4j.cypher.internal.runtime.interpreted.pipes.QueryState) @bci=99, line=119 (Compiled frame)

[andrew_bowman]

You'll need to examine your queries, especially heavy loads.

Locks are taken by transactions when they write to the graph, and released when the transaction is committed to the graph. If you have multiple queries attempting to write too much data at once (especially relationships), you might have a lock contention issue.

So look at your queries, consider if you need to add batching to break down the size of the transactional changes per transaction, make sure your queries are effectively using indexes (long-running MATCH and MERGE operations due to lack of indexes causes the query to run far longer than it needs to, and can cause it to hold onto locks for longer than necessary), and pay special attention to your loading/processing operations. If you need additional help, we'll need to actually see the queries and have a better idea of what your graph and modeling look like.