Read queries are working, any create/update hangs

[terryfranklin82]

Our Neo4J database has recently started exhibiting strange behaviour - we are able to read any amount of data in the usual way, e.g

match (d:Destination)
where d.vendorId.....
return d

but any sort of node/relationship create or update, no matter how complex or simple, seems to hang indefinitely:

match (d:Destination)
where d.vendorId = .....
set d.name = ....

Searching online has lead me to think there may be some sort of transaction / locking issue happening, but I can't get any further details using the logical apoc functions:

call apoc.monitor.tx();
Failed to invoke procedure `apoc.monitor.tx`: Caused by: javax.management.InstanceNotFoundException: org.neo4j:instance=kernel#0,name=Transactions

call apoc.monitor.locks();
Failed to invoke procedure `apoc.monitor.locks`: Caused by: javax.management.InstanceNotFoundException: org.neo4j:instance=kernel#0,name=Locking

Nothing shows up in the logs when this happens, though restarting the server usually yields a number of transaction-related entries in the shutdown log (the longer we leave the hanging query running for, the more entries are generated on restart):

2019-01-14 00:58:06.381+0000 WARN [c.g.t.e.b.IterableInputBatchTransactionExecutor] Rolled back transaction for batch number 248
2019-01-14 00:58:06.381+0000 WARN [c.g.t.e.b.IterableInputBatchTransactionExecutor] Throwing away the head of the queue as the transaction seems to have failed before polling...
2019-01-14 00:58:06.382+0000 WARN [c.g.t.e.s.KeepCalmAndCarryOn] An exception occurred while executing transaction This database is shutdown.
org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
at org.neo4j.kernel.impl.coreapi.CoreAPIAvailabilityGuard.assertDatabaseAvailable(CoreAPIAvailabilityGuard.java:55)
	at org.neo4j.kernel.impl.factory.ClassicCoreSPI.beginTransaction(ClassicCoreSPI.java:194)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:698)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:382)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:370)
.....
2019-01-14 00:58:06.382+0000 WARN [c.g.t.e.b.IterableInputBatchTransactionExecutor] Rolled back transaction for batch number 249
2019-01-14 00:58:06.382+0000 WARN [c.g.t.e.b.IterableInputBatchTransactionExecutor] Throwing away the head of the queue as the transaction seems to have failed before polling...
2019-01-14 00:58:06.382+0000 WARN [c.g.t.e.s.KeepCalmAndCarryOn] An exception occurred while executing transaction This database is shutdown.
......

Then when the server starts up again:

2019-01-14 01:02:23.887+0000 INFO [o.n.k.i.a.i.IndexingService] IndexingService.init: indexes not specifically mentioned above are ONLINE
2019-01-14 01:02:23.896+0000 INFO [o.n.k.NeoStoreDataSource] Commits found after last check point (which is at LogPosition{logVersion=323, byteOffset=47052835}). First txId after last checkpoint: 291548
2019-01-14 01:02:23.896+0000 INFO [o.n.k.NeoStoreDataSource] Recovery required from position LogPosition{logVersion=323, byteOffset=47052835}
2019-01-14 01:02:24.481+0000 INFO [o.n.k.r.Recovery]   10% completed
2019-01-14 01:02:24.551+0000 INFO [o.n.k.r.Recovery]   20% completed
2019-01-14 01:02:24.623+0000 INFO [o.n.k.r.Recovery]   30% completed
....

Any advice or pointers appreciated, thanks.

[andrew_bowman]

You can use :queries in the browser to see what queries are running.

Here's the link to the transaction management section in the docs which provides additional options for inspecting executing transactions and adding transaction timeouts if needed.

Note that this is for Neo4j Enterprise.

Can you also let us know your Neo4j version and if this is a clustering deployment.

[dmiranda2791]

@terryfranklin82, I am facing the same problem. Were you able to find a solution?

[doug]

Facing the same problem here too. Just randomly over the last day or so the DB started doing this exact same thing. Neo4j Server version: 4.2.1 (community).

Any solutions please?

[doug]

Found the issue. It was just a nested set of sessions and transactions with the same variable names. Should have spotted it, but they were nested in functions.