Neo4j 3.5.5 server goes down

[jfacow]

Hello
Neo4j 3.5.5 server goes down when we find a lot of repeatable messages in debug.log
WARN [o.n.k.i.a.t.m.KernelTransactionMonitor] Transaction KernelTransactionImplementationHandle{txReuseCount=48, tx=KernelTransaction[21]} timeout.

Is there any way to find more clear reason?..

[stefan_armbrust]

Pls post the parts of debug.log prior to these messages.

[jfacow]

db down at 01:37:27.977

2019-11-09 01:37:03.577+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by scheduler for continuous threshold @ txId: 58171486 checkpoint completed in 167ms

2019-11-09 01:37:03.577+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] No log version pruned, last checkpoint was made in version 1354

2019-11-09 01:37:03.778+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by scheduler for continuous threshold @ txId: 58171487 checkpoint started...

2019-11-09 01:37:03.889+0000 INFO [o.n.k.i.s.c.CountsTracker] Rotated counts store at transaction 58171487 to [/var/lib/neo4j355/neo4j-enterprise-3.5.5/data/databases/graph.db/neostore.counts.db.a],

from [/var/lib/neo4j355/neo4j-enterprise-3.5.5/data/databases/graph.db/neostore.counts.db.b].

2019-11-09 01:37:03.948+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by scheduler for continuous threshold @ txId: 58171487 checkpoint completed in 170ms

2019-11-09 01:37:03.950+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] No log version pruned, last checkpoint was made in version 1354

2019-11-09 01:37:11.426+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=296, gcTime=0, gcCount=0}

2019-11-09 01:37:18.143+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6192, gcTime=0, gcCount=0}

2019-11-09 01:37:27.977+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=7614, gcTime=0, gcCount=0}

2019-11-09 01:49:55.827+0000 INFO [o.n.g.f.m.e.CommunityEditionModule] No locking implementation specified, defaulting to 'forseti'

2019-11-09 01:49:56.180+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Creating 'graph.db' database.

[stefan_armbrust]

One potential source for the long pause times > 1000ms could be lack of heap space. Make sure you do batch all large operations e.g. via apoc.periodic.iterate