Neo.DatabaseError.Transaction.TransactionStartFailed: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details

[yeshveer_yadav]

Debug logs:

2019-11-13 11:42:36.585+0000 ERROR [o.n.b.v.r.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.Transaction.TransactionStartFailed]: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details., reference 7b77cded-c64a-4964-87ac-2460aeca77a2.
2019-11-13 11:42:36.585+0000 ERROR [o.n.b.v.r.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.Transaction.TransactionStartFailed]: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details., reference 7b77cded-c64a-4964-87ac-2460aeca77a2. The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.
org.neo4j.graphdb.TransactionFailureException: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.
at org.neo4j.graphdb.facade.spi.ClassicCoreSPI.beginTransaction(ClassicCoreSPI.java:193)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:691)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:375)
at org.neo4j.bolt.v1.runtime.TransactionStateMachineV1SPI.beginTransaction(TransactionStateMachineV1SPI.java:139)
at org.neo4j.bolt.v1.runtime.TransactionStateMachineV1SPI.beginTransaction(TransactionStateMachineV1SPI.java:98)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine$State$1.execute(TransactionStateMachine.java:293)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine$State$1.run(TransactionStateMachine.java:269)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine.run(TransactionStateMachine.java:110)
at org.neo4j.bolt.v3.runtime.ReadyState.processRunMessage(ReadyState.java:85)
at org.neo4j.bolt.v3.runtime.ReadyState.processUnsafe(ReadyState.java:56)
at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48)
at org.neo4j.bolt.v1.runtime.BoltStateMachineV1.nextState(BoltStateMachineV1.java:144)
at org.neo4j.bolt.v1.runtime.BoltStateMachineV1.process(BoltStateMachineV1.java:92)
at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:89)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:191)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:139)
at org.neo4j.bolt.runtime.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:171)
at org.neo4j.bolt.runtime.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$2(ExecutorBoltScheduler.java:154)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.neo4j.internal.kernel.api.exceptions.TransactionFailureException: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.
at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.neo4j.kernel.internal.DatabaseHealth.assertHealthy(DatabaseHealth.java:61)
at org.neo4j.kernel.impl.api.KernelImpl.beginTransaction(KernelImpl.java:88)
at org.neo4j.graphdb.facade.spi.ClassicCoreSPI.beginTransaction(ClassicCoreSPI.java:186)
... 21 more
Caused by: org.neo4j.kernel.impl.store.UnderlyingStorageException: Error performing check point
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.constructCombinedFailure(CheckPointScheduler.java:101)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:82)
at org.neo4j.kernel.impl.scheduler.ScheduledJobHandle.lambda$new$0(ScheduledJobHandle.java:83)
at org.neo4j.kernel.impl.scheduler.ThreadPool.lambda$submit$0(ThreadPool.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646092, actual version: 1139426686, wait timeout (ms): 600008
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646092, actual version: 1139426686, wait timeout (ms): 600033
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646092, actual version: 1139426686, wait timeout (ms): 600025
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646092, actual version: 1139426686, wait timeout (ms): 600091
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646095, actual version: 1139426686, wait timeout (ms): 600041
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646096, actual version: 1139426686, wait timeout (ms): 600023
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646096, actual version: 1139426686, wait timeout (ms): 600025
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646096, actual version: 1139426686, wait timeout (ms): 600092
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646096, actual version: 1139426686, wait timeout (ms): 600072
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more
Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1139646096, actual version: 1139426686, wait timeout (ms): 600082
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
... 7 more

Why is this error occurring very frequently?
is there problem with "Failed to rotate logs. Expected version: 1139646096, actual version: 1139426686, wait timeout (ms): 600082"

[paulare]

yeshveer.yadav:

Failed to rotate logs.

Hi, Last Friday I had an incident where the neo4j database crashed.
The reason was that there was not enough space for transaction logs. After cleaning space I managed to restart the database.
But now every day, during nightly data load - database crashes again. Log implies that there is an issue with the log rotate.
Can someone give me a bit more insight into what could be a reason for log-rotate to fail? And what could be a remedy?
using neo4j community edition 3.5.3
dbms.tx_log.rotation.retention_policy=1 days
Could the issue be that tx_log is too big to rotate and I should rotate more frequently?

2019-12-17 08:46:56.852+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by scheduler for time threshold @ txId: 323043862 checkpoint started...
2019-12-17 08:56:56.877+0000 ERROR [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint failed Failed to rotate logs. Expected version: 323043862, actual version: 319521295, wait timeout (ms): 600011
org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 323043862, actual version: 319521295, wait timeout (ms): 600011
        at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
        at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
        at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
        at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
        at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
        at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
        at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:436)
        at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:164)
        at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:138)
        at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
        at org.neo4j.kernel.impl.scheduler.ScheduledJobHandle.lambda$new$0(ScheduledJobHandle.java:83)
        at org.neo4j.kernel.impl.scheduler.ThreadPool.lambda$submit$0(ThreadPool.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[paulare]

I believe I found a solution for my case

Things i did to resolve:

• moved neostore.counts.db.a/.b files from graph database folder
• restarted service

Noticed, that in "grap.db" folder last neostore.transaction.db the file is owned by root, not a service account
So i changed file ownership

Also, found that neo4j cant write log

chown -R neo4j:adm /var/log/neo4j/neo4j.log
chown -R neo4j:adm /var/log/neo4j/debug.log

Restarted server.

Hypothesis
After my neo4j database crashed in the first time, I used

sudo systemctl start neo4j

to restart service. I suspect, in this process some file ownership was changed (ownership changed from neo4j > root)
Incorrect file ownership caused an issue during transaction log rotation.