Can't start database - ChecksumMismatchException

[adi]

Hi

I am using the Community Edition 4.2.6 using the official docker image. My server ran out of space yesterday which may have caused a database inconsistency. debug.log gives me the following:

2021-07-01 11:07:35.615+0000 INFO  [o.n.d.d.DefaultDatabaseManager] Creating 'DatabaseId{c595bc8c[neo4j]}'.
2021-07-01 11:07:35.615+0000 INFO  [o.n.d.d.DefaultDatabaseManager] Starting 'DatabaseId{c595bc8c[neo4j]}'.
2021-07-01 11:07:35.620+0000 INFO  [o.n.k.i.s.f.RecordFormatSelector] [neo4j/c595bc8c] Selected RecordFormat:StandardV4_0[SF4.0.0] record format from store /data/databases/neo4j
2021-07-01 11:07:35.621+0000 INFO  [o.n.k.i.s.f.RecordFormatSelector] [neo4j/c595bc8c] Selected format 'RecordFormat:StandardV4_0[SF4.0.0]' for existing store /data/databases/neo4j with forma
t 'RecordFormat:StandardV4_0[SF4.0.0]'
2021-07-01 11:07:35.630+0000 INFO  [o.n.k.i.t.l.f.c.CheckpointLogFile] [neo4j/c595bc8c] Scanning log file with version 0 for checkpoint entries
2021-07-01 11:07:35.641+0000 WARN  [o.n.k.d.Database] [neo4j/c595bc8c] Exception occurred while starting the database. Trying to stop already started components.
java.lang.RuntimeException: java.io.IOException: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of 0. At position LogPosition{logVersion=
0, byteOffset=85888} and entry version 3
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.DetachedLogTailScanner.findLogTail(DetachedLogTailScanner.java:101) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.AbstractLogTailScanner.getTailInformation(AbstractLogTailScanner.java:241) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.getTailInformation(CheckpointLogFile.java:185) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CompositeCheckpointLogFile.getTailInformation(CompositeCheckpointLogFile.java:105) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.TransactionLogFiles.getTailInformation(TransactionLogFiles.java:103) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:410) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.checkStoreId(Database.java:571) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.validateStoreAndTxLogs(Database.java:534) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.start(Database.java:419) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:152) ~[neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.DefaultDatabaseManager.initialiseDefaultDatabase(DefaultDatabaseManager.java:64) ~[neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.DefaultDatabaseInitializer.start0(DefaultDatabaseInitializer.java:39) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.SafeLifecycle.transition(SafeLifecycle.java:124) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.SafeLifecycle.start(SafeLifecycle.java:138) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:163) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:134) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:90) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35) [neo4j-4.2.6.jar:4.2.6]
Caused by: java.io.IOException: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of 0. At position LogPosition{logVersion=0, byteOffset=858
88} and entry version 3
	at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:123) ~[neo4j-wal-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:53) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.findLatestCheckpoint(CheckpointLogFile.java:114) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.DetachedLogTailScanner.findLogTail(DetachedLogTailScanner.java:68) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	... 22 more
Caused by: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of 0. At position LogPosition{logVersion=0, byteOffset=85888} and entry version
 3
	at org.neo4j.io.fs.ReadAheadChannel.endChecksumAndValidate(ReadAheadChannel.java:159) ~[neo4j-io-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.entry.CheckpointParserSetV4_2$1.parse(CheckpointParserSetV4_2.java:55) ~[neo4j-wal-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:112) ~[neo4j-wal-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:53) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.findLatestCheckpoint(CheckpointLogFile.java:114) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.DetachedLogTailScanner.findLogTail(DetachedLogTailScanner.java:68) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	... 22 more
2021-07-01 11:07:35.644+0000 ERROR [o.n.d.d.DefaultDatabaseManager] Failed to start DatabaseId{c595bc8c[neo4j]}
org.neo4j.dbms.api.DatabaseManagementException: An error occurred! Unable to start `DatabaseId{c595bc8c[neo4j]}`.
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:191) ~[neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:152) ~[neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.DefaultDatabaseManager.initialiseDefaultDatabase(DefaultDatabaseManager.java:64) ~[neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.DefaultDatabaseInitializer.start0(DefaultDatabaseInitializer.java:39) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.SafeLifecycle.transition(SafeLifecycle.java:124) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.SafeLifecycle.start(SafeLifecycle.java:138) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) [neo4j-common-4.2.6.jar:4.2.6]
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:163) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:134) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:90) [neo4j-4.2.6.jar:4.2.6]
	at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35) [neo4j-4.2.6.jar:4.2.6]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of
 0. At position LogPosition{logVersion=0, byteOffset=85888} and entry version 3
	at org.neo4j.kernel.database.Database.handleStartupFailure(Database.java:564) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.start(Database.java:525) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.6.jar:4.2.6]
	... 13 more
Caused by: java.lang.RuntimeException: java.io.IOException: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of 0. At position LogPosition{
logVersion=0, byteOffset=85888} and entry version 3
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.DetachedLogTailScanner.findLogTail(DetachedLogTailScanner.java:101) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.AbstractLogTailScanner.getTailInformation(AbstractLogTailScanner.java:241) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.getTailInformation(CheckpointLogFile.java:185) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CompositeCheckpointLogFile.getTailInformation(CompositeCheckpointLogFile.java:105) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.TransactionLogFiles.getTailInformation(TransactionLogFiles.java:103) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:410) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.checkStoreId(Database.java:571) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.validateStoreAndTxLogs(Database.java:534) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.start(Database.java:419) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.6.jar:4.2.6]
	... 13 more
Caused by: java.io.IOException: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of 0. At position LogPosition{logVersion=0, byteOffset=858
88} and entry version 3
	at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:123) ~[neo4j-wal-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:53) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.findLatestCheckpoint(CheckpointLogFile.java:114) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.DetachedLogTailScanner.findLogTail(DetachedLogTailScanner.java:68) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.AbstractLogTailScanner.getTailInformation(AbstractLogTailScanner.java:241) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.getTailInformation(CheckpointLogFile.java:185) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CompositeCheckpointLogFile.getTailInformation(CompositeCheckpointLogFile.java:105) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.TransactionLogFiles.getTailInformation(TransactionLogFiles.java:103) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:410) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.checkStoreId(Database.java:571) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.validateStoreAndTxLogs(Database.java:534) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.start(Database.java:419) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.6.jar:4.2.6]
	... 13 more
Caused by: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of 0. At position LogPosition{logVersion=0, byteOffset=85888} and entry version
 3
	at org.neo4j.io.fs.ReadAheadChannel.endChecksumAndValidate(ReadAheadChannel.java:159) ~[neo4j-io-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.entry.CheckpointParserSetV4_2$1.parse(CheckpointParserSetV4_2.java:55) ~[neo4j-wal-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:112) ~[neo4j-wal-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:53) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.findLatestCheckpoint(CheckpointLogFile.java:114) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.DetachedLogTailScanner.findLogTail(DetachedLogTailScanner.java:68) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.AbstractLogTailScanner.getTailInformation(AbstractLogTailScanner.java:241) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CheckpointLogFile.getTailInformation(CheckpointLogFile.java:185) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.checkpoint.CompositeCheckpointLogFile.getTailInformation(CompositeCheckpointLogFile.java:105) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.impl.transaction.log.files.TransactionLogFiles.getTailInformation(TransactionLogFiles.java:103) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.recovery.Recovery.validateStoreId(Recovery.java:410) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.checkStoreId(Database.java:571) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.validateStoreAndTxLogs(Database.java:534) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.kernel.database.Database.start(Database.java:419) ~[neo4j-kernel-4.2.6.jar:4.2.6]
	at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.6.jar:4.2.6]
	... 13 more

I get a similar error when I run neo4j-admin check-consistency:

bin/neo4j-admin check-consistency --database=neo4j
Failure when checking for recovery state: java.io.IOException: org.neo4j.io.fs.ChecksumMismatchException: The checksum 1590744313 did not match the stored checksum of 0. At position LogPosition{logVersion=0, byteOffset=85888} and entry version 3

Any suggestions for how I can fix this?

[dana_canzano · Accepted solution]

@adi
try configuring unsupported.dbms.tx_log.fail_on_corrupted_log_files=false as part of neo4j.conf or as in docker/neo4j syntax and restart Neo4j.
This will basically parse out any corrupted transactions from the transaction log as a result of the disk full

[adi]

Thanks! - That fixed it.

[ebrucucen]

I am using neo4j v 4.3.2 and had the same problem with the same error message, but the configuration did not help after a restart. Any other suggestions?

[raymond_james]

Anything on this? I can't start the database.