In general we do not recommend deleting transaction logs, that can have an unexpected impact on the system. That said in this case we may need to do a manual pruning of a log file.
The important bit is the logVersion and byteOffset in this:
Failed to apply transaction: Transaction #347431025 at log position LogPosition{logVersion=5943, byteOffset=18387457}
The logVersion number signifies that the problem is in neostore.transaction.db.5943. We have the byte offset, which we can use to prune this log file, if needed, though it's supposed to be that versions after 3.4.x would attempt to recover automatically (this is after an out of disk space scenario, which you should check for and resolve first before attempting recovery).
For the following instructions, we are using the logVersion and byteOffset values from the exception message...when troubleshooting your own similar issues, use those values from your own exception message when performing the following:
To truncate, we recommend the following (as mentioned, after resolving any issue of disk space):
- Rename the transaction file in question, so we keep it around in case this recovery attempt fails or further analysis is needed:
mv neostore.transaction.db.5943 copy.neostore.transaction.db.5943
- Perform the truncation using the byteOffset (in this case
18387457), naming the truncated file so it takes the place of the original transaction file:
head -c 18387457 copy.neostore.transaction.db.5943 > neostore.transaction.db.5943
- Attempt startup
We also recommend reviewing your transaction log rotation and retention settings if needed. If less transaction logs are desired please correct that via these settings and restarting, since directly deleting transaction logs isn't recommended and may end up inducing corruption.
