2021-03-12 02:47:50.956+0530 ERROR [o.n.k.i.DatabaseHealth] Database panic: The database has encountered a critical error, and needs to be restarted.. No space left on device java.io.IOException: No space left on device

[akshat_mittal]

Hi Everyone,

I am getting below error in debug.log.

2021-03-12 02:47:50.956+0530 ERROR [o.n.k.i.DatabaseHealth] Database panic: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details. No space left on device
java.io.IOException: No space left on device

Please let me know why this error is coming and what can be done to prevent this error in coming future?

Regards
Akshat

[abk]

Hi @akshat.mittal ,

Could you share some details about the DBMS installation? Are you running in Docker, using Desktop, directly using the server distribution?

And of course, which platform: Windows, Linux, MacOS?

Running out of disk space could happen because either the database itself has gotten too big, or because log files need to be rotated more often.

-ABK

[akshat_mittal]

Hi Andreas ,

Greetings !!!

I hope you are doing good !!!

Neo4j version 3.5.4 installed on Linux Server using tarball method.

Regards
Akshat

[akshat_mittal]

Hi Andreas ,

Do let me know if you need logs ?

Regards
Akshat

[abk]

Hello again @akshat.mittal ,

The logs will show us the problem of what wants more disk space, but not what has taken up all the disk space.

Could you take a look at disk space generally, then also specifically within the Neo4j DBMS folder?

This is a useful general reference for diagnosing disk space utilization:

Opensource.com

How to check free disk space in Linux

disk-space.jpg opensource.com Keep track of disk utilization with this handy list of commands.

Within the Neo4j DBMS directory, try running:

du -sh *

Best,
ABK

[akshat_mittal]

Hi Andreas ,

Thanks for sharing the link and details.

I have gone through the link and below are stats -

[sineoj@server1 DataLoad]$ du -sh /datasan01/neo4japp/neo4j-enterprise-3.5.3/data/databases/graph.db/
215G /datasan01/neo4japp/neo4j-enterprise-3.5.3/data/databases/graph.db/

[sineoj@server DataLoad]$ ls -lrth /datasan01/neo4japp/neo4j-enterprise-3.5.3/data/databases/graph.db/neostore.transaction.db.* | wc -l
576

There are log of transaction files getting accumulated in graph.db directory.

Regards
Akshat

[abk]

OK, that looks normal. It is worth setting up system monitoring to look for trends that may run into problems.

You could start with a tool like Neo4j Labs' Halin: Halin - Neo4j Monitoring Tool - Neo4j Labs

For production use, you could review the recommendations here:

Neo4j Graph Database Platform

Monitoring - Operations Manual

This chapter describes the tools that are available for monitoring Neo4j.

Best,
ABK

[akshat_mittal]

Hi Andreas ,

I am aware about this Halin tool and it's very good.
But setting dbms.tx_log.rotation.retention_policy=100M size is also not deleting the file.

Regards
Akshat

[abk]

The file won't be deleted. The retention policy describes how much of the file to keep.

There's a good article here: Checkpointing and Log Pruning interactions - Knowledge Base

Best,
ABK

[akshat_mittal]

Hi Andreas ,

Thanks for sharing the link.

So is there any parameter need to set for automatically transaction files deletion ?

Regards
AKshat