Raft logs not being pruned

[demian]

We have a causal cluster with 5 core nodes. DB size ~60GB.

Every couple of days one of the nodes becomes offline with status: Quarantine marker is present, but unable to read.

This seems to be caused by running out of space even though each instance has volumes of 250GB.

After close inspection we see that the problem is with the raft.log files growing and growing and not being pruned as it is supposed to. In this particular node, we can see now over 700 files of 250MB each, from the last 3 weeks, taking more than 175GB of available space.

This is our current (and also default) config for pruning:

"raft_log_entry_prefetch_buffer.max_entries":"1024" 
"raft_log_implementation": "SEGMENTED" 
"raft_log_prune_strategy": "1g size" 
"raft_log_pruning_frequency": "10m" 
"raft_log_reader_pool_size": "8" 
"raft_log_rotation_size": "250.00MiB"

We would appreciate any ideas. Is there a specific reason why these logs are not pruned?
Can we prune them manually without affecting the causal cluster?

Thanks in advance!

[dana_canzano · Accepted solution]

@demian 

thanks for this detail.   4.4.4 was released Feb 2022 https://neo4j.com/release-notes/database/neo4j-4-4-4/ and our current 4.4.x is 4.4.12 https://neo4j.com/release-notes/database/neo4j-4-4-12 

We did fix a similar issue to this in 4.4.8.   

Please upgrade

[dana_canzano]

@demian 

Neo4j version?  4x?  5x?

[demian]

We are running Neo4j v4.4.4. 

[dana_canzano · Accepted solution]

@demian 

thanks for this detail.   4.4.4 was released Feb 2022 https://neo4j.com/release-notes/database/neo4j-4-4-4/ and our current 4.4.x is 4.4.12 https://neo4j.com/release-notes/database/neo4j-4-4-12 

We did fix a similar issue to this in 4.4.8.   

Please upgrade

[demian · Accepted solution]

Thanks for getting back to me.

I see that the only "related" fix in v4.4.8 is this one:

Fix bug where prefetching log entries could get stuck, unable to read new entries from the log.

Can that cause our issue?

We were planning to upgrade to v4.4.12 in the coming days in any case, so will let you know if we still see the issue after that.

Thanks!

[demian]

We have successfully updated to v4.4.12 and so far we haven't seen this issue anymore.

Thanks for the help, will report back if something happens.