Files keep on growing in number and size under neo4j installation folder/databases/graph.db

[akshat_mittal]

Hi Team,

Greetings !!

I have a concern regarding the files keep on groing in number and size under /databases/graph.db/

May I know the importance of these files ?
Can we delete these files as they piled up and taking about 500 GB on my server ?
What will be the impact if we delete these files ?

Files are given below & all files are attached in topic.Neo4j_files_graphdb.txt (114.2 KB)
-rw-r--r-- 1 root root 1.3G Feb 28 07:55 neostore.transaction.db.30899
-rw-r--r-- 1 root root 1.3G Mar 1 07:58 neostore.transaction.db.31048
-rw-r--r-- 1 root root 1.3G Feb 20 07:47 neostore.transaction.db.29785
-rw-r--r-- 1 root root 1.3G Feb 24 04:34 neostore.transaction.db.30361
-rw-r--r-- 1 root root 1.3G Feb 25 08:00 neostore.transaction.db.30509
-rw-r--r-- 1 root root 1.3G Feb 22 07:52 neostore.transaction.db.30094
-rwxrwxrwx 1 neo4j neo4j 1.8G Mar 1 07:58 neostore.nodestore.db
-rwxrwxrwx 1 neo4j neo4j 2.2G Mar 1 07:58 neostore.relationshipstore.db
-rwxrwxrwx 1 neo4j neo4j 20G Mar 1 07:58 neostore.propertystore.db.strings
-rwxrwxrwx 1 neo4j neo4j 37G Mar 1 07:58 neostore.propertystore.db

Regards
Akshat

[dana_canzano]

1. what version of Neo4j is in play here
2. no you should not be deleting files under graph.db and outside of the products normal operation
3. if your question is about the 1.G neostore.transactin.db* file ???? is this your question ??? or are you concerned about the other files as well???? if it is about neostore.transaction.db then see Transaction logs - Operations Manual

[akshat_mittal]

Hello Dana,

Thank you so much for taking up the query !!

1. Neo4j Version is 3.5.3
2. Ok .. But my these files has eaten up 465 GB. So how to come out from this situation??
3. I am mainly concerned about the neostore.transaction.db.* files as they are huge in number ( 1300 ). Can i delete those files to release the space ??
4. What about these files given below - ( Importance of these files and can we delete these files are well ?)
   -rwxrwxrwx 1 neo4j neo4j 2.2G Mar 1 07:58 neostore.relationshipstore.db
   -rwxrwxrwx 1 neo4j neo4j 20G Mar 1 07:58 neostore.propertystore.db.strings
   -rwxrwxrwx 1 neo4j neo4j 37G Mar 1 07:58 neostore.propertystore.db

Reagrds
Akshat

[dana_canzano]

1. I am mainly concerned about the neostore.transaction.db.* files as they are huge in number ( 1300 ). Can i delete those files to release the space ??

as previously stated

2. no you should not be deleting files under graph.db and outside of the products normal operation

and as to how to manage the size/rotation/retention and as previously stated please see

if it is about neostore.transaction.db then see Transaction logs - Operations Manual

[dana_canzano]

akshat.mittal:

What about these files given below - ( Importance of these files and can we delete these files are well ?)
-rwxrwxrwx 1 neo4j neo4j 2.2G Mar 1 07:58 neostore.relationshipstore.db
-rwxrwxrwx 1 neo4j neo4j 20G Mar 1 07:58 neostore.propertystore.db.strings
-rwxrwxrwx 1 neo4j neo4j 37G Mar 1 07:58 neostore.propertystore.db

1. What about these files given below - ( Importance of these files and can we delete these files are well ?)
   -rwxrwxrwx 1 neo4j neo4j 2.2G Mar 1 07:58 neostore.relationshipstore.db
   -rwxrwxrwx 1 neo4j neo4j 20G Mar 1 07:58 neostore.propertystore.db.strings
   -rwxrwxrwx 1 neo4j neo4j 37G Mar 1 07:58 neostore.propertystore.db

the files in order represent

• all the node relationship details
• all string properties
• all other properties

if you were to manually delete said files ( and no I'm not recommending manually deleteing) but if you were then you would lose all node relationships, all properties which are strings, all other properties... which would thus leave the graph fairly useless

[akshat_mittal]

Hello Dana ,

Apologies for my late reply !!

I tried to set this parameter in my conf file and tried to start neo4j process.
dbms.tx_log.rotation.retention_policy=100M size

Process got started but it has not removed the transaction files from graph.db folder.

Can u pls suggest more ??

Regards
Akshat

[dana_canzano]

@akshat.mittal
Transaction rotation occurs at checkpoint time. Have you experienced a checkpoint and not yet had transaction logs rotated

[akshat_mittal]

Hello Dana ,

Actually , I am unable to understand the things here . My bad.

Can i share my logs and check whether checkpoint is reached or not ??

Regards
Akshat

[dana_canzano]

grep -i check debug.log | grep -i point 

will report when it last checkpointed

[akshat_mittal]

Hello Dana,

Thanks for quick help.

I have ran those commands in my system.
I have attached the output of the command.

Regards
Akshat

debug.txt (56.8 KB)

[akshat_mittal]

Hi Dana,

I found the below entry with check point in logs -

[root@JMNGD1BCE110V01 logs]# grep -i point debug.log
2021-03-02 17:03:29.758+0530 INFO [o.n.i.d.DiagnosticsManager] sun.java.command = com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/localdata/neo4j-enterprise-3.5.3 --config-dir=/localdata/neo4j-enterprise-3.5.3/conf
at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)
2021-03-02 17:09:28.211+0530 INFO [o.n.i.d.DiagnosticsManager] sun.java.command = com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/localdata/neo4j-enterprise-3.5.3 --config-dir=/localdata/neo4j-enterprise-3.5.3/conf
at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)
2021-03-02 17:19:40.738+0530 INFO [o.n.k.NeoStoreDataSource] Commits found after last check point (which is at LogPosition{logVersion=30509, byteOffset=129969818}). First txId after last checkpoint: 11302639
at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)

Regards
Akshat

[dana_canzano]

Provided this is Neo4j 3.5.3? correct? ????

checkpoint will be logged in the logs\debug.log similar to

2021-03-13 13:08:07.536+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by database shutdown @ txId: 4 checkpoint started...
2021-03-13 13:08:07.552+0000 INFO [o.n.k.i.s.c.CountsTracker] Rotated counts store at transaction 4 to [/home/neo4j/cluster/instance1/neo4j-enterprise-3.5.3/data/databases/graph.db/neostore.counts.db.b], from [/home/neo4j/cluster/instance1/neo4j-enterprise-3.5.3/data/databases/graph.db/neostore.counts.db.a].
2021-03-13 13:08:07.588+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by database shutdown @ txId: 4 checkpoint completed in 52ms

not the 1st line that reports Checkpoint triggered by database shutdown and specifically the by database shutdown may be different since checkpoint can happen for various reasons.

So a better and more precise grep is

grep 'Checkpoint triggered' debug.log

which thus would give output similar to

2021-03-13 13:08:07.536+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by database shutdown @ txId: 4 checkpoint started...
2021-03-13 13:08:07.588+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by database shutdown @ txId: 4 checkpoint completed in 52ms

are you not seeing these same lines?

[akshat_mittal]

Hello Dana,

Greetings !!

I can see only 1 line with given key word. Below is the output.

[root@JMNGD1BCE110V01 logs]# cat debug.log | grep 'Checkpoint triggered'
2021-03-03 08:40:36.577+0530 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by database shutdown @ txId: 11456686 checkpoint started...

[root@JMNGD1BCE110V01 logs]# ls -lrth
total 43M
-rw-r--r-- 1 root root 21M Mar 2 06:41 query.log.1
-rw-r--r-- 1 root root 0 Mar 2 16:41 query.log.3
-rw-r--r-- 1 root root 0 Mar 2 16:41 query.log.2
-rwxrwxrwx 1 neo4j neo4j 17K Mar 12 19:03 neo4j.log
-rwxrwxrwx 1 neo4j neo4j 1.9M Mar 12 19:03 security.log
-rw-r--r-- 1 root root 20M Mar 12 19:06 query.log
-rw-r--r-- 1 root root 1.2M Mar 16 17:34 debug.log
[root@JMNGD1BCE110V01 logs]# cat debug.log | grep 'Checkpoint triggered' | wc -l
1

Regards
Akshat

[dana_canzano]

ok.. from your output we see what would appear is the last time checkpoint ran was on March 3, 2021 @08:40:36 and clearly it should be running more frequently.

From this issue you are running Neo4j 3.5.3. Is there a specific requirement for 3.5.3? 3.5.3 was released 11 February 2019 ( Neo4j 3.5.3 - Neo4j Graph Database Platform ) abd so is a bit old. Could you upgrade to our most current 3.5.x release, namely 3.5.26 and see if the issue is resolved or continues to persist?

[akshat_mittal]

Hello Dana,

Right now we are using 3.5.3 and it is running on Production setup and it is not possible for me to upgrade it to 3.5.26.
Can you please help me out in sharing the parameters i need to set up in neo4j.conf file so that transaction files accumulation can be cleared without impacting the system.
This will help me in getting the extra space under graph.db and unwanted files will be removed.

Regards
Akshat

[dana_canzano]

the expected default behavior is checkpoint is to occur every 15 minutes or every 100k transactions and which ever is earlier. So there should be nothing special you need to configure.

However the reason for the request to upgrade, of which the version you are running was released 2 years ago, is that in 3.5.5 we fixed

github.com

neo4j/neo4j - 3.5.5

Neo4j-3.5-changelog

Graphs for Everyone. Contribute to neo4j/neo4j development by creating an account on GitHub.

Kernel
 *  Fix delay calculation in IO limiter

which addresses an issue where checkpoint is known not to run under certain circumstances.
And so if you were to upgrade to 3.5.5 or later, and whereby 3.5.26 is the most recent release we would hopefully expect this to go away

[akshat_mittal]

Hi Dana,

Ok I will try to updgrade the system to atleast 3.5.6.

Please let me know what i need to configure in neo4j.conf file so that transaction files can be moved / removed automatically w/o impacting the DB.

Below are the parameters configured in my conf file.

[sineoj@jmngdprv009693 conf] cat neo4j.conf | grep -v '^#' | grep -v '^'

dbms.directories.import=import

dbms.memory.heap.max_size=30g

dbms.memory.pagecache.size=10g

dbms.connectors.default_listen_address=0.0.0.0

dbms.connector.bolt.enabled=true

dbms.connector.http.enabled=true

dbms.connector.http.listen_address=:7474

dbms.connector.https.enabled=true

dbms.connector.https.listen_address=:7473

apoc.import.file.use_neo4j_config=false

dbms.jvm.additional=-XX:+UseG1GC

dbms.jvm.additional=-XX:-OmitStackTraceInFastThrow

dbms.jvm.additional=-XX:+AlwaysPreTouch

dbms.jvm.additional=-XX:+UnlockExperimentalVMOptions

dbms.jvm.additional=-XX:+TrustFinalNonStaticFields

dbms.jvm.additional=-XX:+DisableExplicitGC

dbms.jvm.additional=-Djdk.tls.ephemeralDHKeySize=2048

dbms.jvm.additional=-Djdk.tls.rejectClientInitiatedRenegotiation=true

dbms.windows_service_name=neo4j

dbms.transaction.timeout=420m

dbms.jvm.additional=-Dunsupported.dbms.udc.source=tarball

dbms.db.timezone=SYSTEM

db.temporal.timezone=+05:30

dbms.memory.heap.initial_size=27000m

apoc.export.file.enabled=true

[sineoj@jmngdprv009693 conf]$ pwd

/datasan01/neo4japp/neo4j-enterprise-3.5.3/conf

[sineoj@jmngdprv009693 conf]$ /datasan01/neo4japp/neo4j-enterprise-3.5.3/bin/neo4j status

Neo4j is running at pid 166479

[sineoj@jmngdprv009693 conf]$ /datasan01/neo4japp/neo4j-enterprise-3.5.3/bin/neo4j version

neo4j 3.5.3

[sineoj@jmngdprv009693 databases]$ du -sh graph.db/

119G graph.db/

-rw-rw-r-- 1 sineoj neo4j 251M Mar 16 20:53 neostore.transaction.db.5781

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 04:52 neostore.transaction.db.5782

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 04:56 neostore.transaction.db.5783

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 04:57 neostore.transaction.db.5784

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 04:58 neostore.transaction.db.5785

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 05:00 neostore.transaction.db.5786

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 05:31 neostore.transaction.db.5787

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 05:37 neostore.transaction.db.5788

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 06:31 neostore.transaction.db.5789

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 06:31 neostore.transaction.db.5790

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 06:32 neostore.transaction.db.5791

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 06:32 neostore.transaction.db.5792

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 06:32 neostore.transaction.db.5793

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 11:55 neostore.transaction.db.5794

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 11:56 neostore.transaction.db.5795

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 11:57 neostore.transaction.db.5796

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 11:57 neostore.transaction.db.5797

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 12:22 neostore.transaction.db.5798

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 12:27 neostore.transaction.db.5799

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 19:46 neostore.transaction.db.5800

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 19:56 neostore.transaction.db.5801

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 19:58 neostore.transaction.db.5802

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 19:59 neostore.transaction.db.5803

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 20:00 neostore.transaction.db.5804

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 20:01 neostore.transaction.db.5805

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 20:21 neostore.transaction.db.5806

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 20:24 neostore.transaction.db.5807

-rw-rw-r-- 1 sineoj neo4j 251M Mar 17 23:53 neostore.transaction.db.5808

-rw-rw-r-- 1 sineoj neo4j 251M Mar 18 04:54 neostore.transaction.db.5809

-rw-rw-r-- 1 sineoj neo4j 251M Mar 18 04:55 neostore.transaction.db.5810

-rw-rw-r-- 1 sineoj neo4j 251M Mar 18 04:56 neostore.transaction.db.5811

-rw-rw-r-- 1 sineoj neo4j 251M Mar 18 04:58 neostore.transaction.db.5812

-rw-rw-r-- 1 sineoj neo4j 251M Mar 18 05:21 neostore.transaction.db.5813

-rw-rw-r-- 1 sineoj neo4j 251M Mar 18 05:24 neostore.transaction.db.5814

-rw-rw-r-- 1 sineoj neo4j 202M Mar 18 07:31 neostore.transaction.db.5815

[sineoj@jmngdprv009693 graph.db]$ ls -lrth neostore.transaction.db.* | wc -l

151

Regards
Akshat

[dana_canzano]

@akshat.mittal
As previously stated

the expected default behavior is checkpoint is to occur every 15 minutes or every 100k transactions and which ever is earlier. So there should be nothing special you need to configure.

if anything you could further deviate from the defaults by changing parameters in neo4j.conf that start with dbms.checkpoint* and as defined at Configuration settings - Operations Manual but again as your current settings have not overriden the defaults you should thus be subject to the defaults of every a checkpoint every 15minutes or every 100k transactions.

We do not see this issue at any other customer implementation in the field. Yes we did experience it a few times and for which this is the reason 3.5.6 was created and so as to address a bug.
So without upgrade I'm not sure of anything else you can do