Neo4j as a process taking 45% Memory on Linux Server

[akshat_mittal]

Hello Team ,

Greetings!!

I have observed that neo4j process is taking approx 50% Memory of the server.
Neo4j Version is 3.5.3 & OS = RHEL 7.4

Below are the detailed outputs of command.

[root@jmngdprv009694 ~]# top
top - 10:05:10 up 20 days, 18:25, 1 user, load average: 0.10, 0.09, 0.05
Tasks: 365 total, 2 running, 363 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 0.3 sy, 0.0 ni, 98.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 89956272 total, 563304 free, 40735908 used, 48657064 buff/cache
KiB Swap: 67108860 total, 66876564 free, 232296 used. 48328184 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
63312 root 20 0 213576 98664 184 R 6.0 0.1 0:00.18 OSWatcher.sh
4663 root 20 0 213576 99848 1368 S 4.0 0.1 1123:11 OSWatcher.sh
4875 root rt 0 204128 107908 83184 S 1.3 0.1 326:16.52 corosync
5307 root 20 0 1045168 51328 13492 S 1.3 0.1 168:19.55 oacore
34356 root 20 0 64.4g 37.2g 26904 S 0.3 43.4 167:17.70 java
56167 root 20 0 622484 33840 10212 S 0.3 0.0 13:52.27 filebeat
62389 root 20 0 162296 2572 1608 R 0.3 0.0 0:00.02 top
76481 root 20 0 0 0 0 S 0.3 0.0 0:02.58 kworker/5:1
1 root 20 0 52248 4396 2484 S 0.0 0.0 10:53.94 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.74 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.97 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:00.93 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 4:47.35 rcu_sched
10 root rt 0 0 0 0 S 0.0 0.0 0:02.41 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:02.65 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:41.75 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:01.76 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
16 root rt 0 0 0 0 S 0.0 0.0 0:02.71 watchdog/2
17 root rt 0 0 0 0 S 0.0 0.0 0:16.20 migration/2
18 root 20 0 0 0 0 S 0.0 0.0 0:02.02 ksoftirqd/2
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
21 root rt 0 0 0 0 S 0.0 0.0 0:02.77 watchdog/3
22 root rt 0 0 0 0 S 0.0 0.0 0:10.22 migration/3
23 root 20 0 0 0 0 S 0.0 0.0 0:02.22 ksoftirqd/3
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
26 root rt 0 0 0 0 S 0.0 0.0 0:02.60 watchdog/4
27 root rt 0 0 0 0 S 0.0 0.0 0:04.54 migration/4
28 root 20 0 0 0 0 S 0.0 0.0 0:03.30 ksoftirqd/4
30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H
31 root rt 0 0 0 0 S 0.0 0.0 0:02.57 watchdog/5
32 root rt 0 0 0 0 S 0.0 0.0 0:02.51 migration/5
33 root 20 0 0 0 0 S 0.0 0.0 0:02.03 ksoftirqd/5
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0H
36 root rt 0 0 0 0 S 0.0 0.0 0:02.64 watchdog/6
37 root rt 0 0 0 0 S 0.0 0.0 0:02.62 migration/6
38 root 20 0 0 0 0 S 0.0 0.0 0:00.99 ksoftirqd/6
[root@jmngdprv009694 ~]# top | grep java
34356 root 20 0 64.4g 37.2g 26904 S 0.7 43.4 167:17.73 java
34356 root 20 0 64.4g 37.2g 26904 S 0.3 43.4 167:17.74 java
34356 root 20 0 64.4g 37.2g 26904 S 0.3 43.4 167:17.75 java

[root@jmngdprv009694 ~]# crontab -l
0 * * * * perl /opt/perf/bin/watchhpcs.pl
07 00 * * 0 bash /home/monadm/cleanup_monitoring_logs.sh
00 0-23 * * * /usr/bin/nmon -f -t -s 10 -c 360 -m /monitoring_logs/nmon/log
[root@jmngdprv009694 ~]# ps -ef | grep -i 34356
root 34356 1 0 Jan14 ? 02:47:17 /usr/bin/java -cp /datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins:/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf:/datasan02/neo4japp/neo4j-enterprise-3.5.3/lib/:/datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins/ -server -Xms16384m -Xmx32768m -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dunsupported.dbms.udc.source=tarball -Dfile.encoding=UTF-8 com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3 --config-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf
root 64437 61709 0 10:06 pts/0 00:00:00 grep --color=auto -i 34356
[root@jmngdprv009694 ~]# cd /datasan02/neo4japp/neo4j-enterprise-3.5.3/bin/
[root@jmngdprv009694 bin]# ./neo4j version
neo4j 3.5.3
[root@jmngdprv009694 bin]# free -g
total used free shared buff/cache available
Mem: 85 38 17 0 29 46
Swap: 63 0 63
[root@jmngdprv009694 bin]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

[root@jmngdprv009694 plugins]# ll
total 29500
-rw-rw-r-- 1 jioapp jioapp 13749813 Oct 23 12:50 apoc-3.5.0.4-all.jar
-rw-rw-r-- 1 jioapp jioapp 5970938 Oct 23 12:50 neo4j-jdbc-driver-3.4.0.jar
-rw-rw-r-- 1 jioapp jioapp 2739670 Oct 23 12:50 ojdbc6.jar
-rw-rw-r-- 1 jioapp jioapp 3698857 Oct 23 12:50 ojdbc7.jar
-rw-rw-r-- 1 jioapp jioapp 4036257 Oct 23 12:50 ojdbc8.jar
-rw-r--r-- 1 jioapp jioapp 2217 Feb 7 2019 README.txt

Best Regards
Akshat

[maciej_krzywdaa]

Hi @akshat.mittal,

Check this art: https://neo4j.com/developer/kb/understanding-memory-consumption/ and check logs.

[akshat_mittal]

Hello Maciej,

Thanks for sharing the URL.
Can you please explain in brief and what needs to be done?

There are some errors in logs related to memory.
Exception in thread "pool-3-thread-5" Exception in thread "neo4j.StorageMaintenance-25" java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

at java.util.concurrent.SynchronousQueue$TransferStack.snode(SynchronousQueue.java:318)

at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)

at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Exception in thread "CustomProcedureStorage" java.lang.OutOfMemoryError: Java heap space

2020-01-09 13:16:33.554+0000 WARN Java heap space

java.lang.OutOfMemoryError: Java heap space

2020-01-09 13:16:33.555+0000 WARN Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@2549d4fa in QueuedThreadPool[qtp2071816657]@7b7d69d1{STARTED,12<=12<=24,i=3,q=0}[ReservedThreadExecutor@1037c4ca{s=0/2,p=0}]

2020-01-10 06:27:26.595+0000 WARN The client is unauthorized due to authentication failure.

2020-01-10 06:27:33.457+0000 WARN The client is unauthorized due to authentication failure.

2020-01-10 10:23:03.431+0000 WARN The client is unauthorized due to authentication failure.

Best Regards
Akshat

[dana_canzano]

any reason to use Neo4j 3.5.3 ??? only in that there are newer versions of Neo4j in the 3.5.x series, namely 3.5.14

also, per your initial response the ps output includes

 -Xms16384m -Xmx32768m

setting min/max heap at 16G and 32G respectively. Generally we recommend that they be defined to the same value. But so your heap may grow as large as 32G. these values are defined in the conf/neo4j.conf and paramters

dbms.memory.heap.init_size
dbms.memory.heap.max_size

how large have you configured

dbms.memory.pagecache.size

Your

[root@jmngdprv009694 bin]# free -g
total used free shared buff/cache available
Mem: 85 38 17 0 29 46
Swap: 63 0 63

so it appears you have a total of 85G of RAM in the instance. The heap might consume up to 32G. So heap may consume up to 37% of the total RAM. so 45% is not so far off? but again it depends on how much is assigned to dbms.memory.pagecache.size and even if it is not explictly configured in conf/neo4j.conf we will default to a size based upon how much memory is available. However you can always choose to explicitly configure

[akshat_mittal]

Hello Dana,

Due to some organisation reason, we are using 3.5.3 version.

Please find below details

[root@jmngdprv009694 ~]# ps -ef | grep -i neo4j
root 85903 1 2 Feb04 ? 00:26:42 /usr/bin/java -cp /datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins:/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf:/datasan02/neo4japp/neo4j-enterprise-3.5.3/lib/:/datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins/ -server -Xms16384m -Xmx32768m -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dunsupported.dbms.udc.source=tarball -XX:NativeMemoryTracking=detail -Dfile.encoding=UTF-8 com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3 --config-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf
root 124474 124219 0 09:40 pts/0 00:00:00 grep --color=auto -i neo4j

[root@jmngdprv009694 ~]# top | grep -i 85903
85903 root 20 0 63.1g 37.2g 13576 S 6.2 43.4 26:42.30 java
85903 root 20 0 63.1g 37.2g 13576 S 0.3 43.4 26:42.31 java
85903 root 20 0 63.1g 37.2g 13576 S 0.3 43.4 26:42.32 java
85903 root 20 0 63.1g 37.2g 13576 S 0.3 43.4 26:42.33 java

[root@jmngdprv009694 ~]# cd /datasan02/neo4japp/neo4j-enterprise-3.5.3/conf/
[root@jmngdprv009694 conf]# ls
jmx.access jmx.password neo4j.conf

[root@jmngdprv009694 conf]# cat neo4j.conf | grep -i memory
dbms.memory.heap.initial_size=16384m
dbms.memory.heap.max_size=32768m

The amount of memory to use for mapping the store files, in bytes (or

The default page cache memory assumes the machine is dedicated to running

#dbms.memory.pagecache.size=10g

entirely in memory. The only drawback of this setting is that longer checkpoint times

Make sure that initmemory is not only allocated, but committed to

the process, before starting the database. This reduces memory

Try reducing the heap memory, if this flag degrades performance.

Best Regards
Akshat

[dana_canzano]

ok... so again you state

[root@jmngdprv009694 conf]# cat neo4j.conf | grep -i memory
dbms.memory.heap.initial_size=16384m
dbms.memory.heap.max_size=32768m

so the heap could consume upwards of 32G.

You have not explicitly defined the pagecache, as evidence

#dbms.memory.pagecache.size=10g

however that simply means it will default to a % of remaining available memory at startup.

Still nothing looks unusual. If ALL you want to do is have less 45% of memory usage set

dbms.memory.heap.initial_size=4096m
dbms.memory.heap.max_size=4096m
dbms.memory.pagecache.size=1g

and that will then result in Neo4j generally consuming around 5G of memory which will be less than 45% of the total RAM. But whether or not this makes sense from a performance standpoint ????

Further https://neo4j.com/docs/operations-manual/3.5/performance/memory-configuration/ details these settings and how you would come to choosing the values for said settings.

[akshat_mittal]

Hello Dana,

I think there is a gap from my side in explaining my concern to you.

Let me try again... " Concern is : Java process ( Neo4j process ) taking more than 50% of memory even if it is not loading any data in neo4j. Want to know why this is happening?
Expectation is : It should not take this amount of memory.

I have used the neo4j-admin memrec tool for setting the 3 memory parameters suggested by this tool in my config file. But I can a more usage of memory now with zero processing / loading of data in neo4j.

Below are the updated details:

[root@jmngdprv009694 bin]# ps -ef | grep -i neo4j

root 95772 1 8 09:56 pts/0 00:01:06 /usr/bin/java -cp /datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins:/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf:/datasan02/neo4japp/neo4j-enterprise-3.5.3/lib/:/datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins/ -server -Xms27000m -Xmx27000m -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dunsupported.dbms.udc.source=tarball -XX:NativeMemoryTracking=detail -Dfile.encoding=UTF-8 com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3 --config-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf
root 107355 92449 0 10:08 pts/0 00:00:00 grep --color=auto -i neo4j

[root@jmngdprv009694 bin]# top
top - 10:10:37 up 22 days, 18:30, 1 user, load average: 0.09, 0.08, 0.10
Tasks: 312 total, 1 running, 311 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 89956272 total, 607284 free, 51983236 used, 37365756 buff/cache
KiB Swap: 67108860 total, 67087752 free, 21108 used. 37060368 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 52248 4160 2240 S 0.0 0.0 12:06.17 systemd

[root@jmngdprv009694 bin]# top | grep -i 95772
95772 root 20 0 56.0g 48.0g 27136 S 0.7 56.0 1:06.54 java
95772 root 20 0 56.0g 48.0g 27136 S 3.3 56.0 1:06.64 java
95772 root 20 0 56.0g 48.0g 27136 S 0.7 56.0 1:06.66 java

[root@jmngdprv009694 bin]# free -g
total used free shared buff/cache available
Mem: 85 49 0 0 35 35
Swap: 63 0 63

Best Regards
Akshat

[akshat_mittal]

Hello Dana,

I think there is a gap from my side in explaining my concern to you.

Let me try again... " Concern is : Java process ( Neo4j process ) taking more than 50% of memory even if it is not loading any data in neo4j. Want to know why this is happening?
Expectation is : It should not take this amount of memory.

I have used the neo4j-admin memrec tool for setting the 3 memory parameters suggested by this tool in my config file. But I can a more usage of memory now with zero processing / loading of data in neo4j.

Below are the updated details:

[root@jmngdprv009694 bin]# ps -ef | grep -i neo4j

root 95772 1 8 09:56 pts/0 00:01:06 /usr/bin/java -cp /datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins:/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf:/datasan02/neo4japp/neo4j-enterprise-3.5.3/lib/:/datasan02/neo4japp/neo4j-enterprise-3.5.3/plugins/ -server -Xms27000m -Xmx27000m -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dunsupported.dbms.udc.source=tarball -XX:NativeMemoryTracking=detail -Dfile.encoding=UTF-8 com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3 --config-dir=/datasan02/neo4japp/neo4j-enterprise-3.5.3/conf
root 107355 92449 0 10:08 pts/0 00:00:00 grep --color=auto -i neo4j

[root@jmngdprv009694 bin]# top
top - 10:10:37 up 22 days, 18:30, 1 user, load average: 0.09, 0.08, 0.10
Tasks: 312 total, 1 running, 311 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 89956272 total, 607284 free, 51983236 used, 37365756 buff/cache
KiB Swap: 67108860 total, 67087752 free, 21108 used. 37060368 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 52248 4160 2240 S 0.0 0.0 12:06.17 systemd

[root@jmngdprv009694 bin]# top | grep -i 95772
95772 root 20 0 56.0g 48.0g 27136 S 0.7 56.0 1:06.54 java
95772 root 20 0 56.0g 48.0g 27136 S 3.3 56.0 1:06.64 java
95772 root 20 0 56.0g 48.0g 27136 S 0.7 56.0 1:06.66 java

[root@jmngdprv009694 bin]# free -g
total used free shared buff/cache available
Mem: 85 49 0 0 35 35
Swap: 63 0 63

Best Regards
Akshat

[akshat_mittal]

Hello Team,

Please update on this.

Regards
Akshat

[tony_chiboucas]

Hello Akshat,

Your configuration is telling Neo4j to use that memory, and keep it reserved for it's use. @dana.canzano has twice explained the why and how it is doing that.

We cannot configure your environment for you, nor hand you a simple set of instructions and configurations as every system and set of requirements are unique.

Please carefully read through the memory configuration docs, and the Neo4j 3.5 Operations Manual. Yes, it is a lot of reading, learning, and experimentation to get to the point where you can make these systems do what you want them to, but there are no shortcuts here.

Thank you,
Tony

[akshat_mittal]

Hello Tony,

Thanks a lot for your reply.
Actually , I am following the given link only. It suggests to use the memrec tool and set the value accordingly. When I set those suggested values , Memory was shoot up to 45%.
Every one wants to run the system with best configurations but suggested configuration lands me in consuming higher memory which is a little bit of concern. But this concern will turn into Major if memory usage goes beyond 70%.
In a nut shell , We need a high performance configuration without affecting the system resources much.

Regards
Akshat

[tony_chiboucas]

akshat.mittal:

It suggests to use the memrec tool

It only suggests that as a tool to get a better idea of the demands of your existing dataset, so you can use some knowledge and math to figure out the configuration best for you. The rest of the page explains what each of the other settings do, so you can better adjust the configuration to your needs.

• init_size sets an amount to always keep for Neo. The bigger this is, the faster Neo will be.
• max_size sets the limit of how much Neo can use for queries and manipulations. If an inefficient query or result-set exceeds this, it will get very slow, and may fail.
• pagecache caches query results, so subsequent requests to the same query will be much faster, but will make new queries a bit slower.

You need to know your system, your goals, and the workings of Neo4j, to performance tune your environment.

akshat.mittal:

When I set those suggested values , Memory was shoot up to 45%.

Because you set init_size to almost half of your system RAM, as explained in 11.1 Memory configuration

You have to read the documentation to be successful with Neo4j.
NOT JUST SKIM IT LOOKING FOR A QUICK FIX.

akshat.mittal:

We need a high performance configuration without affecting the system resources much.

So does everyone, but that's not how databases work. Larger inefficient queries consume more ram. There is no escaping this basic fact.

Database performance, including Neo4j performance, starts with database design for the intended dataset and queries. Then it requires careful structuring of queries so they don't need to load half of the graph into memory. Only then, once these two tasks are complete, can configuration improve performance in any way.

If you have a graph and a query which requires loading 20G of data to complete the query in less than a minute, there's no magic configuration which is going to solve that issue.

Configuration lets you chose between setting aside more RAM, letting RAM get bigger, and caching results. That's all it does.