Neo4j server slow startup time

[petercheng]

After importing 17 million nodes and 85 million relationships into the Neo4j database (150gb), starting up and shutting down the Neo4j database is much slower. Start up now takes over 15 minutes as compared to 4 seconds prior to the data being imported:

2019-07-11 21:13:13.616+0000 INFO Starting...
2019-07-11 21:13:16.049+0000 INFO Bolt enabled on 9.42.117.142:7687.
2019-07-11 21:13:17.611+0000 INFO Started.
2019-07-11 21:13:18.790+0000 INFO Remote interface available at http://9.42.117.142:7474/
2019-07-12 03:41:05.896+0000 INFO Neo4j Server shutdown initiated by request
2019-07-12 03:41:05.951+0000 INFO Stopping...
2019-07-12 03:41:06.005+0000 INFO Stopped.
nohup: ignoring input
2019-07-15 15:53:34.281+0000 INFO ======== Neo4j 3.5.3 ========
2019-07-15 15:53:34.302+0000 INFO Starting...
2019-07-15 16:09:01.795+0000 INFO Bolt enabled on 9.42.117.142:7687.
2019-07-15 16:09:05.829+0000 INFO Started.
2019-07-15 16:09:06.999+0000 INFO Remote interface available at http://9.42.117.142:7474/

1. Is this slow start up time to be expected? This is a small subset of what we expect to import into the database, so if increasing the amount of data in the database causes start up times to slow down linearly, this would be a significant weakness of the database.

2. If this is not expected, what can I do to debug why its starting up so slowly?

Neo4j Version: Community 3.5.3
Platform: Red Hat Enterprise Linux Server 7.6 (Maipo)

Thank you,
Peter

[dana_canzano]

does your logs\debug.log provide detail on the slowness from 2019-07-15 15:53 to 2019-07-15 16:09

[petercheng]

@dana.canzano There was nothing that stood out to me that provided any detail on slowness. I would attach the debug.log here, but I guess attachments aren't allowed and the text is too large to fit here.

Is there a debug option to see what is it exactly doing during this period of time?

[dana_canzano]

how long does this line report

2019-06-26 14:48:04.547+0000 INFO [o.n.k.i.p.PageCacheWarmer] Page cache warmup completed. 0 pages loaded. Duration: 26ms.

in your instance?
Are you seeing a number of lines similar to

2019-06-26 14:50:02.274+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=102, gcTime=0, gcCount=0}

or a line similar to

2019-07-10 11:15:35.784+0000 INFO [o.n.k.NeoStoreDataSource] Recovery required from position

[petercheng]

@dana.canzano I don't see any line with "PageCacheWarmer" or "Recovery required from position" in it.

However, I do see "VmPauseMonitorComponent" in these two lines:

2019-07-15 16:09:02.915+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=862, gcTime=933, gcCount=1}
2019-07-15 16:09:04.163+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=446, gcTime=525, gcCount=1}

[dana_canzano]

ok... the VmPauseMonitorComponent would be indicative of a Java Garbage Collection event.

How much RAM is on the machine? if this is a linux environment please return free -m

Also, how have you defined the heap and pagecache in the conf/neo4j.conf. Again if linux return the results of grep dbms.memory conf/neo4j.conf

[petercheng]

@dana.canzano There is 32GB of RAM on the machine. Is this enough memory or is more recommended? Here is the output of the commands, I set the memory size to what is recommended by the neo4j-admin command.

free -m

          total        used        free      shared  buff/cache   available

Mem: 32146 18890 4121 15 9135 12738
Swap: 16127 372 15755

grep dbms.memory conf/neo4j.conf

dbms.memory.heap.initial_size=12000m
dbms.memory.heap.max_size=12000m
dbms.memory.pagecache.size=12100m

[dana_canzano]

thanks. so it looks as if initial and max heap are set at 12GB and pagecache is also at 12GB thus consuming up to 24G of the total 32G of RAM.
maybe lat question ;). what is the value returned by the line in the debug.log which indicates

2019-06-26 14:48:04.145+0000 INFO [o.n.i.d.DiagnosticsManager]   Total size of mapped files: 4

[petercheng]

@dana.canzano Here is the output:

Total size of mapped files: 37.04 GB

What is the size of mapped files referring to?

[dana_canzano]

Total size of mapped files: 37.04 GB

represents the total amount of data which is eligible to be put in the pagecache, provided it is sufficiently large enough. given a pagecache of 12GB and mapped files of 37 GB, this would indicate at most 1/3 of your database is in RAM. So as a starter, regardless of startup time, your system is not optimally configured in terms of RAM. In a perfect world your should have RAM > (heap + 'Total size of mapped files: ` + 1-2GB for general OS operations). But as you only have 32GB of RAM this is not optimal.

However this still does not explain startup time degradation. There was a fix in 3.5.6 which addressed a significant memory allocation for query stats. Maybe this has something to do with the slowdown. any opportunity to upgrade to 3.5.6

[petercheng]

@dana.canzano Okay I will work on getting more hardware for the server so it can run optimally.

Yes we can upgrade as this is prototyping server. To upgrade, it looks like a can unzip a new download, update the configuration settings, copy the database directory, and set the dbms.allow_upgrade=true. When upgrade is done, shut down and set dbms.allow_upgrade=false and restart the server. This sound about right to you?

[dana_canzano]

correct for upgrade steps. the complete steps are detailed at https://neo4j.com/docs/operations-manual/current/upgrade/

[petercheng]

@dana.canzano I've upgraded to 3.5.7 and the startup/shutdown time is still slow as before.

Any other thoughts on where to look?

[dana_canzano]

Out of curiosity what vendor/version of java are you using ,?

[petercheng]

@dana.canzano I'm using 1.8.0

neo4j-community-3.5.7# java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

[dana_canzano]

hmm... ok. had you been running IBM JDK maybe, just maybe I might suggest this is the source as anecdotally we have seen slower peformance here though I dont have specifics. I'm still a bit stumped without debug.log

[petercheng]

@dana.canzano for sure I'm not using the IBM JDK, I recall it had some issue running with the IBM JDK if I'm not mistaken.

This is from the debug output:

2019-07-15 21:30:16.770+0000 INFO [o.n.i.d.DiagnosticsManager] JVM information:
2019-07-15 21:30:16.770+0000 INFO [o.n.i.d.DiagnosticsManager] VM Name: OpenJDK 64-Bit Server VM
2019-07-15 21:30:16.770+0000 INFO [o.n.i.d.DiagnosticsManager] VM Vendor: Oracle Corporation
2019-07-15 21:30:16.770+0000 INFO [o.n.i.d.DiagnosticsManager] VM Version: 25.212-b04
2019-07-15 21:30:16.771+0000 INFO [o.n.i.d.DiagnosticsManager] JIT compiler: HotSpot 64-Bit Tiered Compilers
2019-07-15 21:30:16.771+0000 INFO [o.n.i.d.DiagnosticsManager] VM Arguments: [-Xms12000m, -Xmx12000m, -XX:+UseG1GC, -XX:-OmitStackTraceInFastThrow, -XX:+AlwaysPreTouch, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.tls.ephemeralDHKeySize=2048, -Djdk.tls.rejectClientInitiatedRenegotiation=true, -Dunsupported.dbms.udc.source=tarball, -Dfile.encoding=UTF-8]

[petercheng]

@dana.canzano can I email you the debug.log or share it somehow via a dropbox link?

[sumanth818]

Hi my database is about 900GB. And I have set

dbms.memory.heap.initial_size=90g
dbms.memory.heap.max_size=90g
dbms.memory.pagecache.size=70g

I have about 350 Million nodes and 350 Million relationships. Right now it takes me about 2 hours to restart the graph. My degug log is as follows

2020-02-19 15:47:37.070+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db
2020-02-19 15:47:37.114+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db
2020-02-19 15:47:37.114+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Format not configured. Selected format from the store: RecordFormat:StandardV3_4[v0.A.9]
2020-02-19 15:47:37.848+0000 INFO [o.n.k.i.i.l.NativeLabelScanStore] Label index cleanup job registered
2020-02-19 15:47:37.850+0000 INFO [o.n.k.i.i.l.NativeLabelScanStore] Label index cleanup job started
2020-02-19 15:47:37.859+0000 INFO [o.n.k.NeoStoreDataSource] Commits found after last check point (which is at LogPosition{logVersion=1325, byteOffset=8147740}). First txId after last checkpoint: 68680 
2020-02-19 15:47:37.859+0000 INFO [o.n.k.NeoStoreDataSource] Recovery required from position LogPosition{logVersion=1325, byteOffset=8147740}
2020-02-19 15:47:38.922+0000 INFO [o.n.k.i.i.l.NativeLabelScanStore] Label index cleanup job finished: Number of pages visited: 36516, Number of cleaned crashed pointers: 437, Time spent: 1s 71ms
2020-02-19 15:47:38.922+0000 INFO [o.n.k.i.i.l.NativeLabelScanStore] Label index cleanup job closed
2020-02-19 16:19:10.212+0000 INFO [o.n.k.r.Recovery]   10% completed
2020-02-19 16:25:40.898+0000 INFO [o.n.k.r.Recovery]   20% completed
2020-02-19 16:31:52.686+0000 INFO [o.n.k.r.Recovery]   30% completed
2020-02-19 16:38:11.355+0000 INFO [o.n.k.r.Recovery]   40% completed
2020-02-19 16:59:29.585+0000 INFO [o.n.k.r.Recovery]   50% completed
2020-02-19 16:59:33.281+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job registered: descriptor=IndexRule[id=1, descriptor=Index( GENERAL, :label[0](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\1\index-1
2020-02-19 16:59:33.281+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job started: descriptor=IndexRule[id=1, descriptor=Index( GENERAL, :label[0](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\1\index-1
2020-02-19 16:59:33.287+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job finished: descriptor=IndexRule[id=1, descriptor=Index( GENERAL, :label[0](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\1\index-1 Number of pages visited: 112, Number of cleaned crashed pointers: 0, Time spent: 5ms
2020-02-19 16:59:33.287+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job closed: descriptor=IndexRule[id=1, descriptor=Index( GENERAL, :label[0](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\1\index-1
2020-02-19 16:59:33.296+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job registered: descriptor=IndexRule[id=3, descriptor=Index( GENERAL, :label[1](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\3\index-3
2020-02-19 16:59:33.296+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job started: descriptor=IndexRule[id=3, descriptor=Index( GENERAL, :label[1](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\3\index-3
2020-02-19 16:59:33.297+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job finished: descriptor=IndexRule[id=3, descriptor=Index( GENERAL, :label[1](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\3\index-3 Number of pages visited: 8, Number of cleaned crashed pointers: 0, Time spent: 1ms
2020-02-19 16:59:33.297+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job closed: descriptor=IndexRule[id=3, descriptor=Index( GENERAL, :label[1](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\3\index-3
2020-02-19 16:59:33.301+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job registered: descriptor=IndexRule[id=5, descriptor=Index( GENERAL, :label[2](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\5\index-5
2020-02-19 16:59:33.301+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job started: descriptor=IndexRule[id=5, descriptor=Index( GENERAL, :label[2](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\5\index-5
2020-02-19 16:59:33.307+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job finished: descriptor=IndexRule[id=5, descriptor=Index( GENERAL, :label[2](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\5\index-5 Number of pages visited: 109, Number of cleaned crashed pointers: 0, Time spent: 6ms
2020-02-19 16:59:33.307+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job closed: descriptor=IndexRule[id=5, descriptor=Index( GENERAL, :label[2](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\5\index-5
2020-02-19 16:59:33.346+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job registered: descriptor=IndexRule[id=7, descriptor=Index( GENERAL, :label[55](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\7\index-7
2020-02-19 16:59:33.346+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job started: descriptor=IndexRule[id=7, descriptor=Index( GENERAL, :label[55](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\7\index-7
2020-02-19 16:59:33.490+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job registered: descriptor=IndexRule[id=9, descriptor=Index( GENERAL, :label[55](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\9\index-9
2020-02-19 16:59:33.500+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job registered: descriptor=IndexRule[id=11, descriptor=Index( GENERAL, :label[56](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\11\index-11
2020-02-19 16:59:33.508+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job registered: descriptor=IndexRule[id=13, descriptor=Index( GENERAL, :label[56](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\13\index-13
2020-02-19 16:59:33.509+0000 INFO [o.n.k.i.a.i.IndexingService] IndexingService.init: indexes not specifically mentioned above are ONLINE
2020-02-19 17:03:18.796+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job finished: descriptor=IndexRule[id=7, descriptor=Index( GENERAL, :label[55](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\7\index-7 Number of pages visited: 3278616, Number of cleaned crashed pointers: 458209, Time spent: 3m 45s 450ms
2020-02-19 17:03:18.796+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job closed: descriptor=IndexRule[id=7, descriptor=Index( GENERAL, :label[55](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\7\index-7
2020-02-19 17:03:18.797+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job started: descriptor=IndexRule[id=9, descriptor=Index( GENERAL, :label[55](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\9\index-9
2020-02-19 17:09:52.755+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job finished: descriptor=IndexRule[id=9, descriptor=Index( GENERAL, :label[55](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\9\index-9 Number of pages visited: 6651465, Number of cleaned crashed pointers: 18574175, Time spent: 6m 33s 957ms
2020-02-19 17:09:52.755+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job closed: descriptor=IndexRule[id=9, descriptor=Index( GENERAL, :label[55](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\9\index-9
2020-02-19 17:09:52.755+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job started: descriptor=IndexRule[id=11, descriptor=Index( GENERAL, :label[56](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\11\index-11
2020-02-19 17:09:52.778+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job finished: descriptor=IndexRule[id=11, descriptor=Index( GENERAL, :label[56](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\11\index-11 Number of pages visited: 284, Number of cleaned crashed pointers: 0, Time spent: 23ms
2020-02-19 17:09:52.778+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job closed: descriptor=IndexRule[id=11, descriptor=Index( GENERAL, :label[56](property[0]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\11\index-11
2020-02-19 17:09:52.778+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job started: descriptor=IndexRule[id=13, descriptor=Index( GENERAL, :label[56](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\13\index-13
2020-02-19 17:09:52.825+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job finished: descriptor=IndexRule[id=13, descriptor=Index( GENERAL, :label[56](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\13\index-13 Number of pages visited: 674, Number of cleaned crashed pointers: 0, Time spent: 47ms
2020-02-19 17:09:52.825+0000 INFO [o.n.k.i.i.s.GenericNativeIndexProvider] Schema index cleanup job closed: descriptor=IndexRule[id=13, descriptor=Index( GENERAL, :label[56](property[1876]) ), provider={key=native-btree, version=1.0}], indexFile=E:\DIRTTcast\data\Neo4j\neo4jDatabases\database-05393b80-9eb5-4e3b-8310-7936545fa66b\installation-3.5.14\data\databases\graph.db\schema\index\native-btree-1.0\13\index-13

I would appreciate if you could let me know, what the issue is and how I can speedup the restart process. Thank you

[dana_canzano]

@sumanth818 what version of Neo4j.
How have you determined your database is 900G?
How did you determine

dbms.memory.heap.initial_size=90g
dbms.memory.heap.max_size=90g
dbms.memory.pagecache.size=70g

and at a minimum, if you database is 900G and you have set the dbms.memory.pagecache.size=70G thenn less than 10% of the database is in RAM. How much total RAM is on the machine/instance?

[sumanth818]

@dana.canzano I have Neo4j 4.0, but my database uses 3.5.14. Sorry my data base of 900GB includes neostore.transaction.db. Without that it is about 300 GB. I got this number using

:sysinfo

My server has 256GB RAM. And regarding the heap size and page cache I set them to 90GB and 70 GB respectfully in the Neo4j config file. I have other services that run on this server. So I had to allot some RAM to those.

If I dedicate this server to neo4j, what should be my Pagecache and heap size allocation if I have a 256 GB RAM

[dana_canzano]

sumanth818:

My server has 256GB RAM. And regarding the heap size and page cache I set them to 90GB and 70 GB respectfully in the Neo4j config file.

understood. How did you determine to set them to said values? if the size of the database is 300GB (i.e. not including neostore.transaction.db* files) then i would suggest setting setting

dbms.memory.pagecache.size=300GB
dbms.memory.heap.init_size=32G
dbms.memory.heap.max_size=32G

unless there was a specific reason for setting pagecache to 70G and heap to 90G? Was there a reason these values were choosen?

Also, I'm not sure I understand

@dana.canzano I have Neo4j 4.0, but my database uses 3.5.14

[sumanth818]

As I said earlier,I have other services running on the server, and they need about 50 GB of RAM. So, I set the remaining RAM to pagecache and heap size. But the main reason to allot more RAM to heap is because I was getting heap errors, while doing certain queries, so I allotted more RAM to it. Other than that, I didn't follow any specific guidelines on how much to allot to heapsize and pagecache.

But 300GB for pagecache would be higher than the available RAM (256GB).

And regarding the Neo4j version, what I meant is I have the neo4j 4.0 installed. But since I'm having issues with apoc procedures on Neo4j 4.0, i used version 3.5.14 while creating the database. In Neo4j desktop you can have multiple versions for different databases.

[dana_canzano]

my apologies on suggesting dbms.memory.pagecache.size=300G when the server itself only has 256G.
So if those other services require 50G, thus leaving a max of 176G of RAM left over for Neo4j and general OS operations, I would reset to

dbms.memory.pagecache.size=125GB
dbms.memory.heap.init_size=32G
dbms.memory.heap.max_size=32G

and regarding setting heap to 90G because you had encountered 'heap errors' then ideally we would want to investigate what queries were responsible and see if they were properly written.

[dittyvkm]

I face an issue with neo4j while deploying my project in the Ubuntu (16.04) server. For some reason, whenever I start using neo4j on the server the swap memory slowly creeps up to 100% (of just ~4GB of swap space) while regular memory remains largely underutilized (around 9% usage of ~128GB of RAM). In my local machine, I am not facing this issue and the python script to create neo4j database from csv files takes only 15 minutes to execute. But in the server it takes days and finally it consume full swap memory. I tried deploying in aws server and I am facing the same issue there also. In both servers, the I have upgraded the java version to 11.0.5 to make it compatible with my local.