Assertion Failed on start up 3.5.12 Enterprise

[daniel2]

Hi there.

After upgrading ram in our on prem server from 32GB to 64GB Neo4j does not want to start up and gives the following error:
2019-11-01 11:47:14.358+0000 INFO ======== Neo4j 3.5.12 ========
2019-11-01 11:47:14.362+0000 INFO Starting...
2019-11-01 11:47:14.855+0000 INFO Initiating metrics...
2019-11-01 11:47:16.742+0000 INFO Sending metrics to CSV file at /var/lib/neo4j/metrics
2019-11-01 11:47:17.240+0000 INFO Bolt enabled on 0.0.0.0:7687.
java: malloc.c:2401: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

Any help would be appreciated

[stefan_armbrust]

Whats your settings for heap size (init and max) and pagecache?

[daniel2]

So at 32GB it was something along the lines of 24GB Heap Max and 5GB page cache.
At 64GB it is currently 48GB heap max and 10g page cache.

However after the upgrade I tried both values and still got a value. I was able to resolve the issue by delete all neo4j data, including the database and restoring from yesterdays dump. Luckily no writes had been made between those 2 days.

[daniel2]

So turns out the issue is back, same error as well as a jvm error below:
2019-11-07 12:38:08.771+0000 INFO ======== Neo4j 3.5.12 ========
2019-11-07 12:38:08.777+0000 INFO Starting...
2019-11-07 12:38:09.315+0000 INFO Initiating metrics...
2019-11-07 12:38:10.053+0000 INFO Sending metrics to CSV file at /var/lib/neo4j/metrics
2019-11-07 12:38:10.352+0000 INFO Bolt enabled on 127.0.0.1:7687.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007efbfef6c0dd, pid=2287, tid=0x00007efaded4b700

JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)

Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode linux-amd64 compressed oops)

Problematic frame:

C [libc.so.6+0x950dd]# [ timer expired, abort... ]

This occurred after an abrupt power cut, after restoring the database from my daily backups the problem still persists. If I remove the database and start fresh then the database starts up fine.
My Neo4j config file can be found here: https://gist.github.com/dllz/94ea9dcd61ebee31a48d9e90aa5e290a

[daniel2]

Soooo
Got rid of that error by deleting count store .a and count store .b
However we are back to the original error of:
2019-11-07 13:38:53.081+0000 INFO ======== Neo4j 3.5.12 ========
2019-11-07 13:38:53.090+0000 INFO Starting...
2019-11-07 13:38:54.042+0000 INFO Initiating metrics...
2019-11-07 13:38:59.459+0000 INFO Sending metrics to CSV file at /var/lib/neo4j/metrics
2019-11-07 13:38:59.799+0000 INFO Bolt enabled on 0.0.0.0:7687.
java: malloc.c:2401: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

Here is my debug.log: https://gist.github.com/dllz/aefd8b637cc712bd6ae7bd8133659eb8

Edit: Seems to be zig zagging between the 2 errors now.

[stefan_armbrust]

I'd leave up more memory for the system. JVM heap of > 24GB does make sense in a lot of cases.