Cannot allocate memory when trying to backup database

[fredrik_leijon]

Hi

I have a problem with a database that I'm unable to take a backup of. The error i get is when running neo4j-admin backup --from=127.0.0.1 --backup-dir=/data/backups --name=graph.db-backup is:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000004de800000, 8249147392, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8249147392 bytes for committing reserved memory.
# An error report file with more information is saved as:

Server specs is this
AWS EC2 with 32GB ram (r5a.xlarge) running ubuntu
Config settings are based on memrec

dbms.memory.heap.initial_size=11800m
dbms.memory.heap.max_size=11800m
dbms.memory.pagecache.size=12000m

Java version is

openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)

Neo4j is 3.5.6 (same problem with 3.5.5)

Any tips on what to tweek, I guess i could lower the amount of memory the sever is given but I need that to keep performance up.

[stefan_armbrust]

Try to limit the amout of heap used for backup using env variable HEAP_SIZE, e.g.

HEAP_SIZE=2G neo4j-admin backup --from=127.0.0.1 --backup-dir=/data/backups --name=graph.db

Not 100% sure, but backup uses standard neo4j.conf, so you have two processes competing for memory.

[fredrik_leijon]

HEAP_SIZE doesn't seem to do anything

$ HEAP_SIZE=2G neo4j-admin backup --from=127.0.0.1 --backup-dir=/data/backups --name=graph.db-backup
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000004de800000, 8249147392, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8249147392 bytes for committing reserved memory.
# An error report file with more information is saved as:
#hs_err_pid27281.log

Passing -Xms2048m -Xmx2048m as parameters doesn't change the allocation amount either.

[stefan_armbrust]

Try to supply additionally a custom neo4j.conf that uses a much smaller pagecache setting via:

HEAP_SIZE=2G NEO4J_CONF=<path to special neo4j conf> neo4j-admin backup ...

If that doesn't work, try to export this env variables.

[fredrik_leijon]

Ok

after a bit of testing this worked for me
NEO4J_HOME=/usr/share/neo4j HEAP_SIZE=512m NEO4J_CONF=/home/pinmeto/neo4j.conf neo4j-admin backup --from=127.0.0.1 --backup-dir=/data/backups --name=graph.db-backup

[dmiranda2791]

@fredrik.leijon, by any chance did you experience this error while you were testing the command you shared?

2019-11-26 17:22:25.193+0000 INFO [o.n.b.i.BackupOutputMonitor] Finished, took 27m 28s 106ms
unexpected error: Error starting org.neo4j.com.storecopy.ExternallyManagedPageCache$GraphDatabaseFactoryWithPageCacheFactory$1, /home/ubuntu/neo4j-backups

I was able to start the backup process with a custom neo4j.conf, then I got an error saying that the AdminTool was not found but I fix it with the variable NEOJ_HOME as in you command. I am stuck here now though .

Thanks in advance.