This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Head's Up! These forums are read-only. All users and content have migrated. Please join us at community.neo4j.com.
- Neo4j
- Technical Discussions
- Neo4j Graph Platform
- Re: Can't perform a Backup on new instance after r...
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Printer Friendly Page
Can't perform a Backup on new instance after restoring a backup to a new instance
03-30-2020 04:07 AM
Hi,
I am in the process of upgrading our neo4j stack to 3.5.16 (Running via aws [Neo4j Enterprise Causal Cluster] Subscription) to make it easier i just created a new Causal Cluster and performed an import of the backup i made on the older instance (3.5.12).
The reason for the move was we were previously getting this issue https://github.com/neo4j/neo4j/issues/12337
To restore i ran the following (on all 3 servers), there seemed to be some permissions issues on the restored db so i fixed those too.
sudo -u neo4j neo4j-admin unbind
sudo neo4j-admin restore --from=/var/lib/neo4j/backups/graph.db-backup-manual --database=graph.db --force
sudo chown -R neo4j /var/lib/neo4j/data/databases/graph.db/
At this point the DB came up with all the data cluster overview is showing all 3 instances, i can query data and use the database perfectly fine.
Now when trying to perform the backup it its 30% and then just hangs, CPU usage sitting on 100%
root@ip-10-0-0-50:/home/ubuntu# neo4j-admin backup --from=127.0.0.1 --backup-dir=/var/lib/neo4j/backups/ --name=graph.db-backup-manual --pagecache=2G
2020-03-30 10:27:00.738+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
// Omitted for brevity
2020-03-30 10:27:22.750+0000 INFO [o.n.b.i.BackupOutputMonitor] Finish receiving transactions at 7293882, took 16s 252ms
Full Consistency Check
.................... 10%
.................... 20%
.................... 30%
..........
I have tried to change the heap size in case that was the issue
export JAVA_OPTS='-Xms8192m -Xmx8192m'
At this point i am kind of out of ideas not sure what is exactly the issue, apart from what the neo4j-admin backup command outputs is there any other log i can check to see if there is an isssue?
Labels:
- Labels:
-
Import-Export
Reply
6 REPLIES 6
03-30-2020 02:06 PM
I've had similar issues before, when pushing a database into an Aura cluster.
Have you considered trying to restore the backup into a local graph running through Neo4j desktop? I can't say for certain but you might get some additional debug info that could be helpful (in our case, it was a corrupted index).
03-31-2020 02:03 AM
Will give that a try.
03-31-2020 05:01 AM
Interestingly I get the exact same issue if i restore to my local graph and try and run a backup.
Your corrupt index thing got me thinking, so i removed all the indexes on the database and tried to run the backup again.
This time it gets further, but then seems to run out of memory, not sure what i need to change to see if it completes.
F:\neo4j\neo4jDatabases\database-8c93f0af-eab1-468d-adeb-90fa81518bb6\installation-3.5.16\bin>neo4j-admin backup --from=127.0.0.1 --backup-dir=F:\neo4j\backup --name=graph.db-backup-manual --pagecache=4G
2020-03-31 11:23:10.355+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
//removed for brevity
2020-03-31 11:23:27.806+0000 INFO [o.n.b.i.BackupOutputMonitor] Finish receiving store files, took 17s 449ms
2020-03-31 11:23:31.940+0000 INFO [o.n.b.i.BackupOutputMonitor] Start recovering store
2020-03-31 11:23:48.639+0000 INFO [o.n.b.i.BackupOutputMonitor] Finish recovering store, took 16s 699ms
2020-03-31 11:23:54.052+0000 INFO [o.n.b.i.BackupOutputMonitor] Finished, took 43s 697ms
Full Consistency Check
.................... 10%
.................... 20%
.................... 30%
.................... 40%
....Exception in thread "ParallelRecordScanner-Stage8_PS_Props-2" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-3" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-6" java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
at org.neo4j.consistency.store.DirectRecordAccess.referenceTo(DirectRecordAccess.java:174)
at org.neo4j.consistency.store.DirectRecordAccess.string(DirectRecordAccess.java:121)
at org.neo4j.consistency.store.DelegatingRecordAccess.string(DelegatingRecordAccess.java:90)
at org.neo4j.consistency.checking.PropertyRecordCheck.checkDataBlock(PropertyRecordCheck.java:76)
at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:50)
at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:32)
at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
at org.neo4j.consistency.report.ConsistencyReporter.forProperty(ConsistencyReporter.java:430)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:78)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
... 24 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706434560 bytes committed virtual memory, and 9467637760 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
... 22 more
Exception in thread "ParallelRecordScanner-Stage8_PS_Props-1" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-5" java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706328064 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
at org.neo4j.consistency.store.DirectRecordAccess.referenceTo(DirectRecordAccess.java:174)
at org.neo4j.consistency.store.DirectRecordAccess.string(DirectRecordAccess.java:121)
at org.neo4j.consistency.store.DelegatingRecordAccess.string(DelegatingRecordAccess.java:90)
at org.neo4j.consistency.checking.PropertyRecordCheck.checkDataBlock(PropertyRecordCheck.java:76)
at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:50)
at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:32)
at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
at org.neo4j.consistency.report.ConsistencyReporter.forProperty(ConsistencyReporter.java:430)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:78)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
... 24 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706221568 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
... 22 more
Exception in thread "ParallelRecordScanner-Stage8_PS_Props-4" Exception in thread "ParallelRecordScanner-Stage8_PS_Props-0" java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706106880 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
... 22 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467621376 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706000384 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
at org.neo4j.consistency.store.DirectRecordAccess.referenceTo(DirectRecordAccess.java:174)
at org.neo4j.consistency.store.DirectRecordAccess.string(DirectRecordAccess.java:121)
at org.neo4j.consistency.store.DelegatingRecordAccess.string(DelegatingRecordAccess.java:90)
at org.neo4j.consistency.checking.PropertyRecordCheck.checkDataBlock(PropertyRecordCheck.java:76)
at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:50)
at org.neo4j.consistency.checking.PropertyRecordCheck.check(PropertyRecordCheck.java:32)
at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
at org.neo4j.consistency.report.ConsistencyReporter.forProperty(ConsistencyReporter.java:430)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:78)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
... 24 more
java.lang.RuntimeException: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1706516480 bytes committed virtual memory, and 9467617280 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:131)
Caused by: org.neo4j.unsafe.impl.internal.dragons.NativeMemoryAllocationRefusedError: Failed to allocate 524288 bytes. So far 2823695012 bytes have already been successfully allocated. The system currently has 20872359936 bytes total physical memory, 1705893888 bytes committed virtual memory, and 9465552896 bytes free physical memory. Relevant system properties: "java.vm.name" = "Java HotSpot(TM) Client VM", "java.vm.vendor" = "Oracle Corporation", "os.arch" = "x86". The allocation was refused by the operating system: java.lang.OutOfMemoryError
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:400)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:420)
at org.neo4j.io.mem.GrabAllocator$Grab.<init>(GrabAllocator.java:110)
at org.neo4j.io.mem.GrabAllocator$Grabs.allocateAligned(GrabAllocator.java:263)
at org.neo4j.io.mem.GrabAllocator.allocateAligned(GrabAllocator.java:83)
at org.neo4j.io.pagecache.impl.muninn.PageList.initBuffer(PageList.java:299)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:391)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:341)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:309)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:60)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:137)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1054)
at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1024)
at org.neo4j.consistency.checking.full.PropertyReader.visitPropertyRecordChain(PropertyReader.java:91)
at org.neo4j.consistency.checking.full.PropertyReader.getPropertyRecordChain(PropertyReader.java:57)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:80)
at org.neo4j.consistency.checking.full.PropertyAndNodeIndexedCheck.check(PropertyAndNodeIndexedCheck.java:60)
at org.neo4j.consistency.report.ConsistencyReporter.dispatch(ConsistencyReporter.java:116)
at org.neo4j.consistency.report.ConsistencyReporter.forNode(ConsistencyReporter.java:416)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:63)
at org.neo4j.consistency.checking.full.PropertyAndNode2LabelIndexProcessor.process(PropertyAndNode2LabelIndexProcessor.java:39)
at org.neo4j.consistency.checking.full.RecordCheckWorker.run(RecordCheckWorker.java:77)
at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.Workers$Worker.run(Workers.java:126)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:396)
... 22 more
what is even more interesting if i add all the indexes that i removed back, then the backup gets stuck again at 30% point like before but just sits and hangs no errors
04-06-2020 11:22 PM
Anyone have any ideas that i can try?
05-12-2020 06:32 AM
I am actually still stuck with this 
05-13-2020 10:31 AM
Hi, You can try to change the parameters in neo4j.conf file. The parameters are:
dbms.memory.heap.initial_size=3500m
dbms.memory.heap.max_size=3500m
dbms.memory.pagecache.size=3000m
The values depends your RAM size
Thanks
Nodes 2022
NODES 2022, Neo4j Online Education Summit
All the sessions of the conference are now available online
Related Content
- Running Neodash on Azure web app for containers in Integrations
- Unexpected "stop-the-world" pauses while Neo4J is inactive? in Neo4j Graph Platform
- Neo4j Bloom AuraDB in Neo4j Graph Platform
- GDS functions aren't working (e.g. gds.graph.create) in Neo4j Graph Platform
- Query seems to not execute properly. Is it an issue? in Neo4j Graph Platform
