Remote Neo4j Backups from a Docker Container

[mike_blum_neo4j]

Neo4j Version: 3.4.3

I'm working on this question here:

Running neo4j-admin backup from inside docker container on cron interval Docker

I wonder if you have found some solution for that ?

for our own backup solution and as a part of that effort, I want to standup Neo4j inside a Docker container for our CI testing. While my backup script works well against a bare neo4j instance running as a plain java process, remote backups don't seem to work so well when the database is inside a container.

Here is my Docker command:

# start neo4j instance
sudo mkdir -p /tmp/neo4j/data
sudo mkdir -p /tmp/neo4j/logs
sudo chown mblum:staff -R /tmp/neo4j
docker run --name neo4j-ci --detach \
  --publish=7474:7474 --publish=7687:7687 --publish=6362:6362 \
  --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
  --env=NEO4J_dbms_memory_pagecache_size=4G \
  --env=NEO4J_dbms.backup.enabled=true \
  --env=NEO4J_dbms.backup.address=0.0.0.0:6362 \
  --volume=/tmp/neo4j/data:/data \
  --volume=/tmp/neo4j/logs:/logs \
  neo4j:3.4-enterprise

When i test it with the neo4j-admin tool I get some buffer writing issues:

neo4j-admin backup --backup-dir=/tmp/backups --name=neo4j-ci
2018-11-06 18:27:10.288+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
org.neo4j.commandline.admin.CommandFailed: Failed to run a backup using the available strategies.
	at org.neo4j.backup.impl.BackupStrategyCoordinator.performBackup(BackupStrategyCoordinator.java:99)
	at org.neo4j.backup.impl.OnlineBackupCommand.execute(OnlineBackupCommand.java:74)
	at org.neo4j.commandline.admin.AdminTool.execute(AdminTool.java:127)
	at org.neo4j.commandline.admin.AdminTool.main(AdminTool.java:51)
	Suppressed: org.neo4j.causalclustering.catchup.storecopy.StoreIdDownloadFailedException: org.neo4j.causalclustering.catchup.CatchUpClientException: Completed exceptionally when executing operation org.neo4j.causalclustering.catchup.storecopy.GetStoreIdRequest@f1da57d on localhost:6362
		at org.neo4j.causalclustering.catchup.storecopy.StoreCopyClient.fetchStoreId(StoreCopyClient.java:203)
		at org.neo4j.backup.impl.BackupDelegator.fetchStoreId(BackupDelegator.java:87)
		at org.neo4j.backup.impl.CausalClusteringBackupStrategy.performFullBackup(CausalClusteringBackupStrategy.java:68)
		at org.neo4j.backup.impl.BackupStrategyWrapper.fullBackupWithTemporaryFolderResolutions(BackupStrategyWrapper.java:146)
		at org.neo4j.backup.impl.BackupStrategyWrapper.performBackupWithoutLifecycle(BackupStrategyWrapper.java:111)
		at org.neo4j.backup.impl.BackupStrategyWrapper.doBackup(BackupStrategyWrapper.java:72)
		at org.neo4j.backup.impl.BackupStrategyCoordinator.performBackup(BackupStrategyCoordinator.java:86)
		... 3 more
	Caused by: org.neo4j.causalclustering.catchup.CatchUpClientException: Completed exceptionally when executing operation org.neo4j.causalclustering.catchup.storecopy.GetStoreIdRequest@f1da57d on localhost:6362
		at org.neo4j.causalclustering.catchup.TimeoutLoop.exception(TimeoutLoop.java:87)
		at org.neo4j.causalclustering.catchup.TimeoutLoop.waitForCompletion(TimeoutLoop.java:57)
		at org.neo4j.causalclustering.catchup.CatchUpClient.makeBlockingRequest(CatchUpClient.java:94)
		at org.neo4j.causalclustering.catchup.storecopy.StoreCopyClient.fetchStoreId(StoreCopyClient.java:199)
		... 9 more
	Caused by: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
		at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
		at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
		at org.neo4j.causalclustering.catchup.TimeoutLoop.waitForCompletion(TimeoutLoop.java:48)
		... 11 more
	Caused by: java.nio.channels.ClosedChannelException
		at org.neo4j.causalclustering.catchup.TrackingResponseHandler.onClose(TrackingResponseHandler.java:146)
		at org.neo4j.causalclustering.catchup.CatchUpClient$CatchUpChannel.lambda$connect$1(CatchUpClient.java:168)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
		at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
		at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
		at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:611)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
		at java.lang.Thread.run(Thread.java:748)
		at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:110)
	Suppressed: org.neo4j.com.ComException: Channel has been closed, so no need to try to write to it anymore. Client closed it?
		at org.neo4j.com.ChunkingChannelBuffer.writeCurrentChunk(ChunkingChannelBuffer.java:640)
		at org.neo4j.com.ChunkingChannelBuffer.done(ChunkingChannelBuffer.java:698)
		at org.neo4j.com.Protocol.serializeRequest(Protocol.java:190)
		at org.neo4j.com.Client.sendRequest(Client.java:305)
		at org.neo4j.com.Client.sendRequest(Client.java:290)
		at org.neo4j.backup.impl.BackupClient.fullBackup(BackupClient.java:68)
		at org.neo4j.backup.impl.BackupProtocolService$FullBackupStoreCopyRequester.copyStore(BackupProtocolService.java:493)
		at org.neo4j.com.storecopy.StoreCopyClient.copyStore(StoreCopyClient.java:134)
		at org.neo4j.backup.impl.BackupProtocolService.fullBackup(BackupProtocolService.java:165)
		at org.neo4j.backup.impl.BackupProtocolService.doFullBackup(BackupProtocolService.java:138)
		at org.neo4j.backup.impl.HaBackupStrategy.performFullBackup(HaBackupStrategy.java:86)
		at org.neo4j.backup.impl.BackupStrategyWrapper.fullBackupWithTemporaryFolderResolutions(BackupStrategyWrapper.java:146)
		at org.neo4j.backup.impl.BackupStrategyWrapper.performBackupWithoutLifecycle(BackupStrategyWrapper.java:111)
		at org.neo4j.backup.impl.BackupStrategyWrapper.doBackup(BackupStrategyWrapper.java:72)
		at org.neo4j.backup.impl.BackupStrategyCoordinator.performBackup(BackupStrategyCoordinator.java:86)
		... 3 more
command failed: Failed to run a backup using the available strategies.

I've noted that on the enterprise Docker images 3.4.3 has been struck from the repository: Neo4j on Docker. I've had good luck using the 3.4.9 backup driver targeting a 3.4.3 instance but I assume this won't always be the case and will break at some point.

Are there any caveats to doing remote backups from a Docker container?

Thanks!
Mike

[mike_blum_neo4j]

I downloaded the latest 3.4.9 and got the same error:

neo4j version
neo4j 3.4.9

sudo mkdir -p /tmp/neo4j/data
sudo mkdir -p /tmp/neo4j/logs
sudo chown mblum:staff -R /tmp/neo4j
docker run --name neo4j-ci --detach \
  --publish=7474:7474 --publish=7687:7687 --publish=6362:6362 \
  --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
  --env=NEO4J_dbms_memory_pagecache_size=4G \
  --env=NEO4J_dbms.backup.enabled=true \
  --env=NEO4J_dbms.backup.address=0.0.0.0:6362 \
  --volume=/tmp/neo4j/data:/data \
  --volume=/tmp/neo4j/logs:/logs \
  neo4j:3.4.9-enterprise

neo4j-admin backup --backup-dir=/tmp/backups --name=neo4j-ci
2018-11-06 18:59:02.928+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
command failed: Failed to run a backup using the available strategies.

[mike_blum_neo4j]

Figured out my mistake:

  --env=NEO4J_dbms.backup.enabled=true \
  --env=NEO4J_dbms.backup.address=0.0.0.0:6362 

needs to be:

--env=NEO4J_dbms_backup_enabled=true \
--env=NEO4J_dbms_backup_address=0.0.0.0:6362

as specified in the docs here: https://neo4j.com/docs/operations-manual/current/installation/docker/#docker-environment-variables

[markrity]

Thank you for the info.
But do you run the neo4j-admin back outside the container ? Because I can't see you doing docker exec or though ?

[mike_blum_neo4j]

I ran it outside the container so that the backup ends up in a durable location. I wonder if a docker volume for the backup would be performant?

[mike_blum_neo4j]

@bakk if you're on here.