How to backup remote Neo4j database running in Kubernetes

[david_daiell]

We wish to run a scheduled backup of a Neo4j database running in an Azure Kubernetes cluster. The database pod has the following specs:
Name: neo4j-core-0
Image: neo4j:4.1.0
Environment Variables: NEO4J_dbms_backup_enabled: "true"
NEO4J_dbms_backup_listen__address: 0.0.0.0:6362

We created a separate pod to remotely perform the online backups, which has the following specs:
Name: neo4j-core-backup
Image: neo4j:4.1.0-enterprise
Environment Variables: NEO4J_ACCEPT_LICENSE_AGREEMENT: yes
NEO4J_dbms_backup_enabled: "true"
NEO4J_dbms_backup_listen__address: 0.0.0.0:6362

We are able to query neo4j-core-0 from neo4j-core-backup using cypher-shell:
cypher-shell -A neo4j-core-0-URL:7687 -u neo4j -p mypassword

However, we are getting a 'Connection Refused" error when trying to backup neo4j-core-0 from neo4j-core-backup using neo4j-admin backup (using either port 7687 or 6362). Any suggestions how to resolve?
neo4j-admin backup --backup-dir=./backup --verbose --from=neo4j-core-0-URL:7687 --database=neo4j --pagecache=4G

--> 2020-08-20 16:40:04.445+0000 INFO [c.n.c.p.i.ClientChannelInitializer] Initializing client channel [id: 0x235e58c8]
org.neo4j.cli.CommandFailedException: Execution of backup failed. finishConnect(..) failed: Connection refused
at com.neo4j.backup.impl.OnlineBackupCommand.execute(OnlineBackupCommand.java:134)
at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:67)
at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:33)
at picocli.CommandLine.executeUserObject(CommandLine.java:1783)
at picocli.CommandLine.access$900(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975)
at picocli.CommandLine.execute(CommandLine.java:1904)
at org.neo4j.cli.AdminTool.execute(AdminTool.java:77)
at org.neo4j.cli.AdminTool.main(AdminTool.java:58)
Caused by: com.neo4j.backup.impl.BackupExecutionException: com.neo4j.causalclustering.catchup.storecopy.DatabaseIdDownloadFailedException: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /XX.XXX.XX.XX:6362

[amsilf]

Good day,

What do you use as a persistent storage? Could you verify that 6362 is open on your core-0? There is a good article on k8s - https://medium.com/neo4j/how-to-backup-neo4j-running-in-kubernetes-3697761f229a, so you may try their helm chart as well.