Neo4j

dhartig · ‎08-27-2019

Hello all:

I am wondering if there is anybody that is running a Neo4j setup like mine that might be experiencing the same crash I am experiencing.

The Setup

Running a the official Docker image from Docker Hub as an Azure Container instance. I have mapped the /data and /import directory to Azure Files.

The Problem:

I am trying to do an import, something like this:

//bus to iso bus relationship
USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:///import.csv" as row
MATCH(a:Bus{GID:row.GID_Bus}),
     (b:PSSE_Bus{Number:row.PSSE_Bus})
MERGE (b)-[:IS_REFERENCE_OF]->(a);

works fine on my laptop (no docker), but fails on my Cloud setup with a log message as below.

I am sure that it has something to do with the Azure File setup, but can anyone give me an idea about what to look for from the log output below?

2019-08-27 13:12:03.224+0000 INFO [o.n.k.i.s.c.CountsTracker] Rotated counts store at transaction 90 to [/data/databases/graph.db/neostore.counts.db.a], from [/data/databases/graph.db/neostore.counts.db.b].
2019-08-27 13:12:03.586+0000 ERROR [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint failed Failed to flush
org.neo4j.kernel.impl.store.UnderlyingStorageException: Failed to flush
	at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:257)
	at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
	at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
	at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
	at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
	at org.neo4j.kernel.impl.scheduler.ScheduledJobHandle.lambda$new$0(ScheduledJobHandle.java:83)
	at org.neo4j.kernel.impl.scheduler.ThreadPool.lambda$submit$0(ThreadPool.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.IOException: Permission denied
	at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.flushAllPagesParallel(MuninnPageCache.java:649)
	at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.flushAndForce(MuninnPageCache.java:605)
	at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:253)
	... 11 more
Caused by: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.io.IOException: Permission denied

this keeps repeating itself forever until the DB is wiped out.

Thanks,

dhartig

david_allen · ‎08-27-2019

I'm not 100% sure, but you should double check the permissions on the /data and /import folders given your mount approach. Permissions issues with folder mounts are a common problem with the Neo4j Docker container, because (clearly) it needs the ability to write to the /data directory.

More information and detail here:

I have mapped the /data and /import directory to Azure Files.

This is the thing to check -- provide detail on how you did this, and ensure that the container can actually write to this mount point. Because your error is suggesting this is the issue.

dhartig · ‎09-06-2019

@david.allen

These are mapped using Azure container as in the script below.

However I know it has access most of the time, as Neo4j works pretty much without a hitch until I try and do an import. And there there is the flushing process that apparently corrupts the DB from which we cannot recover.
So the question I have, is there another process the is spawned during import that runs as a different user? That may explain why it gets a permission denied.

apiVersion: '2018-10-01'
location: eastus2
name: <name>
properties:
  containers:
  - name: neo4j
    properties:
      environmentVariables: []
      image: neo4j:latest
      ports:
        - port: 7474
        - port: 7687
      resources:
        requests:
          cpu: 2.0
          memoryInGB: 8.0
      volumeMounts:
      - mountPath: /data
        name: filesharevolume
      - mountPath: /import
        name: importsharevolume
  osType: Linux
  restartPolicy: Always
  ipAddress:
    type: Public
    ports:
      - port: 7474
      - port: 7687
    dnsNameLabel: <name>
  volumes:
  - name: filesharevolume
    azureFile:
      sharename: sharenamedata
      sharename: sharenameimport
      storageAccountName: <id>
      storageAccountKey: <key> 
  - name: importsharevolume
    azureFile:
      sharename: sharenameimport
      storageAccountName: <id>
      storageAccountKey: <key>      
tags: {}
type: Microsoft.ContainerInstance/containerGroups

david_allen · ‎09-06-2019

I'm sorry - I can't help with the storageAccount specifics as I haven't done this with Azure before and I don't know how to advise there.

Have you checked debug.log for any other exceptions which may be happening at the same time or prior to your file access issue?

If you run in this way, and don't do an import, but do a simple CREATE (:TestNode) does it work or fail in the same way?

When you mount this data to Azure, can you separately in Azure's explorer see the file & directory structure that Neo4j creates?

pvillads · ‎11-12-2019

I run neo4j in a container on Azure using the following deployment template:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "variables": {
        "container1name": "neo4jcontainer",
        "container1image": "neo4j:latest",
        "storageaccountname": "*****",
        "storageaccountkey": "****"
    },
    "resources": [
        {
            "name": "neo4j-containers",
            "type": "Microsoft.ContainerInstance/containerGroups",
            "apiVersion": "2018-10-01",
            "location": "[resourceGroup().location]",
            "properties": {
                "containers": [
                    {
                        "name": "[variables('container1name')]",
                        "properties": {
                            "environmentVariables": [
                                {
                                    "name": "NEO4J_AUTH",
                                    "value": "****"
                                }
                            ],
                            "image": "[variables('container1image')]",
                            "resources": {
                                "requests": {
                                    "cpu": 1,
                                    "memoryInGB": 8
                                }
                            },
                            "ports": [
                                {
                                    "port": 7474
                                },
                                {
                                    "port": 7687
                                }
                            ],
                            "volumeMounts": [
                                {
                                    "name": "data",
                                    "mountPath": "/data"
                                },
                                {
                                    "name": "logs",
                                    "mountPath": "/logs"
                                },
                                {
                                    "name": "import",
                                    "mountPath": "/var/lib/neo4j/import"
                                },
                                {
                                    "name": "plugins",
                                    "mountPath": "/plugins"
                                },
                                {
                                    "name": "config",
                                    "mountPath": "/conf"
                                }
                            ]
                        }
                    }
                ],
                "osType": "Linux",
                "ipAddress": {
                    "type": "Public",
                    "ports": [
                        {
                            "protocol": "TCP",
                            "port": 7474
                        },
                        {
                            "protocol": "TCP",
                            "port": 7687
                        }
                    ],
                    "dnsNameLabel": "pvillads-neo4j-dns"
                },
                "volumes": [
                    {
                        "name": "data",
                        "azureFile": {
                            "shareName": "data",
                            "storageAccountName": "[variables('storageaccountname')]",
                            "storageAccountKey": "[variables('storageaccountkey')]"
                        }
                    },
                    {
                        "name": "logs",
                        "azureFile": {
                            "shareName": "logs",
                            "storageAccountName": "[variables('storageaccountname')]",
                            "storageAccountKey": "[variables('storageaccountkey')]"
                        }
                    },
                    {
                        "name": "import",
                        "azureFile": {
                            "shareName": "import",
                            "storageAccountName": "[variables('storageaccountname')]",
                            "storageAccountKey": "[variables('storageaccountkey')]"
                        }
                    },
                    {
                        "name": "plugins",
                        "azureFile": {
                            "shareName": "plugins",
                            "storageAccountName": "[variables('storageaccountname')]",
                            "storageAccountKey": "[variables('storageaccountkey')]"
                        }
                    },
                    {
                        "name": "config",
                        "azureFile": {
                            "shareName": "config",
                            "storageAccountName": "[variables('storageaccountname')]",
                            "storageAccountKey": "[variables('storageaccountkey')]"
                        }
                    }
                ]
            }
        }
    ]
}

I can see that the files are created correctly in the data directory and I can read the files in the import directory etc. It all works as a charm until I get more data than can be handled in around 1GB of storage. It then breaks very ungracefully and the data is lost (which takes Neo4j out of consideration for most production scenarios).

I will create another ticket for that issue once I have researched it a little more.

Neo4j

Problems Running Neo4j in Azure Container Instance (Docker)