Cant seem to mount volume with neo4j 3.3.9 (path in use?)

[njmmeibergen]

I have created a custom dockerfile for neo4j that imports necessary plugins. Next I created a deployment yaml which I am running on Kubernetes on Azure. The deployment works very well and I can reach the browser on the expected URL, which works great. However,

1. the moment I attach a Volume, neo4j becomes unable to start with the below provided error stack.
2. There is data in the mount volume (azure files), but it is only /data/dbms, not /data/databases. Probably because the last directory already comes pre-filled image? in

How would I be able to initialize Neo4j successfully and start mapping the content of /data/databases/ when it data is added to the graph?

Find below the error stack and deployment yaml.

Deployment yaml

apiVersion: v1
kind: Service
metadata:
  name: myservice
spec:
  selector:
    app: mygraph
  ports:
    - name: bolt
      protocol: TCP
      port: 7687
      targetPort: 7687
      nodePort: 30076
    - name: http
      protocol: TCP
      port: 7474
      targetPort: 7474
      nodePort: 30074
    - name: https
      protocol: TCP
      port: 7473
      targetPort: 7473
      nodePort: 30073
  type: LoadBalancer
  loadBalancerIP: 40.118.172.46
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mygraph-deployment
  labels:
    app: mygraph
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mygraph
  template:
    metadata:
      labels:
        app: mygraph
    spec:
      containers:
      - name: mygraph
        image: mycontainer.azurecr.io/mygraph:dev
        securityContext:
          privileged: true
        ports:
        - containerPort: 7687
        - containerPort: 7474
        - containerPort: 7473
        env:
          - name: NEO4J_dbms_jvm_additional
            value: "-Xms512m -Xmx6g"
        # When adding the below lines the deployment failes. Without it, it works great!
        volumeMounts:
        - name: neo4jdata
          mountPath: /var/lib/neo4j/data/
      volumes:
        - name: neo4jdata
          persistentVolumeClaim:
            claimName: azurefile

Error stack

2019-09-05 07:02:12.522+0000 INFO ======== Neo4j 3.3.9 ========

2019-09-05 07:02:12.560+0000 INFO Starting...

2019-09-05 07:02:14.122+0000 INFO Bolt enabled on 0.0.0.0:7687.

2019-09-05 07:02:14.274+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)

at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:220)

at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:111)

at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:79)

at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@149dd36b' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)

at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)

at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:212)

... 3 more

Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/graph.db

at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:211)

at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:126)

at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:58)

at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:88)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)

... 5 more

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.internal.locker.StoreLockerLifecycleAdapter@134d26af' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.".

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:466)

at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)

at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:207)

... 9 more

Caused by: org.neo4j.kernel.StoreLockException: Unable to create path for store dir: /var/lib/neo4j/data/databases/graph.db. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)

at org.neo4j.kernel.internal.locker.StoreLocker.storeLockException(StoreLocker.java:116)

at org.neo4j.kernel.internal.locker.StoreLocker.checkLock(StoreLocker.java:79)

at org.neo4j.kernel.internal.locker.GlobalStoreLocker.checkLock(GlobalStoreLocker.java:60)

at org.neo4j.kernel.internal.locker.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:36)

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)

... 11 more

Caused by: java.io.IOException: Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.

at org.neo4j.io.fs.DefaultFileSystemAbstraction.mkdirs(DefaultFileSystemAbstraction.java:124)

at org.neo4j.kernel.internal.locker.StoreLocker.checkLock(StoreLocker.java:73)

... 14 more

2019-09-05 07:02:14.317+0000 INFO Neo4j Server shutdown initiated by request

[david_allen]

A common error with the Neo4j docker container is that the process running Neo4j doesn't have sufficient privileges to write files to the /data directory, or to change permissions on those folders. It looks possible that this is what your'e seeing here, since the root cause of your problem is Unable to create directory path [/var/lib/neo4j/data/databases/graph.db] for Neo4j store.

To fix this, you should look into how you're creating that Azure volume and attaching it to your container. This I'm not sure how to help with because I haven't done this in Azure before, but I would check the docs on where azurefile comes from, and what its permissions as attached are.

Finally -- the Neo4j docker guidance says that you should attach this at /data, not at /var/lib/neo4j/data so you should certainly change that. Under the covers, /data is usually a symlink for /var/lib/neo4j/data inside of the container - and so this wrong mount point may be confusing issues.

Docker volumes for Neo4j:

https://neo4j.com/docs/operations-manual/current/docker/introduction/#docker-volumes

[njmmeibergen]

Dear David,

Thank you for your repsonse. I am not fully sure whether you have seen the latest edit I made to my question, as I am still trying to find out what is going wrong.

Meanwhile I have experimented with both /data and /var/lib/neo4j/data, but as you mention I'll leave it at /data. Furthermore, I see that in my Azure Files directory data has been written. In particular the file /data/dbms/auth.ini is created. This to me indicates that permissions are set correctly. Am I right with this conclusion? Therefore it is surprising that /data/databases/graph.db cannot be written. Something that is different between /data/dbms/ and /data/databases/ is that /data/databases/ already exists within the image, whereas /data/dbms/ is created during container initiation.

Would you see any other possible issue besides permissions? And am I right that permissions do not seem to be the issue?

[david_allen]

From the error that I could see, it appeared it was a disk IO error, this could be a full disk, an unwritable disk, or so on. This might not be the only error though.

I would recommend trying to get a full copy of debug.log from the container, and see if something else is in there. It would be located in /logs/debug.log. If the file doesn't exist (for example because the database couldn't create it, because there is a disk IO problem) then again I think you'd need to look carefully into how the Azure disk is set up.

Inside of the container, Neo4j knows nothing about docker, it's just a regular file path -- and if the error is that it can't write a file to a path, this requires investigation of the disk is my thinking.

[njmmeibergen · Accepted solution]

Hi David,

Indeed, it was an issue of rights of writing data to Azure. In particular in the setup of the StorageClass in Kubernetes, you can set additional mountOptions. One of those is the uid which should be set to 100 - the user id used for neo4j.

Example of the StorageClass yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefile-neo4j
provisioner: kubernetes.io/azure-file
reclaimPolicy: Retain
mountOptions:
  - dir_mode=0755
  - file_mode=0755
  - uid=100 # Allow write access for neo4j
  - gid=1000
parameters:
  skuName: Standard_LRS

It is curious though that the file /data/dbms was written successful before, while /data/databases could not be written. Seems that dbms and databases are created by different users? But that would be a separate question.

[david_allen]

Very glad that you worked this out and found the solution!

As for the different users -- this is the script that runs when the container starts. In this code, you can check whether any user switching is done, and can see where the chmod stuff is necesary.

github.com

neo4j/docker-neo4j/blob/master/docker-image-src/3.5/docker-entrypoint.sh

#!/bin/bash -eu

cmd="$1"

function running_as_root
{
    test "$(id -u)" = "0"
}

function secure_mode_enabled
{
    test "${SECURE_FILE_PERMISSIONS:=no}" = "yes"
}

containsElement () {
  local e match="$1"
  shift
  for e; do [[ "$e" == "$match" ]] && return 0; done
  return 1
}

This file has been truncated. show original