SSL config with helm installation

[cloudcuckoohome]

Hello!
I'm currently trying to setup a standalone instance ofe neo4j in my Kubernetes cluster.

I tried the following things:

• I set up cert-manager and got a tls certificate (secret) via letsencrypt. Named: neo4j-tls
  Secret looks like:

kind: Secret
apiVersion: v1
metadata:
  name: neo4j-xxxxx
  generateName: neo4j-
  namespace: default
  uid: xxxxxxxxxxxxxxxxxxxxxxxx
  resourceVersion: '337554'
  creationTimestamp: '2022-06-04T11:32:23Z'
  labels:
    cert-manager.io/next-private-key: 'true'
  ownerReferences:
    - apiVersion: cert-manager.io/v1
      kind: Certificate
      name: neo4j
      uid: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      controller: true
      blockOwnerDeletion: true
  
data:
  tls.key: >-
    LS0.......​

• Then i created a PersistentVolume with the following yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: neo4j-data
  labels:
    app: "sharedneo4j-20220603"
    helm.neo4j.com/volume-role: data
spec:
  storageClassName: "manual"
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  hostPath:
    path: /mnt/kube_volumes/databases/neo4j
    type: Directory​

and gave ownership and rights for that path to user:group 7474:7474

• I created a helm values yaml like that:

# Default values for Neo4j.
# This is a YAML-formatted file.

neo4j:
  # Name of your cluster
  name: ""

  # If the password is not set or empty a random password will be generated during installation
  password: ""

  # Neo4j Edition to use (community|enterprise)
  edition: "community"
  # set edition: "enterprise" to use Neo4j Enterprise Edition
  #
  # To use Neo4j Enterprise Edition you must have a Neo4j license agreement.
  #
  # More information is also available at: https://neo4j.com/licensing/
  # Email inquiries can be directed to: licensing@neo4j.com
  #
  # Set acceptLicenseAgreement: "yes" to confirm that you have a Neo4j license agreement.
  acceptLicenseAgreement: "no"
  #
  # set offlineMaintenanceModeEnabled: true to restart the StatefulSet without the Neo4j process running
  # this can be used to perform tasks that cannot be performed when Neo4j is running such as `neo4j-admin dump`
  offlineMaintenanceModeEnabled: false
  #
  # set resources for the Neo4j Container. The values set will be used for both "requests" and "limit".
  resources:
    cpu: "1000m"
    memory: "5Gi"

# Volumes for Neo4j
volumes:
  data:
    # REQUIRED: specify a volume mode to use for data
    # Valid values are share|selector|defaultStorageClass|volume|volumeClaimTemplate|dynamic
    # To get up and running quickly, for development or testing, use "defaultStorageClass" for a dynamically provisioned volume of the default storage class.
    mode: "selector"

    # Only used if the mode is set to "selector"
    # Will attach to existing volumes that match the selector
    selector:
      storageClassName: "manual"
      accessModes:
        - ReadWriteOnce
      requests:
        storage: 100Gi
      # A helm template to generate a label selector to match existing volumes n.b. both storageClassName and label selector must match existing volumes
      #selectorTemplate:
      #  matchLabels:
      #    app: "{{elm. .Values.neo4j.name }}"
      #    hneo4j.com/volume-role: "data"

    # Only used if mode is set to "defaultStorageClass"
    # Dynamic provisioning using the default storageClass
    defaultStorageClass:
      accessModes:
        - ReadWriteOnce
      requests:
        storage: 10Gi

    # Only used if the mode is set to "dynamic"
    # Dynamic provisioning using the provided storageClass
    dynamic:
      storageClassName: "neo4j"
      accessModes:
        - ReadWriteOnce
      requests:
        storage: 100Gi

    # Only used if mode is set to "volume"
    # Provide an explicit volume to use
    volume:
      # If set an init container (running as root) will be added that runs:
      #   `chown -R <securityContext.fsUser>:<securityContext.fsGroup>` AND `chmod -R g+rwx`
      # on the volume. This is useful for some file systems (e.g. NFS) where Kubernetes fsUser or fsGroup settings are not respected
      setOwnerAndGroupWritableFilePermissions: true

      # Example (using a specific Persistent Volume Claim)
      # persistentVolumeClaim:
      #   claimName: my-neo4j-pvc

    # Only used if mode is set to "volumeClaimTemplate"
    # Provide an explicit volumeClaimTemplate to use
    volumeClaimTemplate: {}

  # provide a volume to use for backups
  # n.b. backups will be written to /backups on the volume
  # any of the volume modes shown above for data can be used for backups
  backups:
    mode: "share" # share an existing volume (e.g. the data volume)
    share:
      name: "data"

  # provide a volume to use for logs
  # n.b. logs will be written to /logs/$(POD_NAME) on the volume
  # any of the volume modes shown above for data can be used for logs
  logs:
    mode: "share" # share an existing volume (e.g. the data volume)
    share:
      name: "data"

  # provide a volume to use for csv metrics (csv metrics are only available in Neo4j Enterprise Edition)
  # n.b. metrics will be written to /metrics/$(POD_NAME) on the volume
  # any of the volume modes shown above for data can be used for metrics
  metrics:
    mode: "share" # share an existing volume (e.g. the data volume)
    share:
      name: "data"

  # provide a volume to use for import storage
  # n.b. import will be mounted to /import on the underlying volume
  # any of the volume modes shown above for data can be used for import
  import:
    mode: "share" # share an existing volume (e.g. the data volume)
    share:
      name: "data"

  # provide a volume to use for licenses
  # n.b. licenses will be mounted to /licenses on the underlying volume
  # any of the volume modes shown above for data can be used for licenses
  licenses:
    mode: "share" # share an existing volume (e.g. the data volume)
    share:
      name: "data"

# Services for Neo4j
services:
  # A ClusterIP service with the same name as the Helm Release name should be used for Neo4j Driver connections originating inside the
  # Kubernetes cluster.
  default:
    # Annotations for the K8s Service object
    annotations: { }

  # A LoadBalancer Service for external Neo4j driver applications and Neo4j Browser
  neo4j:
    enabled: true

    # Annotations for the K8s Service object
    annotations: { }

    spec:
      # Type of service.
      type: LoadBalancer

      # in most cloud environments LoadBalancer type will receive an ephemeral public IP address automatically. If you need to specify a static ip here use:
      # loadBalancerIP: ...

    # ports to include in neo4j service
    ports:
      http:
        enabled: true #Set this to false to remove HTTP from this service (this does not affect whether http is enabled for the neo4j process)
      https:
        enabled: true #Set this to false to remove HTTPS from this service (this does not affect whether https is enabled for the neo4j process)
      bolt:
        enabled: true #Set this to false to remove BOLT from this service (this does not affect whether https is enabled for the neo4j process)

  # A service for admin/ops tasks including taking backups
  # This service is available even if the deployment is not "ready"
  admin:
    enabled: true
    # Annotations for the admin service
    annotations: { }
    spec:
      type: ClusterIP
    # n.b. there is no ports object for this service. Ports are autogenerated based on the neo4j configuration

  # A "headless" service for admin/ops and Neo4j cluster-internal communications
  # This service is available even if the deployment is not "ready"
  internals:
    enabled: false
    # Annotations for the internals service
    annotations: { }
    # n.b. there is no ports object for this service. Ports are autogenerated based on the neo4j configuration

# Neo4j Configuration (yaml format)
config:
  db.temporal.timezone: "Europe/Berlin"
  dbms.default_database: "neo4j"
  dbms.config.strict_validation: "true"
  # https://community.neo4j.com/t5/neo4j-graph-platform/troubleshooting-connection-issues-to-neo4j/m-p/47959
  dbms.connector.bolt.tls_level: "OPTIONAL" # allows both encrypted and unencrypted driver connections
  dbms.ssl.policy.bolt.enabled: "true"
  dbms.ssl.policy.bolt.privateKey.secretName: "neo4j-tls"
  dbms.ssl.policy.bolt.privateKey.subPath: "tls.key"
  dbms.ssl.policy.bolt.publicCertificate.secretName: "neo4j-tls"
  dbms.ssl.policy.bolt.publicCertificate.subPath: "tls.key"
  #dbms.ssl.policy.default.client_auth: "NONE"
  dbms.ssl.policy.bolt.client_auth: "NONE"
  dbms.ssl.policy.https.client_auth: "NONE"

# securityContext defines privilege and access control settings for a Pod or Container. Making sure that you do not run Neo4j as root user.
securityContext:
  runAsNonRoot: true
  runAsUser: 7474
  runAsGroup: 7474
  fsGroup: 7474
  fsGroupChangePolicy: "Always"

# Readiness probes are set to know when a container is ready to be used.
# Because Neo4j uses Java these values are large to distinguish between long Garbage Collection pauses (which don't require a restart) and an actual failure.
# These values should mark Neo4j as not ready after at most 5 minutes of problems (20 attempts * max 15 seconds between probes)
readinessProbe:
  failureThreshold: 20
  timeoutSeconds: 10
  periodSeconds: 5

# Liveness probes are set to know when to restart a container.
# Because Neo4j uses Java these values are large to distinguish between long Garbage Collection pauses (which don't require a restart) and an actual failure.
# These values should trigger a restart after at most 10 minutes of problems (40 attempts * max 15 seconds between probes)
livenessProbe:
  failureThreshold: 40
  timeoutSeconds: 10
  periodSeconds: 5

# Startup probes are used to know when a container application has started.
# If such a probe is configured, it disables liveness and readiness checks until it succeeds
# When restoring Neo4j from a backup, it's important that the startup probe gives time for Neo4j to recover and/or upgrade store files
# When using Neo4j clusters, it's important that the startup probe gives the Neo4j cluster time to form
startupProbe:
  failureThreshold: 1000
  periodSeconds: 5

# top level setting called ssl to match the "ssl" from "dbms.ssl.policy"
#ssl:
#  # setting per "connector" matching neo4j config
#  bolt:
#    privateKey:
#      secretName:  # we set up the template to grab `private.key` from this secret
#      subPath:  # we specify the privateKey value name to get from the secret
#    publicCertificate:
#      secretName:  # we set up the template to grab `public.crt` from this secret
#      subPath:  # we specify the publicCertificate value name to get from the secret
#    trustedCerts:
#      sources: [ ] # a sources array for a projected volume - this allows someone to (relatively) easily mount multiple public certs from multiple secrets for example.
#    revokedCerts:
#      sources: [ ]  # a sources array for a projected volume
#  https:
#    privateKey:
#      secretName:
#      subPath:
#    publicCertificate:
#      secretName:
#      subPath:
#    trustedCerts:
#      sources: [ ]
#    revokedCerts:
#      sources: [ ]

# Kubernetes cluster domain suffix
clusterDomain: "cluster.local"

# Override image settings in Neo4j pod
image:
  imagePullPolicy: IfNotPresent
  # set a customImage if you want to use your own docker image
  # customImage: my-image:my-tag

# additional environment variables for the Neo4j Container
env: {}

# Other K8s configuration to apply to the Neo4j pod
podSpec:
  # Anti Affinity
  # If set to true then an anti-affinity rule is applied to prevent database pods with the same `neo4j.name` running on a single Kubernetes node.
  # If set to false then no anti-affinity rules are applied
  # If set to an object then that object is used for the Neo4j podAntiAffinity
  podAntiAffinity: true

  # Name of service account to use for the Neo4j Pod (optional)
  # this is useful if you want to use Workload Identity to grant permissions to access cloud resources e.g. cloud object storage (AWS S3 etc.)
  serviceAccountName: ""

  # How long the Neo4j pod is permitted to keep running after it has been signaled by Kubernetes to stop. Once this timeout elapses the Neo4j process is forcibly terminated.
  # A large value is used because Neo4j takes time to flush in-memory data to disk on shutdown.
  terminationGracePeriodSeconds: 3600

  # initContainers for the Neo4j pod
  initContainers: [ ]

  # additional runtime containers for the Neo4j pod
  containers: [ ]

# print the neo4j user password set during install to the `helm install` log
logInitialPassword: true

# Jvm configuration for Neo4j
jvm:
  # If true any additional arguments are added after the Neo4j default jvm arguments.
  # If false Neo4j default jvm arguments are not used.
  useNeo4jDefaultJvmArguments: true
  # additionalJvmArguments is a list of strings. Each jvm argument should be a separate element
  additionalJvmArguments: []
  # - "-XX:+HeapDumpOnOutOfMemoryError"
  # - "-XX:HeapDumpPath=/logs/neo4j.hprof"​

and installed with this command:
helm install sharedneo4j-20220603 neo4j/neo4j-standalone -f helm-parameters.yaml

• So far everything fine. Except: when the pod tries to start, currently i get the following error in the neo4j.log:

2022-06-04 12:05:27.193+0000 INFO  Starting...
2022-06-04 12:05:28.004+0000 ERROR Failed to start Neo4j on 0.0.0.0:7474.
java.lang.IllegalArgumentException: Base directory '/var/lib/neo4j/certificates/bolt' for SSL policy with name 'bolt' does not exist.
        at org.neo4j.ssl.config.SslPolicyLoader.createSslPolicy(SslPolicyLoader.java:169) ~[neo4j-ssl-4.4.6.jar:4.4.6]
        at org.neo4j.ssl.config.SslPolicyLoader.addPolicy(SslPolicyLoader.java:153) ~[neo4j-ssl-4.4.6.jar:4.4.6]
        at java.util.HashMap$Values.forEach(HashMap.java:977) ~[?:?]
        at org.neo4j.ssl.config.SslPolicyLoader.load(SslPolicyLoader.java:143) ~[neo4j-ssl-4.4.6.jar:4.4.6]
        at org.neo4j.ssl.config.SslPolicyLoader.create(SslPolicyLoader.java:106) ~[neo4j-ssl-4.4.6.jar:4.4.6]
        at org.neo4j.graphdb.factory.module.edition.CommunityEditionModule.<init>(CommunityEditionModule.java:137) ~[neo4j-4.4.6.jar:4.4.6]
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:134) ~[neo4j-4.4.6.jar:4.4.6]
        at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) ~[neo4j-4.4.6.jar:4.4.6]
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:142) [neo4j-4.4.6.jar:4.4.6]
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:95) [neo4j-4.4.6.jar:4.4.6]
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:34) [neo4j-4.4.6.jar:4.4.6]
2022-06-04 12:05:28.006+0000 INFO  Neo4j Server shutdown initiated by request
2022-06-04 12:05:28.007+0000 INFO  Stopped.
​

I tried diferent combinations under "config:" with different results. 

Any ideas on how to config this right, to have a running neo4j standalone instance?

[Vincenzo_Calia]

Hi there, can you show the Certificate file? how did you set it up?
I'm trying to set it up but I'm having difficulties with setting the dnsNames.
Thank you very much