Expose helm-installed Neo4Jj using ingress-nginx

[lhumphreys]

Hi,

I am working with the following setup:

• an AWS EKS Kubernetes cluster version 1.16
• helm version 3.2.1
• nginx-ingress helm chart version 1.39.1
• neo4j helm chart version 4.0.4-2

I am trying to deploy an exposed version of the Neo4j helm chart so that my clients can access it in their browsers.

I am 95% of the way there - the neo4j chart deploys and the logs of the core pods report no errors, and I can access the cluster through my browser. The problem comes when I want the browser client to connect to the database. The Connect URL in the form is neo4j://neo4j.my-platform.com:7687, when I hit Connect the network tab shows an attempted connection to wss://bolt-2.neo4j.my-platform.com:7687/ which times out every time.

I am attempting to be "fancy", as the helm chart documentation puts it, by using ingress-nginx. My ingress looks like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/alias: "true"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  labels:
  name: neo4j
  namespace: neo4j
spec:
  rules:
  - host: neo4j.my-platform.com
    http:
      paths:
      - backend:
          serviceName: neo4j-external-access
          servicePort: 7474
  - host: bolt-0.neo4j.my-platform.com
    http:
      paths:
      - backend:
          serviceName: neo4j-external-access
          servicePort: 7687
  - host: bolt-1.neo4j.my-platform.com
    http:
      paths:
      - backend:
          serviceName: neo4j-external-access
          servicePort: 7687
  - host: bolt-2.neo4j.my-platform.com
    http:
      paths:
      - backend:
          serviceName: neo4j-external-access
          servicePort: 7687
  tls:
  - hosts:
    - neo4j.my-platform.com
    secretName: cert-manager-neo4

my service to expose the cluster looks like:

apiVersion: v1
kind: Service
metadata:
  name: neo4j-external-access
  namespace: neo4j
spec:
  clusterIP: redacted
  ports:
  - name: https
    port: 7473
    protocol: TCP
    targetPort: 7473
  - name: http
    port: 7474
    protocol: TCP
    targetPort: 7474
  - name: bolt
    port: 7687
    protocol: TCP
    targetPort: 7687
  selector:
    app.kubernetes.io/component: core
    app.kubernetes.io/instance: neo4j
    app.kubernetes.io/name: neo4j
  sessionAffinity: None
  type: ClusterIP

and the exposure configmap I wrote looks like:

apiVersion: v1
data:
  NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
  NEO4J_dbms_mode: CORE
  neo4j_neo4j_core_0_NEO4J_dbms_connector_bolt_advertised__address: bolt-0.neo4j.my-platform.com:7687
  neo4j_neo4j_core_0_NEO4J_dbms_connector_http_advertised__address: bolt-0.neo4j.my-platform.com:7474
  neo4j_neo4j_core_0_NEO4J_dbms_connector_https_advertised__address: bolt-0.neo4j.my-platform.com:7473
  neo4j_neo4j_core_0_NEO4J_dbms_default__advertised__address: bolt-0.neo4j.my-platform.com
  neo4j_neo4j_core_1_NEO4J_dbms_connector_bolt_advertised__address: bolt-1.neo4j.my-platform.com:7687
  neo4j_neo4j_core_1_NEO4J_dbms_connector_http_advertised__address: bolt-1.neo4j.my-platform.com:7474
  neo4j_neo4j_core_1_NEO4J_dbms_connector_https_advertised__address: bolt-1.neo4j.my-platform.com:7473
  neo4j_neo4j_core_1_NEO4J_dbms_default__advertised__address: bolt-1.neo4j.my-platform.com
  neo4j_neo4j_core_2_NEO4J_dbms_connector_bolt_advertised__address: bolt-2.neo4j.my-platform.com:7687
  neo4j_neo4j_core_2_NEO4J_dbms_connector_http_advertised__address: bolt-2.neo4j.my-platform.com:7474
  neo4j_neo4j_core_2_NEO4J_dbms_connector_https_advertised__address: bolt-2.neo4j.my-platform.com:7473
  neo4j_neo4j_core_2_NEO4J_dbms_default__advertised__address: bolt-2.neo4j.my-platform.com
kind: ConfigMap
metadata:
  name: neo4j-neo4j-externally-addressable-config
  namespace: neo4j

which I'm pretty sure works because each core pods logs look like:

2020-06-15 14:31:18.421+0000 INFO  ======== Neo4j 4.0.4 ========
2020-06-15 14:31:18.427+0000 INFO  Starting...
2020-06-15 14:31:51.547+0000 INFO  Called db.clearQueryCaches(): Query cache already empty.
2020-06-15 14:31:51.832+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2020-06-15 14:31:51.833+0000 INFO  Started.
2020-06-15 14:31:54.653+0000 INFO  Remote interface available at http://bolt-0.neo4j.my-platform.com:7474/

The nginx-ingress configmap for tcp exposure looks like:

apiVersion: v1
data:
  "7473": neo4j/neo4j-external-access:7473
  "7474": neo4j/neo4j-external-access:7474
  "7687": neo4j/neo4j-external-access:7687
kind: ConfigMap
metadata:
  name: nginx-ingress-tcp
  namespace: neo4j

I can't think of what I'm missing, I would very much appreciate some pointers from anyone who can see my error(s)!

[david_allen]

lhumphreys:

neo4j_core_0_NEO4J_dbms_default__advertised__address

These settings right here are telling the database to advertise itself to the client under the bad address that you don't want. If all the other bits are working, you need to tell the members to advertise themselves as whatever externally valid & routable address you intend to use.

[gfontanilla]

Hi lhumphreys,

I have patterned my testing on what you have set above, one thing I noticed is that from this is that

lhumphreys:

wss://bolt-2.neo4j.my-platform.com:7687

This uses secure websocket, but seeing the config for the ingress, only the web url: neo4j.my-platform.com is using tls.

so I tried to removing the tls to test it out and it worked, confirming that we need the bolt to use tls as well.