How to choose the right netty_tcnative JAR for Enterprise version on Docker?

[mhood]

Hi, my company just signed up for Neo4J Enterprise and I'm hitting roadblocks while configuring TLS for Bolt and enabling HTTPS. I'm currently running Docker using neo4j:enterprise (Neo4j 4.3.5 w/ JVM - Version:11.0.12).

Everything works fine if I do not try to enable TLS for bolt or HTTPS. However, when following the Ops Guide for enabling those features, Neo4j fails on startup. The error for the stack trace is very long and I have attached it as a file to prevent it from clobbering the post:
neo4j_stack_trace.txt (59.2 KB)

However, this particular line leads me to believe the error has to do with Neo4J not being able to find an appropriate JAR for netty:

Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.bolt.BoltServer@365e65bb' failed to initialize. Please see the attached cause exception "Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative_x86_64, netty_tcnative]"

Here is my docker-compose.yml file.

version: "3.8"
services:
  neo4j:
    image: "neo4j:enterprise"
    container_name: "neo4j"
    environment:
      NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
      NEO4J_AUTH: "neo4j/test"
      NEO4J_dbms_ssl_policy_bolt_enabled: "true"
      NEO4J_dbms_ssl_policy_bolt_base__directory: "/certificates/bolt"
      NEO4J_dbms_ssl_policy_bolt_private__key: "private.key"
      NEO4J_dbms_ssl_policy_bolt_public__certificate: "public.crt"
      NEO4J_dbms_ssl_policy_https_base__directory: "/certificates/https"
      NEO4J_dbms_ssl_policy_https_private__key: "private.key"
      NEO4J_dbms_ssl_policy_https_public__certificate: "public.crt"
      NEO4J_dbms_ssl_policy_https_enabled: "true"
      NEO4J_dbms_connector_http_enabled: "true"
      NEO4J_dbms_connector_https_enabled: "true"
      NEO4J_dbms_connector_bolt_tls__level: "REQUIRED"
      NEO4J_dbms_netty_ssl_provider: "OPENSSL"
    ports:
      - "7473:7473"
      - "7474:7474"
      - "7687:7687"
    expose:
      - 7473
      - 7474
      - 7687
    volumes:
      - ${HOME}/neo4j/data:/data
      - ${HOME}/neo4j/plugins:/plugins
      - ${HOME}/neo4j/conf:/conf
      - ${HOME}/neo4j/logs:/logs
      - ${HOME}/neo4j/certificates:/certificates

Additionally here is my neo4j home directory

drwxr-xr-x  certificates
drwxr-xr-x  conf
drwxr-xr-x  data
drwx------  import
drwxr-xr-x  logs
drwxrwxrwx  plugins

I placed a copy of netty-tcnative-2.0.43.Final-linux-x86_64.jar in neo4j/plugins. However, to be honest, I'm not really sure how one would decide which version to use for a specific version of Neo4J -- the documentation appears silent on this issue. I found a few related posts around the internet, but they all seem to lead to unresolved threads or github issues that are still open. So, I apologize if this has been adequately addressed elsewhere.

I appreciate any pointers or suggestions on how to resolve this.

Thanks!

[michael_hunger]

If you have an enterprise license you can just file a support ticket for a quick resolution. TLS is always annoying and tricky to resolve.

You shouldn't need any extra plugins it should work out of the box.

[mhood]

Somebody on Discord recommended this document to get SSL up and running when using Docker: How to set up SSL communcation when running Neo4j within a Docker Container - Knowledge Base.

It works for 3.4-enterprise, but not latest -- which appears to be 4.3.5 right now. So, the knowledge base might be a little dated.

[mhood · Accepted solution]

For posterity sake, my issues ended up having nothing to with Netty or JARS, but rather I could not figure out what the best way was to setup secure connections for HTTPS and Bolt.

I was able to get neo4j:4.3-enterprise running correctly using this docker-compose.yml:

version: "3.8"
services:
  neo4j:
    image: "neo4j:4.3-enterprise"
    container_name: "neo4j"
    environment:
      NEO4J_ACCEPT_LICENSE_AGREEMENT: "yes"
      NEO4J_AUTH: "neo4j/test"
      NEO4J_dbms_connector_bolt_enabled: "true"
      NEO4J_dbms_ssl_policy_bolt_enabled: "true"
      NEO4J_dbms_ssl_policy_bolt_base__directory: "certificates/bolt"
      NEO4J_dbms_ssl_policy_bolt_private__key: "private.key"
      NEO4J_dbms_ssl_policy_bolt_public__certificate: "public.crt"
      NEO4J_dbms_ssl_policy_bolt_client__auth: "NONE"
      NEO4J_dbms_connector_bolt_tls__level: "REQUIRED"
      NEO4J_dbms_connector_https_enabled: "true"
      NEO4J_dbms_ssl_policy_https_enabled: "true"
      NEO4J_dbms_ssl_policy_https_base__directory: "certificates/https"
      NEO4J_dbms_ssl_policy_https_private__key: "private.key"
      NEO4J_dbms_ssl_policy_https_public__certificate: "public.crt"
      NEO4J_dbms_ssl_policy_https_client__auth: "NONE"
      NEO4J_dbms_connector_http_enabled: "false"
    ports:
      - "7473:7473"
      - "7687:7687"
    expose:
      - 7473
      - 7687
    volumes:
      - ${HOME}/neo4j/data:/data
      - ${HOME}/neo4j/logs:/logs
      - ${HOME}/neo4j/certificates:/var/lib/neo4j/certificates

Use these commands to get the certificates directory properly configured:

NEO4J_HOME=$HOME/neo4j

#### Setup SSL over Bolt ####
certsdir=$NEO4J_HOME/certificates
rm -rf $certsdir
mkdir -p $certsdir/bolt/trusted
mkdir -p $certsdir/bolt/revoked
chmod 755 $certsdir/bolt
chmod 755 $certsdir/bolt/revoked
chmod 755 $certsdir/bolt/trusted

# Generate private key and cert
openssl \
    req -x509 \
    -nodes \
    -newkey rsa:4096 \
    -keyout $certsdir/bolt/private.key \
    -out $certsdir/bolt/public.crt \
    -days 3650 \
    -subj '/C=<COUNTRY>/ST=<STATE>/L=<CITY>/CN=<DOMAIN>'

chmod 400 $certsdir/bolt/private.key
chmod 644 $certsdir/bolt/public.crt

# Also put cert in $NEO4J_HOME/certificates/bolt/trusted
cp $certsdir/bolt/public.crt $certsdir/bolt/trusted

mkdir -p $certsdir/https/trusted
mkdir -p $certsdir/https/revoked
chmod 755 $certsdir/https
chmod 755 $certsdir/https/trusted
chmod 755 $certsdir/https/revoked

# Copy private key and cert to https directory
cp $certsdir/bolt/private.key $certsdir/https/
cp $certsdir/bolt/public.crt $certsdir/https/
cp $certsdir/bolt/public.crt $certsdir/https/trusted