Neo4j

mike_blum_neo4j · ‎02-13-2020

With Bolt configured like this cypher-shell with Neo4j 4.0 seems to not like the self-signed certificates anymore:

./bin/cypher-shell --database system -u neo4j -p $secret --encryption true --debug
org.neo4j.driver.exceptions.SecurityException: Failed to establish secured connection with the server
	at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
	at org.neo4j.driver.internal.InternalSession.run(InternalSession.java:69)
	at org.neo4j.driver.internal.InternalSession.run(InternalSession.java:51)
	at org.neo4j.driver.internal.AbstractQueryRunner.run(AbstractQueryRunner.java:37)
	at org.neo4j.driver.internal.AbstractQueryRunner.run(AbstractQueryRunner.java:55)
	at org.neo4j.shell.state.BoltStateHandler.reconnect(BoltStateHandler.java:196)
	at org.neo4j.shell.state.BoltStateHandler.reconnect(BoltStateHandler.java:173)
	at org.neo4j.shell.state.BoltStateHandler.connect(BoltStateHandler.java:161)
	at org.neo4j.shell.CypherShell.connect(CypherShell.java:143)
	at org.neo4j.shell.Main.connectMaybeInteractively(Main.java:129)
	at org.neo4j.shell.Main.startShell(Main.java:92)
	at org.neo4j.shell.Main.main(Main.java:41)
	Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause
		at org.neo4j.driver.internal.async.connection.HandshakeHandler.transformError(HandshakeHandler.java:195)
		at org.neo4j.driver.internal.async.connection.HandshakeHandler.exceptionCaught(HandshakeHandler.java:96)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268)
		at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1095)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:831)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:376)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at org.neo4j.driver.internal.shaded.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
		at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
		at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1499)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1513)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1397)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
	at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
	at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
	at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at org.neo4j.driver.internal.shaded.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
	at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
	at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:623)
	... 36 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
	... 42 more

Here are the relevant sections from the neo4j.conf:

# Bolt connector
dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=REQUIRED
dbms.connector.bolt.listen_address=0.0.0.0:7687
dbms.connector.bolt.advertised_address=10.132.13.45:7687

# SSL settings (dbms.ssl.policy.<scope>.*)
#  .base_directory       Base directory for SSL policies paths. All relative paths within the
#                        SSL configuration will be resolved from the base dir.
#
#  .private_key          A path to the key file relative to the '.base_directory'.
#
#  .private_key_password The password for the private key.
#
#  .public_certificate   A path to the public certificate file relative to the '.base_directory'.
#
#  .trusted_dir          A path to a directory containing trusted certificates.
#
#  .revoked_dir          Path to the directory with Certificate Revocation Lists (CRLs).
#
#  .verify_hostname      If true, the server will verify the hostname that the client uses to connect with. In order
#                        for this to work, the server public certificate must have a valid CN and/or matching
#                        Subject Alternative Names.
#
#  .client_auth          How the client should be authorized. Possible values are: 'none', 'optional', 'require'.
#
#  .tls_versions         A comma-separated list of allowed TLS versions. By default only TLSv1.2 is allowed.
#
#  .trust_all            Setting this to 'true' will ignore the trust truststore, trusting all clients and servers.
#                        Use of this mode is discouraged. It would offer encryption but no security.
#
#  .ciphers              A comma-separated list of allowed ciphers. The default ciphers are the defaults of
#                        the JVM platform.

# Bolt SSL configuration
dbms.ssl.policy.bolt.enabled=true
dbms.ssl.policy.bolt.base_directory=/etc/certs
dbms.ssl.policy.bolt.private_key=private.key
dbms.ssl.policy.bolt.public_certificate=public.crt

# Https SSL configuration
dbms.ssl.policy.https.enabled=true
dbms.ssl.policy.https.base_directory=/etc/certs
dbms.ssl.policy.https.private_key=private.key
dbms.ssl.policy.https.public_certificate=public.crt

# Cluster SSL configuration
dbms.ssl.policy.cluster.enabled=true
dbms.ssl.policy.cluster.base_directory=/etc/certs
dbms.ssl.policy.cluster.private_key=private.key
dbms.ssl.policy.cluster.public_certificate=public.crt

# Backup SSL configuration
dbms.ssl.policy.backup.enabled=true
dbms.ssl.policy.backup.base_directory=/etc/certs
dbms.ssl.policy.backup.private_key=private.key
dbms.ssl.policy.backup.public_certificate=public.crt

Looking at the cypher-shell docs (this is on the same box as the database) there doens't appear to be an easy way to tell cypher-shell about the certificate. Is there a way to enable encrypted Bolt connections without needing to have all clients (cypher-shell, java apps, etc) needing to know the certificate via JKS or other PKI implementations?

david_allen · ‎02-14-2020

Due to a change in 4.0, the server doesn't come packaged with a self-signed cert anymore, so the reason this isn't working is because you don't have a cert at all on the server side.

There's a PR open right now to add the ability to generate these certs for 4.0 in docker. I'm pointing you to this just because it has a bit of shell code showing you how you can generate your own.

mike_blum_neo4j · ‎02-14-2020

Interesting! Thanks for the bash script. Right now I have this:

# Bolt SSL configuration
dbms.ssl.policy.bolt.enabled=true
dbms.ssl.policy.bolt.base_directory=/etc/certs
dbms.ssl.policy.bolt.private_key=private.key
dbms.ssl.policy.bolt.public_certificate=public.crt

And here is the directory structure for my self-signed cert that i generated and loaded onto the box:

ls -la /etc/certs/
total 24
drwxr-xr-x  4 neo4j neo4j 4096 Feb 13 22:21 .
drwxr-xr-x 98 root  root  4096 Feb 13 23:22 ..
-rw-r--r--  1 neo4j neo4j 1257 Feb 13 22:21 fullchain.pem
lrwxrwxrwx  1 root  root    22 Feb 13 22:21 private.key -> /etc/certs/privkey.pem
-rw-r--r--  1 neo4j neo4j 1704 Feb 13 22:21 privkey.pem
lrwxrwxrwx  1 root  root    24 Feb 13 22:21 public.crt -> /etc/certs/fullchain.pem
drwxr-xr-x  2 neo4j neo4j 4096 Feb 13 22:21 revoked
drwxr-xr-x  2 neo4j neo4j 4096 Feb 13 22:21 trusted

ls -lar /etc/certs/trusted/
total 8
lrwxrwxrwx 1 root  root    24 Feb 13 22:21 public.crt -> /etc/certs/fullchain.pem

I can load the browser at https://addr:7473 and login from there after accepting the 7473 and 7687 danger notices in firefox. But for some reason cypher-shell has a difference of opinion.

david_allen · ‎02-14-2020

I'm at the end of my knowledge here, but I believe some separate toggle may be needed to make cypher-shell specifically accept self-signed certificates. At the driver level that is certainly the case, but sorry I don't know how the driver options map to cypher-shell flags.

mike_blum_neo4j · ‎03-06-2020

This still seems to be an issue in neo4j 4.0.1. Opened a github issue to track it: https://github.com/neo4j/cypher-shell/issues/209

Graham_Lea · ‎04-27-2020

Is this possibly because browsers give the option of accepting a dodgy certificate (i.e. out of date or mismatching Common Name) to establish an encrypted (but potentially MITM'd) connection, but cypher-shell doesn't provide a similar option, so its TLS stack always aborts if the certificate isn't perfect?

bawa_deepak · ‎07-01-2021

Bit old post now.
But solution that worked for me: I had to add the cert to cacerts of Java used by neo4j as that is the one used by scripts such as cypher-shell.
Cheers.

jo_nathan · ‎08-03-2022

To elaborate, here is how I was able to do it and why it works.

Requirements:

You have set up TLS configuration in neo4j, you own certificates.

dbms.connector.bolt.tls_level=REQUIRED

Explanation

cypher-shell uses Java cacerts. If the certificate of the neo4j server you want to connect to is not known to or not trusted by cacerts, cypher-shell will not allow to connect.
Use "keytool" to import the certificate into a keystore. I am running containers, therefore I used default store with -cacerts parameter. Replace path to certificate file with your path as configured in neo4j.conf. Alias can be chosen to your liking.

Default password is "changeit" (without quotation).

keytool -importcert -alias neo4jcert -cacerts -file /var/lib/neo4j/certificates/default/cert.pem

Keep in mind, that when connecting using encryption you can either specify the protocol or specify --encryption true/false not both. <HOST> must match name of certificate, it should not be IP.

cypher-shell -a bolt+s://<HOST>:<PORT>

cypher-shell -a <HOST>:<PORT> --encryption true

Expected Results:

cypher-shell -a <HOST>:<PORT> --encryption false

Connection to the database terminated. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver. Note that the default encryption setting has changed in Neo4j 4.0.

cypher-shell -a <HOST>:<PORT> --encryption true

prompt for username, password, connection succesful.

username: neo4j
password: *********
Connected to Neo4j using Bolt protocol version 4.4 at #REDACTED# as user neo4j

🙂 that's it for my setup.

Further reading:

tms · ‎08-20-2021

I'm running my Neo4J (enterprise edition) servers on a Linux platform (CentOS 7, soon to be RockyLinux), so my approach may not work for those in other platforms.

After arguing with self-signed certs for years, about a month ago I switched to LetsEncrypt + Certbot.

It's dirt simple -- download the zipfile for your platform, unzip it, and follow the directions. Just a heads-up from bitter first-hand experience -- be sure to open the https port with firewall-cmd (run firewall-cmd --permanent --add-service https from a root shell and reboot), and if you're on AWS EC2 like me, be sure to open 443 in your security group(s).

Certbot automates all the headaches of renewing and installing DV certificates (so far as I know it only works for DV). Each cert you get has a 90-day lifetime, and Certbot installs a cron job to automagically renew it. That helps limit any vulnerability to bad guys.

Once installed, run sudo certbot certificates to get the path information needed for various config files. I also had to fiddle with ownership and permissions so that non-root services can read what they need. Nothing too hard, though -- in my opinion, MUCH easier than all the self-signed malarky.

So far as I know, the browsers all happily accept these certificates and you'll end up with a green padlock on any website.

I'm here because I'm about to try moving my Neo4J connections from http to https.

Neo4j

Cypher-shell certificates

Requirements:

Explanation

Expected Results: