Remote neo4j server - port 7474 connection refused

[mcbr]

I cannot connect from my local machine to a remote server using the server host name and port 7474 in the browser (HTTP, not HTTPS).

I have the remote server running Debian with the following version:

Linux mem 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux

I installed Neo4j Enterprise and have the following network connector configuration which I think is set to accept connections from outside localhost:

vim /etc/neo4j/neo4j.conf

#*****************************************************************
# Network connector configuration
#*****************************************************************

# With default configuration Neo4j only accepts local connections.
# To accept non-local connections, uncomment this line:
dbms.default_listen_address=0.0.0.0

# You can also choose a specific network interface, and configure a non-default
# port for each connector, by setting their individual listen_address.

# The address at which this server can be reached by its clients. This may be the server's IP address or DNS name, or
# it may be the address of a reverse proxy which sits in front of the server. This setting may be overridden for
# individual connectors below.
dbms.default_advertised_address=0.0.0.0

# You can also choose a specific advertised hostname or IP address, and
# configure an advertised port for each connector, by setting their
# individual advertised_address.

# By default, encryption is turned off.
# To turn on encryption, an ssl policy for the connector needs to be configured
# Read more in SSL policy section in this file for how to define a SSL policy.

# Bolt connector
dbms.connector.bolt.enabled=true
#dbms.connector.bolt.tls_level=DISABLED
#dbms.connector.bolt.listen_address=:7687
#dbms.connector.bolt.advertised_address=:7687

# HTTP Connector. There can be zero or one HTTP connectors.
dbms.connector.http.enabled=true
#dbms.connector.http.listen_address=:7474
#dbms.connector.http.advertised_address=:7474

# HTTPS Connector. There can be zero or one HTTPS connectors.
dbms.connector.https.enabled=false
#dbms.connector.https.listen_address=:7473
#dbms.connector.https.advertised_address=:7473

# Cluster Routing Connector. Enables the opening of an additional port to allow
# for internal communication using the same security configuration as CLUSTER
#dbms.routing.enabled=false

# Customize the listen address and advertised address used for the routing connector.
#dbms.routing.listen_address=0.0.0.0:7688
#dbms.routing.advertised_address=:7688

# Number of Neo4j worker threads.
#dbms.threads.worker_count=

I then launch neo4j with:

sudo systemctl start neo4j

I can see that the addresses and ports are being listened:

sudo lsof -i -P -n

java 58622 neo4j 1182u IPv6 17299554 0t0 TCP 127.0.0.1:6362 (LISTEN)
java 58622 neo4j 1272u IPv6 17235865 0t0 TCP *:7687 (LISTEN)
java 58622 neo4j 1274u IPv6 17235870 0t0 TCP *:7474 (LISTEN)

Another check with netstat:

sudo netstat -ltnp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1154/sshd
tcp6 0 0 127.0.0.1:6362 :::* LISTEN 58622/java
tcp6 0 0 :::7687 :::* LISTEN 58622/java
tcp6 0 0 :::7474 :::* LISTEN 58622/java
tcp6 0 0 :::22 :::* LISTEN 1154/sshd

From what I can see, neo4j is using ipv6 (tcp6) but from what I read online, this would be able to receive connections using ipv4 as well?
I note that I can ssh successfully into the remote server, but I can't access any of the ports of the network connector configuration from my local machine.

If I log into the remote server via ssh, I can test the response of the Neo4j server web interface:

ssh my_user@server_host_name

wget http://server_host_name:7474/

{
"bolt_routing" : "neo4j://server_host_name:7687",
"transaction" : "http://server_host_name7474/db/{databaseName}/tx",
"bolt_direct" : "bolt://server_host_name:7687",
"neo4j_version" : "4.2.1",
"neo4j_edition" : "enterprise"
}

However, if I try to access remotely, I get get this on Firefox:

Firefox can’t establish a connection to the server at server_host_name:7474.

Now lets test using nc from my local machine.
Port 22 for ssh of the remote server works fine:

nc -zvw10 server_host_name 22
Connection to server_host_name 22 port [tcp/ssh] succeeded!

But port 7474 does not:

nc -zvw10 server_host_name 7474
nc: connect to server_host_name port 7474 (tcp) failed: Connection refused

I have considered this to be a firewall problem, but this is what I get from iptables inside the remote server:

sudo iptables --list-rules -v

-P INPUT ACCEPT -c 0 0
-P FORWARD ACCEPT -c 0 0
-P OUTPUT ACCEPT -c 0 0

So, I believe the following hold true:

• There are no firewall rules prohibiting the connection.
• neo4j is listening on the ports defined in the configuration.

What I gather from this is that perhaps neo4j is using ipv6 instead of ipv4 (from the netstat output) and that could be a problem.

If that is the case, how do I force it to use ipv4 only?
If anyone has additional suggestions I would appreciate it because I think I've gone through all possibilities already.

Thanks for your attention.

[david_allen]

The problem is probably right here. Read the comment:

# The address at which this server can be reached by its clients. This may be the server's IP address or DNS name, or
# it may be the address of a reverse proxy which sits in front of the server. This setting may be overridden for
# individual connectors below.
dbms.default_advertised_address=0.0.0.0

The way Neo4j clients work, the server needs to know what it's externally valid IP address is, and that's what this setting controls. 0.0.0.0 isn't a valid IP address, so probably your client is getting confused by Neo4j's routing table.

So I think your solution is going to be to set that setting to the externally valid IP that your client (the browser) uses to contact the server.

Full gory details on why are here: https://medium.com/neo4j/querying-neo4j-clusters-7d6fde75b5b4

[mcbr]

Hello David,

Thanks for the help.
I now have the following configuration:

#*****************************************************************
# Network connector configuration
#*****************************************************************

# With default configuration Neo4j only accepts local connections.
# To accept non-local connections, uncomment this line:
dbms.default_listen_address=0.0.0.0

# You can also choose a specific network interface, and configure a non-default
# port for each connector, by setting their individual listen_address.

# The address at which this server can be reached by its clients. This may be the server's IP address or DNS name, or
# it may be the address of a reverse proxy which sits in front of the server. This setting may be overridden for
# individual connectors below.
dbms.default_advertised_address=server_host_name

# You can also choose a specific advertised hostname or IP address, and
# configure an advertised port for each connector, by setting their
# individual advertised_address.

# By default, encryption is turned off.
# To turn on encryption, an ssl policy for the connector needs to be configured
# Read more in SSL policy section in this file for how to define a SSL policy.

# Bolt connector
dbms.connector.bolt.enabled=true
#dbms.connector.bolt.tls_level=DISABLED
dbms.connector.bolt.listen_address=0.0.0.0:7687
dbms.connector.bolt.advertised_address=server_host_name:7687

# HTTP Connector. There can be zero or one HTTP connectors.
dbms.connector.http.enabled=true
dbms.connector.http.listen_address=0.0.0.0:7474
dbms.connector.http.advertised_address=server_host_name:7474

# HTTPS Connector. There can be zero or one HTTPS connectors.
dbms.connector.https.enabled=false
#dbms.connector.https.listen_address=:7473
#dbms.connector.https.advertised_address=:7473

# Cluster Routing Connector. Enables the opening of an additional port to allow
# for internal communication using the same security configuration as CLUSTER
#dbms.routing.enabled=false

# Customize the listen address and advertised address used for the routing connector.
#dbms.routing.listen_address=0.0.0.0:7688
#dbms.routing.advertised_address=:7688

# Number of Neo4j worker threads.
#dbms.threads.worker_count=

With this I was able to access the Neo4j web app running on the remote server server_host_name from my local computer using Firefox.
However, now I am getting the following problem when trying to log in:

ServiceUnavailable: WebSocket connection failure.
Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver.
Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems.
If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket readyState is: 3

I don't have encryption enabled yet, it is what I intend to do after I manage to connect with the basic configuration.
While looking at Firefox's web developer console, I only find this (and am unable to observe source JavaScript code):

Firefox can’t establish a connection to the server at ws://server_host_name:7687/. neo4j-driver.chunkhash.bundle.js:1:120212

Do you have suggestions?

Thanks again for your time.

[david_allen · Accepted solution]

So the next issue based on that error message is likely an SSL cert trust issue. See discussion & fixes here:

Troubleshooting Connection Issues to Neo4j Browser

Troubleshooting Connection Issues to Neo4j (including Browser, Cypher Shell, and Driver Applications) This post describes common issues users may encounter in connecting Neo4j Browser or cypher-shell to a Neo4j database, and how to address them. Connection Timeout Symptom: connection attempts lag for a long time, and then fail with connection timed out errors. Example: $ cypher-shell -a 37.204.217.197 -u neo4j -p myPassword connection timed out: /37.204.217.197:7687 Troubleshooting steps: E…

[mcbr]

I followed your guide for LetsEncrypt to get the certificate and configure it for the Neo4j installation.
I am now able to access the Neo4j app via HTTPS (7473) on the browser.

However, when I log in, I briefly see different status messages at the top:

Still connecting...
Server is taking a long time to respond...
EDIT: there was a configuration missing on the firewall. Done!

Then the old error message appears at the bottom:

ServiceUnavailable: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket readyState is: 3

I speculate a there is a timeout taking place, though I didn't find anything in the Neo4j logs.

On the developer console of Firefox I got the following too:

Firefox can’t establish a connection to the server at wss://mem.inesc-id.pt:7687/.

Thanks again for the help.

[david_allen]

These error messages about "Still connecting..." don't look like they came from the neo4j logs. I'm not sure what's happening there. Please consider opening a different topic with this connection error, and provide more surrounding context from the log file that it was a part of.