First time Cluster in AWS : Connection Refused

[mike_r_black]

I'm trying to set up my first cluster in AWS. I have 3 EC2 instances. Following the manual and it appears as if the cluster is established, but when I try to cypher-shell from one of the instances I get a Connection Refused. Any tips where to go about looking to try and figure out how to get this cluster going? I'll paste some what I've found that I think could be relevant. Some AWS setup notes:

• Each EC2 instance is in the same VPC and Security group
• All traffic is allowed within the Security Group so all ports between EC2s for the cluster to communicate should be allowed
• Cluster definitions are referencing the private IPv4 addresses

Service Status of Node 1

ubuntu@ip-172-31-24-72:~$ sudo service neo4j status
● neo4j.service - Neo4j Graph Database
   Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-25 04:03:07 UTC; 4min 29s ago
 Main PID: 6189 (java)
    Tasks: 53 (limit: 1152)
   CGroup: /system.slice/neo4j.service
           └─6189 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFina

Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:12.972+0000 INFO  My connection info: [
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Discovery:   listen=0.0.0.0:5000, advertised=localhost:5000,
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Transaction: listen=0.0.0.0:6000, advertised=localhost:6000,
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Raft:        listen=0.0.0.0:7000, advertised=localhost:7000,
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Client Connector Addresses: bolt://localhost:7687,http://localhost:7474,https://localhost:7473
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]: ]
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:12.976+0000 INFO  Discovering other core members in initial members set: [172.31.23.197:5000, 172.31.29.208:5000]
Aug 25 04:03:44 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:44.297+0000 INFO  Bound to cluster with id 13a49e26-3218-4a54-9974-0662e7e530e1
Aug 25 04:03:44 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:44.303+0000 INFO  Discovered core member at 172.31.29.208:5000
Aug 25 04:03:49 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:49.212+0000 INFO  Discovered core member at 172.31.23.197:5000

Service Status Node 2

ubuntu@ip-172-31-29-208:~$ sudo service neo4j status
● neo4j.service - Neo4j Graph Database
   Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-25 04:03:33 UTC; 34min ago
 Main PID: 2667 (java)
    Tasks: 53 (limit: 1152)
   CGroup: /system.slice/neo4j.service
           └─2667 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFina

Aug 25 04:03:38 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:38.658+0000 INFO  Discovering other core members in initial members set: [172.31.23.197:5000, 172.31.24.72:5000]
Aug 25 04:03:48 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:48.572+0000 INFO  Bound to cluster with id 13a49e26-3218-4a54-9974-0662e7e530e1
Aug 25 04:03:48 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:48.635+0000 INFO  Discovered core member at 172.31.24.72:5000
Aug 25 04:03:49 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:49.211+0000 INFO  Discovered core member at 172.31.23.197:5000
Aug 25 04:14:04 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:14:04.353+0000 WARN  Lost core member at 172.31.23.197:5000
Aug 25 04:14:14 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:14:14.647+0000 INFO  Discovered core member at 172.31.23.197:5000
Aug 25 04:24:29 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:24:29.167+0000 WARN  Lost core member at 172.31.23.197:5000
Aug 25 04:24:39 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:24:39.741+0000 INFO  Discovered core member at 172.31.23.197:5000
Aug 25 04:34:53 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:34:53.722+0000 WARN  Lost core member at 172.31.23.197:5000
Aug 25 04:35:04 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:35:04.235+0000 INFO  Discovered core member at 172.31.23.197:5000

Service Status Node 3

ubuntu@ip-172-31-23-197:~$ sudo service neo4j status
● neo4j.service - Neo4j Graph Database
   Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-25 04:34:53 UTC; 3min 45s ago
 Main PID: 4062 (java)
    Tasks: 68 (limit: 1152)
   CGroup: /system.slice/neo4j.service
           └─4062 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFina

Aug 25 04:35:09 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:09.419+0000 INFO  Bound to cluster with id 13a49e26-3218-4a54-9974-0662e7e530e1
Aug 25 04:35:09 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:09.479+0000 INFO  Discovered core member at 172.31.24.72:5000
Aug 25 04:35:09 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:09.496+0000 INFO  Discovered core member at 172.31.29.208:5000
Aug 25 04:35:38 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:38.954+0000 INFO  Waiting to hear from leader...
Aug 25 04:36:06 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:36:06.955+0000 INFO  Waiting to hear from leader...
Aug 25 04:36:34 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:36:34.956+0000 INFO  Waiting to hear from leader...
Aug 25 04:37:02 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:37:02.957+0000 INFO  Waiting to hear from leader...
Aug 25 04:37:30 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:37:30.959+0000 INFO  Waiting to hear from leader...
Aug 25 04:37:58 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:37:58.960+0000 INFO  Waiting to hear from leader...
Aug 25 04:38:26 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:38:26.961+0000 INFO  Waiting to hear from leader...

I don't know why the 3rd node is waiting to hear from the leader?

[stefan_armbrust · Accepted solution]

I guess the advertised address settings are missing. Amend to your neo4j.conf file:

dbms.connectors.default_advertised_address=<internal_ip_address_of_this_host>

Note: if your clients (e.g. neo4j browser) are sitting outside on the public interface you might add as well:

dbms.connector.bolt.advertised_address=<external_ip_of_this_host>
dbms.connector.http.advertised_address=<external_ip_of_this_host>
dbms.connector.https.advertised_address=<external_ip_of_this_host>

[mike_r_black]

Thanks! That solved it!