Neo4j Clustering needs help.(Implemeation AWS marketplace Neo4j causal cluster)

[kgwanghee_han]

Hello,

I have been set up AWS enterprise cluster using AWS Marketplace(AWS Marketplace: Neo4j Enterprise Causal Cluster)
(*Recently we have taken Startup license)

When I test Neo4j connection after complete create resources in AWS, all connection succeed(I can monitor 3 nodes(1 Leader , 2 Follower) in neo4j desktop.

But, when I reboot("sudo reboot" in ssh terminal) a specific EC2 instance, this instance couldn't restart(It means that the instance didn't join neo4j cluster, And I couldn't connect ssh to this EC2 instance)

I can read from system log in AWS EC2 console.

 e[0m] A start job is running for dev-xvdb.device (1min 28s / 1min 30s)
e[K[e[0me[0;31m*     e[0m] A start job is running for dev-xvdb.device (1min 29s / 1min 30s)
e[K[e[0;1;31m*e[0me[0;31m*  ​e[0m] A start job is running for dev-xvdb.device (1min 29s / 1min 30s)
e[K[e[0;1;31m TIME e[0m] Timed out waiting for device dev-xvdb.device.

......

[e[0;32m  OK  e[0m] Started Initial cloud-init job (metadata service crawler).
[e[0;32m  OK  e[0m] Reached target Network is Online.
        ​Starting Availability of block devices...
[e[0;32m  OK  e[0m] Reached target Remote File Systems (Pre).
[e[0;32m  OK  e[0m] Reached target Remote File Systems.
[e[0;32m  OK  e[0m] Reached target Cloud-config availability.
[e[0;32m  OK  e[0m] Started Availability of block devices.
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Press Enter for maintenance

I doubt that what is "dev-xvdb"?

I checked cloudformation template (https://s3.amazonaws.com/neo4j-cloudformation/neo4j-enterprise-stack-4.2.1.json).
In this template EBS doesn't create "xvdb", It create two volumns "/dev/sda1" and "/dev/sdb".

I think "/dev/sda1" will play a role as a data storage, but I can't find codes which initialization and mount this "/dev/sdb".

"BlockDeviceMappings": [
          {
            "DeviceName": "/dev/sda1",
            "Ebs": {
              "VolumeType": {
                "Ref": "VolumeType"
              },
              "VolumeSize": "100",
              "DeleteOnTermination": "true"
            }
          },
          {
            "DeviceName": "/dev/sdb",
            "Ebs": {
              "VolumeType": {
                "Ref": "VolumeType"
              },
              "VolumeSize": {
                "Ref": "VolumeSizeGB"
              },
              "Encrypted": {
                "Ref": "EncryptDataVolume"
              }
            }
          }
        ]

What is next step to solving this problem?

• Do modify cloudformation template?
• Or Set up other cluster options?
• Or I looking forward best(easy) solution to implement Neo4j Causal Cluster in AWS.

Thank you.

[kgwanghee_han]

Is there who can answer this questions?

Or I want to know where I can ask this problems.

[kgwanghee_han · Accepted solution]

Hello,

I figure out what is problems. In my case, I set EC2 instance type to "r5.xlarge".

But, I don't know "r5" instance is some different another general instances types in AWS. It means that "r5" is "Nitro" system which for high performance usecase. It also uses "nvme type EBS volumes".

So, AWS assigns different name in r5 instances which start with "nmve~", but "neo4j enterprise causal cluster" cloudformation hardcoded only "sda / xvdb".

I think neo4j should modify their offical cloudforamtion template, remove "r5" instance in "mapping" field OR set other "userdata" scripts , if user selects "r5" instances to support "nvme volumes"

Anyway, I can find problems, and I switched EC2 types to "r4" from "r5".

I hope that this post some helpful to others.

Thanks.