Cluster is wrong

[drmartiner]

Hello everyone,

I have many problems with running simple cluster. I need to solve simple problem. I have docker-compose file:

version: '3'

networks:
  lan:

services:
  core1:
    image: neo4j:4.1-enterprise
    networks:
      - lan
    ports:
      - 7474:7474
      - 6477:6477
      - 7687:7687
    volumes:
      - $HOME/neo4j/neo4j-core401/data:/data
      - $HOME/neo4j/neo4j-core401/logs:/logs
    environment:
      - NEO4J_AUTH=neo4j/changeme
      - NEO4J_dbms_mode=CORE
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes      
      - NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
      - NEO4J_dbms_connector_http_listen__address=:7474
      - NEO4J_dbms_connector_https_listen__address=:6477
      - NEO4J_dbms_connector_bolt_listen__address=:7687
      - NEO4J_causal__clustering_discovery__advertised__address=core1:5000
      - NEO4J_causal__clustering_transaction__advertised__address=core1:6000
      - NEO4J_causal__clustering_raft__advertised__address=core1:7000
      - NEO4J_dbms_memory_heap_initial__size=1024m
      - NEO4J_dbms_memory_heap_max__size=1024m
      - NEO4J_dbms_memory_pagecache_size=1g
      - NEO4J_dbms_directories_import=import
      - NEO4J_dbms_logs_query_threshold=0s
#      - NEO4JLABS_PLUGINS=["apoc"]

  core2:
    image: neo4j:4.1-enterprise
    networks:
      - lan
    ports:
      - 127.0.0.1:7475:7475
      - 127.0.0.1:6478:6478
      - 127.0.0.1:7688:7688
    volumes:
      - $HOME/neo4j/neo4j-core402/data:/data
      - $HOME/neo4j/neo4j-core402/logs:/logs
    environment:
      - NEO4J_AUTH=neo4j/changeme
      - NEO4J_dbms_mode=CORE
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes      
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
      - NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
      - NEO4J_causal__clustering_discovery__advertised__address=core2:5000
      - NEO4J_causal__clustering_transaction__advertised__address=core2:6000
      - NEO4J_causal__clustering_raft__advertised__address=core2:7000
      - NEO4J_dbms_connector_http_listen__address=:7475
      - NEO4J_dbms_connector_https_listen__address=:6478
      - NEO4J_dbms_connector_bolt_listen__address=:7688
      - NEO4J_dbms_memory_heap_initial__size=1024m
      - NEO4J_dbms_memory_heap_max__size=1024m
      - NEO4J_dbms_memory_pagecache_size=1g
      - NEO4J_dbms_directories_import=import
      - NEO4J_dbms_logs_query_threshold=0s
#      - NEO4JLABS_PLUGINS=["apoc"]

  core3:
    image: neo4j:4.1-enterprise
    networks:
      - lan
    ports:
      - 7476:7476
      - 6479:6479      
      - 7689:7689
    volumes:
      - $HOME/neo4j/neo4j-core403/data:/data
      - $HOME/neo4j/neo4j-core403/logs:/logs
    environment:
      - NEO4J_AUTH=neo4j/changeme
      - NEO4J_dbms_mode=CORE
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
      - NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
      - NEO4J_causal__clustering_discovery__advertised__address=core3:5000
      - NEO4J_causal__clustering_transaction__advertised__address=core3:6000
      - NEO4J_causal__clustering_raft__advertised__address=core3:7000
      - NEO4J_dbms_connector_http_listen__address=:7476
      - NEO4J_dbms_connector_https_listen__address=:6479      
      - NEO4J_dbms_connector_bolt_listen__address=:7689
      - NEO4J_dbms_memory_heap_initial__size=1024m
      - NEO4J_dbms_memory_heap_max__size=1024m
      - NEO4J_dbms_memory_pagecache_size=1g
      - NEO4J_dbms_directories_import=import
      - NEO4J_dbms_logs_query_threshold=0s
#      - NEO4JLABS_PLUGINS=["apoc"]

  read1:
    image: neo4j:4.1-enterprise
    networks:
      - lan
    ports:
      - 127.0.0.1:7479:7479
      - 127.0.0.1:9479:9479
      - 127.0.0.1:9689:9689
    volumes:
      - $HOME/neo4j/neo4j-read1/data:/data
      - $HOME/neo4j/neo4j-read1/logs:/logs
    environment:
      - NEO4J_AUTH=neo4j/changeme
      - NEO4J_dbms_mode=READ_REPLICA
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
      - NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
      - NEO4J_dbms_connector_http_listen__address=:7479
      - NEO4J_dbms_connector_https_listen__address=:9479
      - NEO4J_dbms_connector_bolt_listen__address=:9689
#      - NEO4JLABS_PLUGINS=["apoc"]

I took this example from GitHub's gist. The fist time was worked, but the others times ware failure: No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER

Best regards,
Neo4j newcomers

[drmartiner]

docker's logs:

core2_1  | 2020-07-13T08:14:39.222450513Z 2020-07-13 08:14:39.222+0000 INFO  ======== Neo4j 4.1.0 ========
core2_1  | 2020-07-13T08:15:00.548543583Z 2020-07-13 08:15:00.548+0000 INFO  Bound database 'system' to raft with id '00000000-0000-0000-0000-000000000001' as member id '4e6740e2-a358-4a07-b617-75cba078c076', found on disk.
core2_1  | 2020-07-13T08:15:00.803884828Z 2020-07-13 08:15:00.803+0000 INFO  Connected to core1/172.21.0.5:7000 [raft version:3.0]
core2_1  | 2020-07-13T08:15:03.587373767Z 2020-07-13 08:15:03.587+0000 INFO  Bound database 'neo4j' to raft with id 'ae960579-afef-43a6-b2e7-d986eb9c65aa' as member id '4e6740e2-a358-4a07-b617-75cba078c076', found on disk.
core2_1  | 2020-07-13T08:15:07.897072639Z 2020-07-13 08:15:07.896+0000 INFO  Called db.clearQueryCaches(): Query cache already empty.
core2_1  | 2020-07-13T08:15:18.161540682Z 2020-07-13 08:15:18.161+0000 INFO  Connected to core3/172.21.0.4:7000 [raft version:3.0]
core2_1  | 2020-07-13T08:15:24.875681649Z 2020-07-13 08:15:24.875+0000 INFO  Sending metrics to CSV file at /var/lib/neo4j/metrics
core2_1  | 2020-07-13T08:15:24.964610250Z 2020-07-13 08:15:24.964+0000 INFO  Bolt enabled on 0.0.0.0:7688.
core2_1  | 2020-07-13T08:15:27.017520656Z 2020-07-13 08:15:27.017+0000 INFO  Remote interface available at http://localhost:7475/
core2_1  | 2020-07-13T08:15:27.018300790Z 2020-07-13 08:15:27.018+0000 INFO  Started.```

[david_allen]

In Neo4j's causal cluster architecture, you have 1 leader and 2 followers. All writes must go through the leader. Probably what's happening here is you're using a client to connect to one of the database members that isn't the leader -- if you do that, then write operations will fail.

The right way to go here is to connect using the neo4j:// URI scheme. This will allow your driver to "smart route" your queries so that writes go to the leader, and assuming your database is correctly configured, this should make it work.

If you want much more information on how querying clusters works, read this:

Medium – 8 Mar 19

Querying Neo4j Clusters

How Neo4j clusters and smart query routing work!

Reading time: 8 min read