Neo4j 3.5.12 enterprise starting but not accepting any connections

[vvvasilev]

Hi,
I did an offline installation of neo4j-enterprise-3.5.12-1.noarch.rpm and cypher-shell-1.1.12-1.noarch.rpm on RHEL 7. I changed the neo4j.conf to :

#*****************************************************************
# Network connector configuration
#*****************************************************************

# With default configuration Neo4j only accepts local connections.
# To accept non-local connections, uncomment this line:
dbms.connectors.default_listen_address=0.0.0.0

# You can also choose a specific network interface, and configure a non-default
# port for each connector, by setting their individual listen_address.

# The address at which this server can be reached by its clients. This may be the server's IP address or DNS name, or
# it may be the address of a reverse proxy which sits in front of the server. This setting may be overridden for
# individual connectors below.
dbms.connectors.default_advertised_address=0.0.0.0

# You can also choose a specific advertised hostname or IP address, and
# configure an advertised port for each connector, by setting their
# individual advertised_address.

# Bolt connector
dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=OPTIONAL
dbms.connector.bolt.listen_address=0.0.0.0:7687

# HTTP Connector. There can be zero or one HTTP connectors.
dbms.connector.http.enabled=true
dbms.connector.http.listen_address=0.0.0.0:7474

# HTTPS Connector. There can be zero or one HTTPS connectors.
dbms.connector.https.enabled=true
dbms.connector.https.listen_address=0.0.0.0:7473

# Number of Neo4j worker threads.
#dbms.threads.worker_count=

#*****************************************************************
# SSL system configuration
#*****************************************************************

The neo4j service starts normally:

● neo4j.service - Neo4j Graph Database
   Loaded: loaded (/usr/lib/systemd/system/neo4j.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-21 13:33:41 CET; 2s ago
 Main PID: 78866 (java)
   CGroup: /system.slice/neo4j.service
           └─78866 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+Trust...

Nov 21 13:33:42 clrvxxx neo4j[78866]: logs:         /var/log/neo4j
Nov 21 13:33:42 clrvxxx neo4j[78866]: plugins:      /var/lib/neo4j/plugins
Nov 21 13:33:42 clrvxxx neo4j[78866]: import:       /var/lib/neo4j/import
Nov 21 13:33:42 clrvxxx neo4j[78866]: data:         /var/lib/neo4j/data
Nov 21 13:33:42 clrvxxx neo4j[78866]: certificates: /var/lib/neo4j/certificates
Nov 21 13:33:42 clrvxxx neo4j[78866]: run:          /var/run/neo4j
Nov 21 13:33:42 clrvxxx neo4j[78866]: Starting Neo4j.
Nov 21 13:33:42 clrvxxx neo4j[78866]: 2019-11-21 12:33:42.938+0000 INFO  ======== Neo4j 3.5.12 ========
Nov 21 13:33:42 clrvxxx neo4j[78866]: 2019-11-21 12:33:42.945+0000 INFO  Starting...
Nov 21 13:33:43 clrvxxx neo4j[78866]: 2019-11-21 12:33:43.802+0000 INFO  Initiating metrics...

However it doesn't accept neither local nor remote connections (tried via cypher shell and telnet on 7474). Firewall is stopped and disabled. There isn't any errors in the debug.log either:

2019-11-21 12:40:38.566+0000 INFO [o.n.i.d.DiagnosticsManager] --- STARTED diagnostics for NEO_STORE_RECORDS END ---
2019-11-21 12:40:38.567+0000 INFO [o.n.i.d.DiagnosticsManager] --- STARTED diagnostics for TRANSACTION_RANGE START ---
2019-11-21 12:40:38.567+0000 INFO [o.n.i.d.DiagnosticsManager] Transaction log:
2019-11-21 12:40:38.567+0000 INFO [o.n.i.d.DiagnosticsManager] Oldest transaction 2 found in log with version 0
2019-11-21 12:40:38.567+0000 INFO [o.n.i.d.DiagnosticsManager] --- STARTED diagnostics for TRANSACTION_RANGE END ---
2019-11-21 12:40:38.574+0000 INFO [o.n.i.d.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:StoreFiles START ---
2019-11-21 12:40:38.574+0000 INFO [o.n.i.d.DiagnosticsManager] Disk space on partition (Total / Free / Free %): 6431965184 / 5002346496 / 77
2019-11-21 12:40:38.574+0000 INFO [o.n.i.d.DiagnosticsManager] Storage files: (filename : modification date - size)
2019-11-21 12:40:38.576+0000 INFO [o.n.i.d.DiagnosticsManager]   index:
2019-11-21 12:40:38.577+0000 INFO [o.n.i.d.DiagnosticsManager]   - Total: 2019-07-18T13:27:41+0200 - 0.00 B
2019-11-21 12:40:38.577+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore: 2019-11-21T11:27:55+0100 - 8.00 kB
2019-11-21 12:40:38.577+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.counts.db.a: 2019-11-21T13:38:55+0100 - 928.00 B
2019-11-21 12:40:38.577+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.counts.db.b: 2019-11-21T13:38:51+0100 - 928.00 B
2019-11-21 12:40:38.577+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.labelscanstore.db: 2019-11-21T13:40:38+0100 - 48.00 kB
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.labeltokenstore.db: 2019-09-04T15:38:30+0200 - 8.00 kB
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.labeltokenstore.db.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.labeltokenstore.db.names: 2019-09-04T15:38:30+0200 - 8.00 kB
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.labeltokenstore.db.names.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.nodestore.db: 2019-09-08T11:49:54+0200 - 8.00 kB
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.nodestore.db.id: 2019-11-21T13:40:37+0100 - 1.22 kB
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.nodestore.db.labels: 2019-07-18T13:27:40+0200 - 8.00 kB
2019-11-21 12:40:38.578+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.nodestore.db.labels.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.579+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db: 2019-09-08T11:49:54+0200 - 7.97 kB
2019-11-21 12:40:38.579+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.arrays: 2019-09-08T11:49:54+0200 - 8.00 kB
2019-11-21 12:40:38.579+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.arrays.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.579+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.579+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.index: 2019-09-07T15:48:53+0200 - 8.00 kB
2019-11-21 12:40:38.579+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.index.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.579+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.index.keys: 2019-09-07T15:48:53+0200 - 8.00 kB
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.index.keys.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.strings: 2019-09-08T11:49:54+0200 - 8.00 kB
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.propertystore.db.strings.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshipgroupstore.db: 2019-07-18T13:27:40+0200 - 8.00 kB
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshipgroupstore.db.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshipstore.db: 2019-09-08T11:49:54+0200 - 15.94 kB
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshipstore.db.id: 2019-11-21T13:40:37+0100 - 1.34 kB
2019-11-21 12:40:38.580+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshiptypestore.db: 2019-08-07T14:30:01+0200 - 8.00 kB
2019-11-21 12:40:38.581+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshiptypestore.db.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.581+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshiptypestore.db.names: 2019-08-07T14:30:01+0200 - 8.00 kB
2019-11-21 12:40:38.581+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.relationshiptypestore.db.names.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.581+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.schemastore.db: 2019-09-07T11:27:13+0200 - 8.00 kB
2019-11-21 12:40:38.581+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.schemastore.db.id: 2019-11-21T13:40:37+0100 - 9.00 B
2019-11-21 12:40:38.581+0000 INFO [o.n.i.d.DiagnosticsManager]   neostore.transaction.db.0: 2019-11-21T13:38:55+0100 - 195.91 kB
2019-11-21 12:40:38.581+0000 INFO [o.n.i.d.DiagnosticsManager]   profiles:
2019-11-21 12:40:38.584+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.2250.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.584+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.counts.db.a.1127.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.584+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.counts.db.b.1122.cacheprof: 2019-11-21T13:38:47+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.labelscanstore.db.2251.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.labeltokenstore.db.2249.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.labeltokenstore.db.names.2249.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.nodestore.db.2248.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.nodestore.db.labels.2248.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.propertystore.db.2248.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.propertystore.db.arrays.2248.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.585+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.propertystore.db.index.2248.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.586+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.propertystore.db.index.keys.2248.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.586+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.propertystore.db.strings.2248.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.586+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.relationshipgroupstore.db.2249.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.586+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.relationshipstore.db.2249.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.relationshiptypestore.db.2249.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.relationshiptypestore.db.names.2249.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]     neostore.schemastore.db.2249.cacheprof: 2019-11-21T13:38:51+0100 - 21.00 B
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]   - Total: 2019-11-21T13:38:51+0100 - 378.00 B
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]   schema:
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]     index:
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]       lucene:
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]         1:
2019-11-21 12:40:38.587+0000 INFO [o.n.i.d.DiagnosticsManager]           1:
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]             segments_1t6: 2019-11-21T13:38:55+0100 - 87.00 B
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]             write.lock: 2019-09-07T11:27:05+0200 - 0.00 B
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]           - Total: 2019-11-21T13:38:55+0100 - 87.00 B
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]           failure-message: 2019-09-07T11:27:05+0200 - 16.00 kB
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]         - Total: 2019-09-07T11:27:05+0200 - 16.08 kB
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]       - Total: 2019-09-07T11:27:05+0200 - 16.08 kB
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]     - Total: 2019-09-07T11:27:05+0200 - 16.08 kB
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]     label:
2019-11-21 12:40:38.588+0000 INFO [o.n.i.d.DiagnosticsManager]     - Total: 2019-11-21T11:27:55+0100 - 0.00 B
2019-11-21 12:40:38.589+0000 INFO [o.n.i.d.DiagnosticsManager]   - Total: 2019-09-07T11:27:05+0200 - 16.08 kB
2019-11-21 12:40:38.589+0000 INFO [o.n.i.d.DiagnosticsManager]   store_lock: 2019-07-18T13:27:40+0200 - 0.00 B
2019-11-21 12:40:38.589+0000 INFO [o.n.i.d.DiagnosticsManager] Storage summary:
2019-11-21 12:40:38.589+0000 INFO [o.n.i.d.DiagnosticsManager]   Total size of store: 392.74 kB
2019-11-21 12:40:38.589+0000 INFO [o.n.i.d.DiagnosticsManager]   Total size of mapped files: 175.90 kB
2019-11-21 12:40:38.589+0000 INFO [o.n.i.d.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:StoreFiles END ---
2019-11-21 12:40:38.589+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Fulfilling of requirement 'Database available' makes database graph.db available.
2019-11-21 12:40:38.590+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database graph.db is ready.
2019-11-21 12:40:38.595+0000 INFO [o.n.k.i.DatabaseHealth] Database health set to OK
2019-11-21 12:40:38.597+0000 INFO [o.n.k.i.p.PageCacheWarmer] Page cache warmup started.
2019-11-21 12:40:38.686+0000 INFO [o.n.k.i.p.PageCacheWarmer] Page cache warmup completed. 15 pages loaded. Duration: 88ms.

[mark_needham]

It doesn't look like the database has actually started yet. Normally you would see a line in the log file telling you that it's available on localhost:7474.

Is it just hanging on that last line you pasted, or is there more output after that?

[vvvasilev]

It is just hanging on that. When I do a systemctl restart neo4j , same thing

[dana_canzano]

out of curiosity do you have APOC installed? i.e. any jar files in /var/lib/neo4j/plugins, if so what version of APOC?

[vvvasilev]

This one was on top of a neo4j 3.1 installation. So I installed a new Linux and there it works fine