This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Head's Up! These forums are read-only. All users and content have migrated. Please join us at community.neo4j.com.
- Neo4j
- Technical Discussions
- Neo4j Graph Platform
- Re: Neo4j HA Pods in Statefulset can't connect to ...
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Printer Friendly Page
03-13-2019 07:10 AM
Hi,
My environment is Openshift v3.9 (~Kubernetes 1.9)
I deployed based on the templates in this helm package
I only skipped readreplicas-deployment.yaml and poddisruptionbudget.yaml so I have Statefulset of Core Neo4j pods and headless service + secret.
The issue is after I scale up cluster to 3 replicas every pod got stuck in the state blow:
++ hostname -f
+ export NEO4J_dbms_connectors_default__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local
+ NEO4J_dbms_connectors_default__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local
++ hostname -f
+ export NEO4J_causal__clustering_discovery__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:5000
+ NEO4J_causal__clustering_discovery__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:5000
++ hostname -f
+ export NEO4J_causal__clustering_transaction__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000
+ NEO4J_causal__clustering_transaction__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000
++ hostname -f
+ export NEO4J_causal__clustering_raft__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000
+ NEO4J_causal__clustering_raft__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000
+ exec /docker-entrypoint.sh neo4j
Active database: graph.db
Directories in use:
home: /var/lib/neo4j
config: /var/lib/neo4j/conf
logs: /var/lib/neo4j/logs
plugins: /var/lib/neo4j/plugins
import: /var/lib/neo4j/import
data: /var/lib/neo4j/data
certificates: /var/lib/neo4j/certificates
run: /var/lib/neo4j/run
Starting Neo4j.
2019-03-13 14:03:47.313+0000 INFO ======== Neo4j 3.4.10 ========
2019-03-13 14:03:47.357+0000 INFO Starting...
2019-03-13 14:03:49.413+0000 INFO Initiating metrics...
2019-03-13 14:03:49.580+0000 INFO Resolved initial host 'neo4j.rag-uat.svc.cluster.local:5000' to [10.130.1.54:5000, 10.128.0.45:5000]
2019-03-13 14:03:49.611+0000 INFO My connection info: [
Discovery: listen=0.0.0.0:5000, advertised=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:5000,
Transaction: listen=0.0.0.0:6000, advertised=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000,
Raft: listen=0.0.0.0:7000, advertised=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000,
Client Connector Addresses: bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473
]
2019-03-13 14:03:49.611+0000 INFO Discovering cluster with initial members: [neo4j.rag-uat.svc.cluster.local:5000]
2019-03-13 14:03:49.611+0000 INFO Attempting to connect to the other cluster members before continuing...
As I googled around the history of the issue have an unresolved story
in here , here ,and here
As it is obvious from the logs I am using Bolt protocol instead of the original.
I did a DNS experiment from inside of one pod, here is the result:
/var/lib/neo4j # nslookup neo4j.rag-uat.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve
Name: neo4j.rag-uat.svc.cluster.local
Address 1: 10.128.0.45 neo4j-core-1.neo4j.rag-uat.svc.cluster.local
Address 2: 10.129.0.104 neo4j-core-2.neo4j.rag-uat.svc.cluster.local
Address 3: 10.130.1.54 neo4j-core-0.neo4j.rag-uat.svc.cluster.local
Solved! Go to Solution.
Labels:
- Labels:
-
Aura-and-Cloud
1 ACCEPTED SOLUTION
03-17-2019 11:25 PM
I previously was deploying with the files in the helm chart without using helm directly. maybe I was missing something or so. I was wrong that helm is not possible in Openshift. after I installed helm in openshift and deployed neo4j via helm everything got smooth and also it saves me time too.
thanks for your support
Reply
11 REPLIES 11
03-13-2019 07:27 AM
When you start new nodes, they need to hook into the cluster by locating their buddies. This is controlled by the setting "initial discovery members" which in your yaml should look something like:
- name: NEO4J_causal__clustering_initial__discovery__members
value: "{{ template "neo4j.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:5000"
First confirm that. Second thing to know is that the discovery member set is specifying only a single member by a single DNS address. As you've found, that DNS address looks up to 3 different A records (for the individual nodes). A key piece of magic is this:
- name: NEO4J_causal__clustering_discovery__type
value: DNS
Can you verify this too? This tells neo4j that the DNS address isn't literally a single node, but a multi-lookup to find all of the other cluster members.
If you're attempting to connect to other cluster members and getting stuck, this seems to be the problem. The nslookup verified that you have good DNS setup (via the service in the helm charts)
The missing bit is likely to be the discovery_type = DNS above. Please check.
Relevant docs here to consult, in case I've missed what the problem is -- something else here should help:
03-13-2019 07:46 AM
I don't know if it's a good news or bad one 
I confirm that both specifications are set as you expressed in previous reply.
here is my yaml
- name: NEO4J_causal__clustering_discovery__type
value: DNS
- name: NEO4J_causal__clustering_initial__discovery__members
value: 'neo4j.rag-uat.svc.cluster.local:5000'
NEO4J_causal__clustering_initial__discovery__members is pointing to the address of kubernetes Service which supports the Pods.
What is next?
03-13-2019 08:18 AM
here is the logs from inside one of the Pods: part1
2019-03-13 15:06:11.293+0000 INFO [o.n.k.AvailabilityGuard] Requirement makes database unavailable: Database is stopped
2019-03-13 15:06:11.870+0000 INFO [o.n.k.i.f.CommunityEditionModule] No locking implementation specified, defaulting to 'forseti'
2019-03-13 15:06:11.916+0000 INFO [o.n.c.c.TransactionBackupServiceProvider] Binding backup service on address 127.0.0.1:6362
2019-03-13 15:06:11.920+0000 INFO [o.n.c.u.s.TypicallyConnectToRandomReadReplicaStrategy] Using upstream selection strategy typically-connect-to-random-read-replica
2019-03-13 15:06:11.944+0000 INFO [o.n.k.i.f.StatementLocksFactorySelector] No services implementing StatementLocksFactory found. Using SimpleStatementLocksFactory
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] --- INITIALIZED diagnostics START ---
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] Neo4j Kernel properties:
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.http.enabled=true
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.discovery_listen_address=0.0.0.0:5000
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.mode=CORE
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.security.auth_enabled=false
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.http.listen_address=0.0.0.0:7474
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] unsupported.dbms.directories.neo4j_home=/var/lib/neo4j
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.raft_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7000
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.directories.import=import
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.memory.heap.initial_size=512M
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.raft_listen_address=0.0.0.0:7000
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connectors.default_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] ACCEPT.LICENSE.AGREEMENT=yes
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] ha.pull_interval=10
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.discovery_type=DNS
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] ha.host.coordination=neo4j-core-1:5001
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.https.enabled=true
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.jvm.additional=-Dunsupported.dbms.udc.source=tarball
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.tx_log.rotation.retention_policy=100M size
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.memory.heap.max_size=512M
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connectors.default_listen_address=0.0.0.0
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] EDITION=enterprise
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.windows_service_name=neo4j
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.https.listen_address=0.0.0.0:7473
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.listen_address=0.0.0.0:7687
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.transaction_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:6000
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.discovery_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:5000
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.memory.pagecache.size=512M
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] ha.host.data=neo4j-core-1:6001
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.initial_discovery_members=neo4j.rag-uat.svc.cluster.local:5000
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] unsupported.dbms.edition=enterprise
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.enabled=true
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.transaction_listen_address=0.0.0.0:6000
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] wrapper.java.additional=-Dneo4j.ext.udc.source=docker
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] Diagnostics providers:
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] org.neo4j.kernel.configuration.Config
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] org.neo4j.kernel.info.DiagnosticsManager
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] SYSTEM_MEMORY
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] JAVA_MEMORY
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] OPERATING_SYSTEM
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] JAVA_VIRTUAL_MACHINE
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] CLASSPATH
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] LIBRARY_PATH
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] SYSTEM_PROPERTIES
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] TIMEZONE_DATABASE
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] LINUX_SCHEDULERS
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] NETWORK
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] System memory information:
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Total Physical memory: 7.64 GB
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Free Physical memory: 335.31 MB
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Committed virtual memory: 1.97 GB
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Total swap space: 0.00 B
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Free swap space: 0.00 B
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] JVM memory information:
2019-03-13 15:06:12.195+0000 INFO [o.n.k.i.DiagnosticsManager] Free memory: 437.00 MB
2019-03-13 15:06:12.195+0000 INFO [o.n.k.i.DiagnosticsManager] Total memory: 512.00 MB
2019-03-13 15:06:12.195+0000 INFO [o.n.k.i.DiagnosticsManager] Max memory: 512.00 MB
2019-03-13 15:06:12.199+0000 INFO [o.n.k.i.DiagnosticsManager] Garbage Collector: G1 Young Generation: [G1 Eden Space, G1 Survivor Space]
2019-03-13 15:06:12.199+0000 INFO [o.n.k.i.DiagnosticsManager] Garbage Collector: G1 Old Generation: [G1 Eden Space, G1 Survivor Space, G1 Old Gen]
2019-03-13 15:06:12.199+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: Code Cache (Non-heap memory): committed=6.06 MB, used=6.02 MB, max=240.00 MB, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: Metaspace (Non-heap memory): committed=20.25 MB, used=19.18 MB, max=-1.00 B, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: Compressed Class Space (Non-heap memory): committed=2.75 MB, used=2.48 MB, max=1.00 GB, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Eden Space (Heap memory): committed=322.00 MB, used=66.00 MB, max=-1.00 B, threshold=?
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Survivor Space (Heap memory): committed=1.00 MB, used=1.00 MB, max=-1.00 B, threshold=?
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Old Gen (Heap memory): committed=189.00 MB, used=8.00 MB, max=512.00 MB, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Operating system information:
2019-03-13 15:06:12.201+0000 INFO [o.n.k.i.DiagnosticsManager] Operating System: Linux; version: 3.10.0-514.el7.x86_64; arch: amd64; cpus: 1
2019-03-13 15:06:12.201+0000 INFO [o.n.k.i.DiagnosticsManager] Max number of file descriptors: 1048576
2019-03-13 15:06:12.201+0000 INFO [o.n.k.i.DiagnosticsManager] Number of open file descriptors: 161
2019-03-13 15:06:12.202+0000 INFO [o.n.k.i.DiagnosticsManager] Process id: 1@neo4j-core-1
2019-03-13 15:06:12.202+0000 INFO [o.n.k.i.DiagnosticsManager] Byte order: LITTLE_ENDIAN
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] Local timezone: GMT
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] JVM information:
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] VM Name: OpenJDK 64-Bit Server VM
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] VM Vendor: Oracle Corporation
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] VM Version: 25.181-b13
2019-03-13 15:06:12.208+0000 INFO [o.n.k.i.DiagnosticsManager] JIT compiler: HotSpot 64-Bit Tiered Compilers
2019-03-13 15:06:12.208+0000 INFO [o.n.k.i.DiagnosticsManager] VM Arguments: [-Xms512M, -Xmx512M, -XX:+UseG1GC, -XX:-OmitStackTraceInFastThrow, -XX:+AlwaysPreTouch, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.tls.ephemeralDHKeySize=2048, -Djdk.tls.rejectClientInitiatedRenegotiation=true, -Dunsupported.dbms.udc.source=tarball, -Dfile.encoding=UTF-8]
2019-03-13 15:06:12.208+0000 INFO [o.n.k.i.DiagnosticsManager] Java classpath:
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-com-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jackson-core-asl-1.9.13.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/jaccess.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-crypto-core-1.4.0.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/asm-analysis-6.2.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-planner-3.4-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/server-api-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/concurrentlinkedhashmap-lru-1.4.2.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-command-line-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jPowerShell-1.9.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-lucene-upgrade-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/netty-3.9.9.Final.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-udc-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-bolt-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jersey-server-1.19.3.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-shell-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/jce.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-compress-1.16.1.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient-0.1.0.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-csv-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-planner-spi-3.4-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-kernel-commercial-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-io-2.6.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-util-9.4.11.v20180605.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jersey-core-1.19.3.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-jmx-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-runtime-util-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/asm-6.2.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-configuration-1.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-lang3-3.7.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-slotted-runtime-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-config-core-1.4.0.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/sunec.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jersey-servlet-1.19.3.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-util-3.4-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-security-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-text-1.3.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-values-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-management-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/scala-library-2.11.12.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/classes
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-event-1.4.0.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-security-9.4.11.v20180605.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-interpreted-runtime-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-crypto-hash-1.4.0.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-servlet-9.4.11.v20180605.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-morsel-runtime-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-ssl-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/javax.servlet-api-3.1.0.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/rt.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/localedata.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-security-enterprise-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jProcesses-1.6.4.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-compiler-2.3-2.3.12.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-graph-algo-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/scala-reflect-2.11.12.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/plugins/
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/asm-util-6.2.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/sunjce_provider.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/slf4j-nop-1.7.25.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-lang-2.6.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-core-5.5.5.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-enterprise-kernel-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-graphdb-api-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-codecs-5.5.5.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-codegen-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-metrics-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/rhino-1.7.9.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-compiler-3.3-3.3.8.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-core-1.4.0.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-ir-3.4-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/nashorn.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-compiler-3.1-3.1.9.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-beanutils-1.9.3.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-unsafe-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-kernel-api-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/dnsns.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/jfr.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jline-2.12.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-logging-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-server-enterprise-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-logical-plans-3.4-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/conf/
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-logical-plans-3.3-3.3.8.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/sunpkcs11.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/asm-tree-6.2.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-expression-3.4-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-lucene-index-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/scallop_2.11-2.1.1.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/hazelcast-all-3.7.8.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-webapp-9.4.11.v20180605.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-consistency-check-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cluster-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/openCypher-frontend-1-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/opencsv-2.3.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/parboiled-scala_2.11-1.1.7.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient_common-0.1.0.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-kernel-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-index-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-server-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-procedure-api-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-cache-1.4.0.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/kiama_2.11-2.1.0.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-backup-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-server-9.4.11.v20180605.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-enterprise-cypher-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-primitive-collections-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-collections-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/resources.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/lz4-1.3.0.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/metrics-core-4.0.2.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/charsets.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/slf4j-api-1.7.25.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-http-9.4.11.v20180605.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/metrics-graphite-4.0.2.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jackson-mapper-asl-1.9.13.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-causal-clustering-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/caffeine-2.6.2.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-queryparser-5.5.5.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/bcprov-jdk15on-1.60.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-browser-3.2.13.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jackson-jaxrs-1.9.13.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/zipfs.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-io-3.4.10.jar
03-13-2019 08:19 AM
logs part two:
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-causal-clustering-commercial-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-configuration-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-frontend-3.1-3.1.9.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/WMI4Java-1.6.1.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-crypto-cipher-1.4.0.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/plugins/*
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-frontend-3.3-3.3.8.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient_dropwizard-0.1.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/cldrdata.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jsr311-api-1.1.2.r612.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient_httpserver-0.1.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-xml-9.4.11.v20180605.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-frontend-2.3-2.3.12.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-config-ogdl-1.4.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-server-commercial-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-analyzers-common-5.5.5.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-resource-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-auth-plugin-api-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-ha-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/jsse.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/sunrsasign.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/bcpkix-jdk15on-1.60.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-dbms-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-ir-3.3-3.3.8.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-logging-1.1.1.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-lang-1.4.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-query-logging-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-io-9.4.11.v20180605.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/commons-collections-3.2.2.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-graph-matching-3.1.9.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/parboiled-core-1.1.7.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-common-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/netty-all-4.1.24.Final.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-physical-planning-3.4.10.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-import-tool-3.4.10.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-spatial-index-3.4.10.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] Library path:
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib/jvm/java-1.8-openjdk/lib/amd64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/java/packages/lib/amd64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /lib64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /lib
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] System.properties:
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.shutdownhook.enabled = false
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.boot.library.path = /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] jdk.tls.rejectClientInitiatedRenegotiation = true
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] path.separator = :
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] file.encoding.pkg = sun.io
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.country = US
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.java.launcher = SUN_STANDARD
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.os.patch.level = unknown
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.dir = /var/lib/neo4j
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.jnu.encoding = UTF-8
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.phone.home.enabled = false
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] unsupported.dbms.udc.source = tarball
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.management.compiler = HotSpot 64-Bit Tiered Compilers
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.logging.type = none
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.home = /var/lib/neo4j
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.timezone = GMT
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] file.encoding = UTF-8
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.socket.bind.any = false
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.name = neo4j
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.java.command = com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/var/lib/neo4j --config-dir=/var/lib/neo4j/conf
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.arch.data.model = 64
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] user.language = en
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] awt.toolkit = sun.awt.X11.XToolkit
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] jdk.tls.ephemeralDHKeySize = 2048
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] file.separator = /
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.io.unicode.encoding = UnicodeLittle
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.cpu.endian = little
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.cpu.isalist =
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] (IANA) TimeZone Database Version:
2019-03-13 15:06:12.250+0000 INFO [o.n.k.i.DiagnosticsManager] TimeZone version: 2018e (available for 599 zone identifiers)
2019-03-13 15:06:12.250+0000 INFO [o.n.k.i.DiagnosticsManager] Linux scheduler information:
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] noop deadline [cfq]
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] noop deadline [cfq]
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] Network information:
2019-03-13 15:06:12.263+0000 INFO [o.n.k.i.DiagnosticsManager] Interface eth0:
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] address: fe80:0:0:0:f481:e2ff:fe42:5a4d%eth0
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] address: 10.128.0.45
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] Interface lo:
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] address: 0:0:0:0:0:0:0:1%lo
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] address: 127.0.0.1
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] --- INITIALIZED diagnostics END ---
2019-03-13 15:06:12.271+0000 INFO [o.n.c.c.s.s.DurableStateStorage] term state restored, up to ordinal 19701
2019-03-13 15:06:12.275+0000 INFO [o.n.c.c.s.s.DurableStateStorage] vote state restored, up to ordinal 19701
2019-03-13 15:06:12.279+0000 INFO [o.n.c.c.s.s.DurableStateStorage] membership state restored, up to ordinal 0
2019-03-13 15:06:12.283+0000 INFO [o.n.c.c.s.s.DurableStateStorage] session-tracker state restored, up to ordinal -1
2019-03-13 15:06:12.288+0000 INFO [o.n.c.c.s.s.DurableStateStorage] lock-token state restored, up to ordinal -1
2019-03-13 15:06:12.292+0000 INFO [o.n.c.c.s.s.DurableStateStorage] id-allocation state restored, up to ordinal 0
2019-03-13 15:06:12.295+0000 INFO [o.n.c.c.s.s.DurableStateStorage] last-flushed state restored, up to ordinal 0
2019-03-13 15:06:12.845+0000 INFO [o.n.b.BoltKernelExtension] Bolt Server extension loaded.
2019-03-13 15:06:12.906+0000 INFO [o.n.m.MetricsExtension] Initiating metrics...
2019-03-13 15:06:13.012+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Cluster discovery service starting
2019-03-13 15:06:13.039+0000 INFO [o.n.c.d.DnsHostnameResolver] Resolved initial host 'neo4j.rag-uat.svc.cluster.local:5000' to [10.130.1.54:5000, 10.129.0.104:5000, 10.128.0.45:5000]
2019-03-13 15:06:13.066+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] My connection info: [
Discovery: listen=0.0.0.0:5000, advertised=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:5000,
Transaction: listen=0.0.0.0:6000, advertised=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:6000,
Raft: listen=0.0.0.0:7000, advertised=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7000,
Client Connector Addresses: bolt://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7473
]
2019-03-13 15:06:13.066+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Discovering cluster with initial members: [neo4j.rag-uat.svc.cluster.local:5000]
2019-03-13 15:06:13.066+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Attempting to connect to the other cluster members before continuing...
2019-03-13 15:06:13.082+0000 INFO [o.n.c.c.c.l.s.RecoveryProtocol] Skipping from index -1 to 1.
2019-03-13 15:06:13.089+0000 INFO [o.n.c.c.c.l.s.SegmentedRaftLog] log started with recovered state State{prevIndex=0, prevTerm=-1, appendIndex=0}
2019-03-13 15:06:13.089+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Membership state before recovery: RaftMembershipState{committed=MembershipEntry{logIndex=0, members=[MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]}, appended=null, ordinal=0}
2019-03-13 15:06:13.089+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Recovering from: 0 to: 0
2019-03-13 15:06:13.091+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Membership state after recovery: RaftMembershipState{committed=MembershipEntry{logIndex=0, members=[MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]}, appended=null, ordinal=0}
2019-03-13 15:06:13.091+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: []
2019-03-13 15:06:13.092+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Not safe to remove members [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}] because it would reduce the number of voting members below the expected cluster size of 3. Voting members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:13.175+0000 INFO [o.n.c.n.Server] raft-server: bound to 0.0.0.0:7000
2019-03-13 15:06:14.021+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.130.1.54:58088
2019-03-13 15:06:14.251+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.130.1.54:58090
2019-03-13 15:06:21.790+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Cluster discovery service started
2019-03-13 15:06:21.803+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[{memberId=MemberId{495e65e3}, info=CoreServerInfo{raftServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}, {memberId=MemberId{531e9f4a}, info=CoreServerInfo{raftServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}, {memberId=MemberId{bdf87a06}, info=CoreServerInfo{raftServer=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}], removed=[]}
2019-03-13 15:06:21.803+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{531e9f4a}, MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:06:21.814+0000 INFO [o.n.c.i.ClusterBinder] Published: ClusterId{uuid=1144130e-fdbe-4157-988a-436e58d235b7}
2019-03-13 15:06:21.815+0000 INFO [o.n.c.i.ClusterBinder] Already bound to cluster: ClusterId{uuid=1144130e-fdbe-4157-988a-436e58d235b7}
2019-03-13 15:06:21.828+0000 INFO [o.n.c.c.s.LocalDatabase] Starting with storeId: Store{creationTime:1544642757823, randomId:6597478259518778525, upgradeTime:1544642757823, upgradeId:1}
2019-03-13 15:06:21.876+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store /var/lib/neo4j/data/databases/graph.db
2019-03-13 15:06:21.909+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store /var/lib/neo4j/data/databases/graph.db
2019-03-13 15:06:21.909+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Format not configured. Selected format from the store: RecordFormat:StandardV3_4[v0.A.9]
2019-03-13 15:06:22.170+0000 INFO [o.n.k.NeoStoreDataSource] No commits found after last check point (which is at LogPosition{logVersion=0, byteOffset=90})
2019-03-13 15:06:22.195+0000 INFO [o.n.k.i.DatabaseHealth] Database health set to OK
2019-03-13 15:06:22.504+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Last transaction id in metadata store 2
2019-03-13 15:06:22.510+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Start id of last committed transaction in transaction log 1
2019-03-13 15:06:22.510+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Last committed transaction id in transaction log 2
2019-03-13 15:06:22.511+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Last committed consensus log index committed into tx log -1
2019-03-13 15:06:22.511+0000 INFO [o.n.c.c.s.m.t.ReplicatedTransactionStateMachine] Updated lastCommittedIndex to -1
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.m.t.ReplicatedTokenStateMachine] (Label) Updated lastCommittedIndex to -1
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.m.t.ReplicatedTokenStateMachine] (RelationshipType) Updated lastCommittedIndex to -1
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.m.t.ReplicatedTokenStateMachine] (PropertyKey) Updated lastCommittedIndex to -1
Forum did not let me to attach logs file, sorry if logs are long
03-13-2019 08:19 AM
logs - final part
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.CommandApplicationProcess] Restoring last applied index to 0
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.CommandApplicationProcess] Resuming after startup (count = 0)
2019-03-13 15:06:22.520+0000 INFO [o.n.c.n.Server] catchup-server: bound to 0.0.0.0:6000
2019-03-13 15:06:22.525+0000 INFO [o.n.c.c.c.s.RaftState] First leader elected: MemberId{531e9f4a}
2019-03-13 15:06:22.528+0000 INFO [o.n.c.n.Server] backup-server: bound to 127.0.0.1:6362
2019-03-13 15:06:22.531+0000 INFO [o.n.c.m.SenderService] Creating channel to: [neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000]
2019-03-13 15:06:22.534+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Leader MemberId{bdf87a06} updating leader info for database default and term 19701
2019-03-13 15:06:22.536+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:22.551+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0x4e27fe80, L:/10.128.0.45:55262 - R:neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000]
2019-03-13 15:06:22.553+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:55262 remote neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000
2019-03-13 15:06:22.561+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:06:27.895+0000 WARN [o.n.c.m.SenderService] Lost connection to: neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000 (neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000)
2019-03-13 15:06:27.903+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:27.904+0000 WARN [o.n.c.m.SenderService] Failed to connect to: neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000. Retrying in 100 ms
2019-03-13 15:06:28.006+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:28.208+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:28.611+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:29.413+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:30.523+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member removed MembershipEvent {member=Member [10.130.1.54]:5000 - b78d7d4f-1ebd-477a-9a39-4c13eee6ea0e,type=removed}
2019-03-13 15:06:30.525+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[], removed=[{memberId=MemberId{531e9f4a}, info=CoreServerInfo{raftServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}]}
2019-03-13 15:06:30.525+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:06:30.525+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Not safe to remove member [MemberId{531e9f4a}] because it would reduce the number of voting members below the expected cluster size of 3. Voting members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:31.017+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:32.621+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:34.224+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:35.828+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:36.347+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.129.0.104:59194
2019-03-13 15:06:36.356+0000 INFO [o.n.c.m.SenderService] Creating channel to: [neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000]
2019-03-13 15:06:36.356+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:36.357+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0xc771bf7f, L:/10.128.0.45:45486 - R:neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000]
2019-03-13 15:06:36.357+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:45486 remote neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000
2019-03-13 15:06:36.360+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:06:36.369+0000 INFO [o.n.c.c.c.s.RaftState] Leader changed from MemberId{531e9f4a} to MemberId{495e65e3}
2019-03-13 15:06:36.369+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Leader MemberId{bdf87a06} updating leader info for database default and term 19702
2019-03-13 15:06:37.430+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:39.032+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:40.635+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:42.238+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:42.861+0000 WARN [o.n.c.m.SenderService] Lost connection to: neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000 (neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000)
2019-03-13 15:06:42.862+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:42.863+0000 WARN [o.n.c.m.SenderService] Failed to connect to: neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000. Retrying in 100 ms
2019-03-13 15:06:42.965+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.167+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.570+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.840+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.841+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0x573c8293, L:/10.128.0.45:55462 - R:neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000]
2019-03-13 15:06:43.841+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:55462 remote neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000
2019-03-13 15:06:43.983+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:06:44.219+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member added MembershipEvent {member=Member [10.130.1.54]:5000 - ecbfcc66-9b7b-4d01-9483-8deedb3b817f,type=added}
2019-03-13 15:06:44.220+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[{memberId=MemberId{531e9f4a}, info=CoreServerInfo{raftServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}], removed=[]}
2019-03-13 15:06:44.220+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{531e9f4a}, MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:06:44.371+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:45.621+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member removed MembershipEvent {member=Member [10.129.0.104]:5000 - 890c1610-3d35-45f6-9cde-c5fe27a4804c,type=removed}
2019-03-13 15:06:45.625+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[], removed=[{memberId=MemberId{495e65e3}, info=CoreServerInfo{raftServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}]}
2019-03-13 15:06:45.625+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{bdf87a06}, MemberId{531e9f4a}]
2019-03-13 15:06:45.625+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Not safe to remove member [MemberId{495e65e3}] because it would reduce the number of voting members below the expected cluster size of 3. Voting members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:45.974+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:47.577+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:48.885+0000 INFO [o.n.c.c.c.RaftMachine] Election timeout triggered
2019-03-13 15:06:48.888+0000 INFO [o.n.c.c.c.RaftMachine] Election started with vote request: Vote.Request from MemberId{bdf87a06} {term=19703, candidate=MemberId{bdf87a06}, lastAppended=0, lastLogTerm=-1} and members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:48.888+0000 INFO [o.n.c.c.c.RaftMachine] Moving to CANDIDATE state after successfully starting election
2019-03-13 15:06:48.891+0000 INFO [o.n.c.m.RaftOutbound] No address found for MemberId{495e65e3}, probably because the member has been shut down.
2019-03-13 15:06:49.179+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:50.534+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:06:50.783+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:52.387+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:53.990+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:55.594+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:56.712+0000 INFO [o.n.c.c.c.RaftMachine] Failed to get elected. Got votes from: []
2019-03-13 15:06:56.712+0000 INFO [o.n.c.c.c.RaftMachine] Election started with vote request: Vote.Request from MemberId{bdf87a06} {term=19704, candidate=MemberId{bdf87a06}, lastAppended=0, lastLogTerm=-1} and members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:56.733+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.130.1.54:58408
2019-03-13 15:06:57.196+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:57.648+0000 INFO [o.n.c.c.c.RaftMachine] Moving to FOLLOWER state after receiving vote request from MemberId{531e9f4a} at term 19705 (I am at 19704)
2019-03-13 15:06:57.660+0000 INFO [o.n.c.c.c.s.RaftState] Leader changed from MemberId{495e65e3} to MemberId{531e9f4a}
2019-03-13 15:06:57.660+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Leader MemberId{bdf87a06} updating leader info for database default and term 19705
2019-03-13 15:06:58.800+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:07:00.402+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:07:00.760+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member added MembershipEvent {member=Member [10.129.0.104]:5000 - 02077349-7bf3-4f41-afde-f86e2b348c35,type=added}
2019-03-13 15:07:00.762+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[{memberId=MemberId{495e65e3}, info=CoreServerInfo{raftServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}], removed=[]}
2019-03-13 15:07:00.762+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{531e9f4a}, MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:07:02.005+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:07:02.006+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0xdc46d48e, L:/10.128.0.45:45720 - R:neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000]
2019-03-13 15:07:02.006+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:45720 remote neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000
2019-03-13 15:07:02.010+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:07:18.535+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:07:46.535+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:08:14.535+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:08:42.536+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:09:10.536+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:09:38.536+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:10:06.537+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:10:34.537+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:11:02.537+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:11:30.538+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:11:58.538+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:12:26.538+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
DNS is completely unrelated. Since as far it is obvious from the log it tries to address each of them and other nodes were available to ping but did not reply back
03-13-2019 12:43 PM
The things I'm seeing which are most relevant are:
2019-03-13 15:06:42.861+0000 WARN [o.n.c.m.SenderService] Lost connection to: neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000 (neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000)
2019-03-13 15:06:48.891+0000 INFO [o.n.c.m.RaftOutbound] No address found for MemberId{495e65e3}, probably because the member has been shut down.
These logs indicate DNS isn't the problem, you are successfully connecting to other members of the cluster, but you're not getting a stable cluster because members are leaving/entering the cluster.
The thing to look for next is evidence that the docker containers you're launching are crashing in a loop, or something like that. This would explain this behavior. Can you verify that when you launch the pods they're staying up, or do you see events in the k8s log that indicate the pods are shutting down and getting auto-restarted by kubernetes?
Also, are you using any additional plugins or non-standard configuration which might interfere with any pods ability to start normally?
03-13-2019 10:51 PM
Not at all. All pods are healthy and don't CrashLoopBackOff. They were up and running otherwise I could not get into them via ssh and gather neo4j logs.
About the plugins: I only deployed a standard Openshift 3.9 nothing more or less.
If you look into the history of issue you'll see a kind of pattern happening to people deploying without helm via yamls. Helm is not supported directly in Openshift. I configured our statefulset based on its helm definitions ,that's it.
It's not likable for me to have to turn into single instance architecture
03-15-2019 01:20 PM
Hi @tech-mint - I'm not sure what other reports you're referring to, could you please link them? I have not seen this issue before with folks deploying neo4j via helm and YAML files. Where are you seeing the history of this issue?
I understand this is frustrating and will help out if I can. It appears the logs you've provided are probably from one of the nodes. The best next step is to get debug.log from all three cluster nodes in the core set, and then maybe upload them somewhere where we can take a look. The complete files would be helpful, so we can look at full context. Pasted snippets of this much data will be hard to look at with these forums.
I can tell you what we'll be looking for: it's going to be those raft cluster connection messages. We need a log of all 3 talking to each other, and how each of the 3 sees the progression of events. This will make clear whether a node is restarting, whether there's some intermediate network problem, whether you have bad cluster state on some node, etc. The other thing you have that's odd in your logs are these lines:
Scheduling handshake (and timeout) local null remote null
Those nulls are concerning, but I don't know enough to say why that might be happening.
03-17-2019 06:10 AM
I installed Helm on Openshift and redeploy neo4j. Issue is resolved,
thanks for your follow up
Reply
03-17-2019 06:43 AM
Whoa! Congrats! But I have to ask -- what was different here?
Because I think you had a lot of frustrating trouble here, make sure to say what you did differently so the next person who finds this thread can do the same.
03-17-2019 11:25 PM
I previously was deploying with the files in the helm chart without using helm directly. maybe I was missing something or so. I was wrong that helm is not possible in Openshift. after I installed helm in openshift and deployed neo4j via helm everything got smooth and also it saves me time too.
thanks for your support
Reply
Nodes 2022
NODES 2022, Neo4j Online Education Summit
All the sessions of the conference are now available online
Related Content
- Startup probe failed while deploying standalone instance of neo4j in Neo4j Graph Platform
- Causal Cluster Deployment using Terraform in Drivers & Stacks
- New Enterprise Cluster Implementation in Neo4j Graph Platform
- SSL config with helm installation in Neo4j Graph Platform
- Managing Databases on a K8s cluster in Integrations
