Neo4j enterprise edition memory issues

[susheelpatil]

Hi,

I am running neo4j cluster with 3 cores and 1 read replica. We are ingesting around 72 million nodes data and getting below exception and warnings after couple of hour of data ingestion
ersion:6.0]
2022-05-11 21:03:10.247+0000 WARN [c.n.c.c.s.TxPuller] [neo4j/e86b6ca5] Unexpected exception when pulling transactions. [Address: ntd-neo4j-replica-0.ntd-neo4j-replica.ntd-vendors.svc.cluster.local:6000]
java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]
at com.neo4j.causalclustering.helper.OperationProgressMonitor.get(OperationProgressMonitor.java:45) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.CatchupClient.sendBlocking(CatchupClient.java:50) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.tx.TxPullClient.pullRawTransactions(TxPullClient.java:71) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.TxPuller.execute(TxPuller.java:122) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.TxPuller.pullTransactions(TxPuller.java:83) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.RemoteStore.pullTransactions(RemoteStore.java:186) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.RemoteStore.copy(RemoteStore.java:142) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.StoreCopyProcess.replaceWithStoreFrom(StoreCopyProcess.java:79) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.core.state.snapshot.StoreDownloader.bringUpToDate(StoreDownloader.java:52) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]

After coupl eof hours i am getting blow message continuously

2022-05-12 02:00:50.882+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 290 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:00:59.533+0000 INFO [c.n.c.c.t.TxPullRequestBatchingLogger] TxPullRequests batched latest 5 minutes:
[/172.16.7.244:39942] sent 145 TxPullRequests for DatabaseId{00000000} for transaction above id 79
2022-05-12 02:01:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 291 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:02:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 292 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:03:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 293 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:04:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 294 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:05:50.884+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 295 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:05:59.533+0000 INFO [c.n.c.c.t.TxPullRequestBatchingLogger] TxPullRequests batched latest 5 minutes:
[/172.16.7.244:39942] sent 153 TxPullRequests for DatabaseId{00000000} for transaction above id 79

Also the memory usage has reached allocated RAM limit of 24 GB. it is not coming down. All the queries are hung and ingest program is stuck.

Any help is appreciated.

Thanks,
Susheelstrong text