This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Accept
Reject
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help

Head's Up! These forums are read-only. All users and content have migrated. Please join us at community.neo4j.com.

NPE on connection in container

get-a-grep
Node

‎10-26-2022 08:22 AM - edited ‎10-26-2022 08:27 AM

Hi, don't know if here or on Github is the place for this, however we've come across an issue while using the official neo4j docker that I'm not entirely certain how to place. Here's the setup:

  • Neo4j Version: We run neo4j:latest container
  • Language: Python 3.10 w/neo4j:latest python driver lib
  • OS: docker in AWS Linux
  • Not on a cluster

This is a fatal error that occurs awhile after database startup on connection request by our application. It bubbles up to our driver that the connection was lost because of a fatal DB error, so into the debug logs I went, to find this:

 

2022-06-30 18:43:43.368+0000 ERROR [o.n.b.t.p.ProtocolHandshaker] Fatal error occurred during protocol handshaking: [id: 0x477f0a03, L:/172.19.0.2:7687 - R:/172.19.0.1:57778]
java.lang.NullPointerException: null
        at io.netty.buffer.Unpooled.copiedBuffer(Unpooled.java:363) ~[netty-buffer-4.1.75.Final.jar:4.1.75.Final]
        at org.neo4j.bolt.transport.DiscoveryResponseHandler.channelRead(DiscoveryResponseHandler.java:60) ~[neo4j-bolt-4.4.8.jar:4.4.8]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:255) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:517) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at org.neo4j.bolt.transport.pipeline.AuthenticationTimeoutHandler.channelRead(AuthenticationTimeoutHandler.java:58) [neo4j-bolt-4.4.8.jar:4.4.8]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2022-06-30 18:52:36.788+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0xeb7069d4, L:/172.19.0.2:7687 ! R:/172.19.0.1:48598]
org.neo4j.bolt.runtime.BoltConnectionFatality: Terminated connection '[id: 0xeb7069d4, L:/172.19.0.2:7687 ! R:/172.19.0.1:48598]' as the client failed to authenticate within 30000 ms.
        at org.neo4j.bolt.transport.pipeline.AuthenticationTimeoutHandler.channelIdle(AuthenticationTimeoutHandler.java:78) ~[neo4j-bolt-4.4.8.jar:4.4.8]
        at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:506) [netty-handler-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:478) [netty-handler-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.75.Final.jar:4.1.75.Final]
        at java.lang.Thread.run(Thread.java:829) [?:?]

 

I saw another community post (here) that mentioned a connection pool issue, however this doesn't happen on timeout, rather right away after a connection is attempted. To be clear, this doesn't happen on every connection, rather it happens relatively spontaneously and a restart has cleared up the problem. I checked memory and storage to make sure that there weren't any issues there and so far as I saw there weren't:

Container Memory/Storage:

getagrep_0-1666797458982.png

Storage Host:

getagrep_0-1666798024403.png

 

Any help or pointers on this would be appreciated, even pointers to another ticket that may already be opened or documentation that talks about this.

0 Kudos
Reply
0 REPLIES 0
Nodes 2022
Nodes
NODES 2022, Neo4j Online Education Summit

All the sessions of the conference are now available online

Watch replays