Unable to acquire connection from the pool within configured maximum time

[potato]

One of my servers started throwing this exception for all requests (100% error rate)

Unable to acquire connection from the pool within configured maximum time of 60000ms; Error code 'N/A'; nested exception is org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateImpl(Neo4jPersistenceExceptionTranslator.java:105)

I restarted the server and then this exception was gone, and the server even handled more requests per second than before. I want to understand what caused this exception and what can be done to mitigate this.

Thanks!

[Cobra]

Hello @potato 😀

Did you had a look at the debug.log file?

Regards,
Cobra

[potato]

Im sorry, but where do I find this log when I'm using Neo4j AuraDB?

Thanks!

[Cobra]

You can't 😕

Are you sure to have the right configuration?

Can we see the query that throws that exception?

Regards,
Cobra

[potato]

Exception started coming from here: match(e:Entity) where e.entityId = ?
And then all the queries failed with the same exception. Is there a particular neo4j server configuration I should be looking at?

[Cobra]

Please give the full query.

How many nodes do you have?

What is the size (RAM) of the database?

[potato]

There are 3 nodes.
Memory 16GB
Storage 32GB
CPU 3
Neo4j version 4

I have attached the server log which uses this neo4j instance. The first query for each request is Repository#findById(id); 
match(e:Entity) where e.entityId = 'ENT-id' return e;
If any query would have executed, it would have been logged. 

[date h1:37:05] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:05] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:06] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:08] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:09] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:13] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:13] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:14] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.addRequestHeadersToContext - setting context
[date h1:37:14] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - RES:pong
[date h1:37:14] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - REQGET http://server_ip/ping
[date h1:37:15] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:15] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:15] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:15] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.addRequestHeadersToContext - setting context
[date h1:37:16] [Neo4jDriverIO-2-5] [] [34mINFO [0;39m org.neo4j.driver.ConnectionPool.? - Closing connection pool towards xxx-neo4j-host-xxx.xxx.neo4j.io(xxx-host-ip):7687, it has no active connections and is not in the routing table registry.
[date h1:37:16] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:16] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:17] [Neo4jDriverIO-2-9] [] [34mINFO [0;39m x.x.x.service.ProductUpdate.? - Error occurred
[date h1:37:17] [Neo4jDriverIO-2-9] [] [34mINFO [0;39m SERVER.? - REQPUT http://server-ip/product
[date h1:37:17] [Neo4jDriverIO-2-9] [] [1;31mERROR[0;39m o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler.? - [a628e50e-1]  500 Server Error
org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:204)
Stack trace:
		at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:204)
		at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:105)
		at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
		at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
		at org.neo4j.driver.internal.util.Futures.lambda$asCompletionStage$0(Futures.java:97)
		...
		at java.lang.Thread.run(Thread.java:748)
[date h1:37:17] [Neo4jDriverIO-2-10] [] [34mINFO [0;39m x.x.x.service.ProductUpdate.? - Error occurred
[date h1:37:17] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:17] [Neo4jDriverIO-2-10] [] [34mINFO [0;39m SERVER.? - REQPUT http://server-ip/product
[date h1:37:17] [Neo4jDriverIO-2-10] [] [1;31mERROR[0;39m o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler.? - [9724f418-1]  500 Server Error
org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:204)
Stack trace:
		at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:204)
		at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:105)
		at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
		at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
		at org.neo4j.driver.internal.util.Futures.lambda$asCompletionStage$0(Futures.java:97)
		...
		at java.lang.Thread.run(Thread.java:748)
[date h1:37:17] [reactor-http-epoll] [] [34mINFO [0;39m SERVER.? - setting context
[date h1:37:17] [Neo4jDriverIO-2-11] [] [34mINFO [0;39m x.x.x.service.DataUpdate.? - Error occurred
[date h1:37:17] [Neo4jDriverIO-2-11] [REQ-0f886115-1dd4-4ff5-b602-371c8f839f11] [34mINFO [0;39m SERVER.? - REQPUT http://server-ip/data
[date h1:37:17] [Neo4jDriverIO-2-11] [] [1;31mERROR[0;39m o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler.? - [50b817d1-4]  500 Server Error
org.springframework.dao.InvalidDataAccessResourceUsageException: Unable to acquire connection from the pool within configured maximum time of 60000ms; Error code 'N/A'; nested exception is org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateImpl(Neo4jPersistenceExceptionTranslator.java:105)
Stack trace:
		at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateImpl(Neo4jPersistenceExceptionTranslator.java:105)
		at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateExceptionIfPossible(Neo4jPersistenceExceptionTranslator.java:91)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient.potentiallyConvertRuntimeException(DefaultReactiveNeo4jClient.java:286)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient.access$200(DefaultReactiveNeo4jClient.java:52)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient$DefaultRecordFetchSpec.lambda$one$10(DefaultReactiveNeo4jClient.java:247)
		at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3662)
		...
		at java.lang.Thread.run(Thread.java:748)
Caused by: org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:204)
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:105)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	...
	at java.lang.Thread.run(Thread.java:748)
[date h1:37:17] [Neo4jDriverIO-2-14] [] [34mINFO [0;39m x.x.x.service.ProductUpdate.? - Error occurred
[date h1:37:17] [Neo4jDriverIO-2-14] [] [34mINFO [0;39m SERVER.? - REQPUT http://server-ip/product
[date h1:37:17] [Neo4jDriverIO-2-14] [] [1;31mERROR[0;39m o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler.? - [ef71f692-1]  500 Server Error
org.springframework.dao.InvalidDataAccessResourceUsageException: Unable to acquire connection from the pool within configured maximum time of 60000ms; Error code 'N/A'; nested exception is org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateImpl(Neo4jPersistenceExceptionTranslator.java:105)
Stack trace:
		at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateImpl(Neo4jPersistenceExceptionTranslator.java:105)
		at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateExceptionIfPossible(Neo4jPersistenceExceptionTranslator.java:91)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient.potentiallyConvertRuntimeException(DefaultReactiveNeo4jClient.java:286)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient.access$200(DefaultReactiveNeo4jClient.java:52)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient$DefaultRecordFetchSpec.lambda$one$10(DefaultReactiveNeo4jClient.java:247)
		at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3662)
		...
		at java.lang.Thread.run(Thread.java:748)
Caused by: org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:204)
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:105)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	...
	at java.lang.Thread.run(Thread.java:748)
[date h1:37:17] [Neo4jDriverIO-2-15] [] [34mINFO [0;39m x.x.x.service.ProductUpdate.? - Error occurred
[date h1:37:17] [Neo4jDriverIO-2-15] [] [34mINFO [0;39m SERVER.? - REQPUT http://server-ip/product
[date h1:37:17] [Neo4jDriverIO-2-15] [] [1;31mERROR[0;39m o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler.? - [3f581984-1]  500 Server Error
org.springframework.dao.InvalidDataAccessResourceUsageException: Unable to acquire connection from the pool within configured maximum time of 60000ms; Error code 'N/A'; nested exception is org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateImpl(Neo4jPersistenceExceptionTranslator.java:105)
Stack trace:
		at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateImpl(Neo4jPersistenceExceptionTranslator.java:105)
		at org.springframework.data.neo4j.core.Neo4jPersistenceExceptionTranslator.translateExceptionIfPossible(Neo4jPersistenceExceptionTranslator.java:91)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient.potentiallyConvertRuntimeException(DefaultReactiveNeo4jClient.java:286)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient.access$200(DefaultReactiveNeo4jClient.java:52)
		at org.springframework.data.neo4j.core.DefaultReactiveNeo4jClient$DefaultRecordFetchSpec.lambda$one$10(DefaultReactiveNeo4jClient.java:247)
		at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3662)
	...
		at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.lang.Thread.run(Thread.java:748)
Caused by: org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:204)
	at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:105)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	...
	at java.lang.Thread.run(Thread.java:748)