Why do connections pile up (Java Driver 4.4.9)?

[BairDev]

I am using a single Neo4J driver object (in a Jersey Binder class as an instance of a singleton). The driver is created like this:

Config config = Config.builder()
    .withMaxConnectionLifetime( 30, TimeUnit.MINUTES )
    .withMaxConnectionPoolSize( 50 )
    .withConnectionAcquisitionTimeout( 30, TimeUnit.SECONDS )
    .withDriverMetrics()
    .build();
this.driver = GraphDatabase.driver(String.format("neo4j://%s:%s",
    "localhost", "port no",
    AuthTokens.basic("username",
        "password", config);
this.driver.verifyConnectivity();
this.logger.info("neo4j driver created {}", this.driver.metrics().connectionPoolMetrics());

But it seems like every time I am using this method connections do pile up

private Driver getDriver() {
    if (this.driver == null) {
        throw new RuntimeException("No Neo4J driver. Something wrong with connection or DB?");
    }
    this.logger.debug("getDriver metrics neo4j {}", this.driver.metrics().connectionPoolMetrics());
    return this.driver;
}
// in public Session getSession();
Session session = null;
try {
    session = this.getSession();
    // do stuff
} catch (Exception ex) {
    // stuff

    // either
    session.readTransaction( /* TransactionWork */ );

   // or
   Transaction tx = session.beginTransaction();
   tx.run("wonderfull query", Map.of("field", "value"));
   tx.commit();
   tx.close();
} finally {
    if (session != null) {
        session.close();
    }
}

until we have 50 connections and we cannot create more and get

org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 30000ms

13:37:33 [ajp-nio-127.0.0.1-8009-exec-3] DEBUG dbConnection.Neo4JDriver - getDriver metrics neo4j [localhost:7687-138669777=[
    created=50,
    closed=0,
    creating=0,
    failedToCreate=0,
    acquiring=1,
    acquired=120,
    timedOutToAcquire=0,
    inUse=50,
    idle=0,
    // ...
    totalInUseCount=70],
localhost:7687-571435580=[created=1,
    closed=0,
    creating=0,
    failedToCreate=0,
    acquiring=0,
    acquired=59,
    timedOutToAcquire=0,
    inUse=1,
    idle=0,
    // ...
    totalInUseCount=58]]
13:38:03 [ajp-nio-127.0.0.1-8009-exec-2] WARN  restResources.SomeEndpointCollection - Exception in myMethod Unable to acquire connection from the pool within configured maximum time of 30000ms
org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 30000ms
	at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:111) ~[neo4j-java-driver-4.4.9.jar:4.4.9-e855bcc800deff6ddcf064c822314bb5c8d08c53]
	at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:90) ~[neo4j-java-driver-4.4.9.jar:4.4.9-e855bcc800deff6ddcf064c822314bb5c8d08c53]
	at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:85) ~[neo4j-java-driver-4.4.9.jar:4.4.9-e855bcc800deff6ddcf064c822314bb5c8d08c53]

So I think I am doing something fundamentally wrong (for two years now!). How can I manage connections effectively? I have already learned that I am not supposed to close the driver, because it should be a single instance for the whole lifetime of the application. But why should I let it create more and more connections?

You may have a look at this SO-question.

Maybe there is one problem: the connection is not closed when I do not retrieve/consume the results? Usually I need the results from the queries (of course), but in rare cases I don't. Should I check the code and make sure that I always consume the results (in TransactionWork or after tx.run())?

Any other hint?

I can increase the number of max-connection to 300 or 3000, but why would I need such a large number of connections unless I have to fire up queries, which need a large amount of time?

PS: I am wondering what is a Session? Does it represent a connection?

From the docs:

The session lifetime extends from session construction to session closure. In languages that support them, simple sessions are usually scoped within a context block; this ensures that they are properly closed and that any underlying connections are released and not leaked.

But what does closing a session actually mean? Does it just mean that you cannot perform any more actions with it?

[dmitriy_tverdia · Accepted solution]

Hello and thank you for the detailed post.

Unfortunately, I have not been able to reproduce this using the provided code. It would be very useful if we could find some reproduction steps though.

The only way I have managed to reproduce is by misusing API using the following code:

var poolSize = 15;
var token = AuthTokens.basic(user, password);
var config = Config.builder()
        .withMaxConnectionPoolSize(poolSize)
        .withConnectionAcquisitionTimeout(1, TimeUnit.SECONDS)
        .build();
try (var driver = GraphDatabase.driver(uri, token, config)) {
    for (var i = 0; i < poolSize + 1; i++) {
        var session = driver.session();
        session.run("UNWIND range(0, 2000) AS x RETURN x");
    }
}

In the code above sessions do not get closed and they have unconsumed results, which causes the sessions to hold acquired connections.

Session ensures causal consistency between transactions in it. Session does not represent a connection, but it may hold a connection when it is needed. For instance, when there is an unconsumed result or an unfinished transaction. There is a bit more detail in the Javadoc.

It is important to ensure every session gets closed to avoid resource leaks. The mentioned documentation suggests using the try-with-resources statement, but it may also be managed in other way providing that sessions get closed.

[BairDev]

Thanks for your work and your response(s).

The actual problem was - of course - in my code. I have thrown an exception in a loop before calling (Transaction) tx.commit(), tx.close() and session.close().

Connections do not pile up anymore after fixing this bug. I guess that the main problem was that an uncommitted transaction keeps lingering.

I am just wondering about one thing: the result of

connectionPoolMetrics()

contains an id. This id changes for each call and I've firstly thought that it represented the driver and that I have different drivers in different threads, which was not the case. But then what is this id (localhost:7687-571435580)?

[dmitriy_tverdia]

Hello and thank you for your comments. I am really glad we have made some progress. 🙂

Open transaction would have to keep connection occupied, yes.

The connectionPoolMetrics() returns a collection of ConnectionPoolMetrics, which represent metrics for each connection pool that the driver manages. The ConnectionPoolMetrics.id() method returns a unique identifier for the given pool metrics that could be used for monitoring management. Please note that driver may have multiple connection pools. For instance, if it connects to multiple servers.

The id should not change on every call unless pool changes on every call.

Some of the reasons for pool change may be:

1. 1. Another driver instance is used.
2. 2. Server is dropped from routing table in clustered setup.

At present the id value is implemented as a copy of the id value of a specific instance of a connection pool. It has the following format: 'host:port-poolHashcode'. As can be seen, the id may change if there is a different instance of pool for a given address.

[dmitriy_tverdia]

If it is possible that multiple sessions with long living results or transactions may exist, connection pool should be sized appropriately to accomodate this.

Please ensure that every session is closed as this ensures resources are released properly.

Not holding unconsumed result or opened transaction on session helps with connection management because session is then able to release its connection. It will grab another one when another transaction is initiated.