Neo4j 4.x Query Logging Enhancements

[greta]

Neo4j 4.x Query Logging

and how the new features in Query Log Analyzer 1.0.2 support that

Since Neo4j version 4.0 the query log possibilities have been extended. Meanwhile version 1.0.2 of the query log analyzer was released.

Photo by Isaac Smith on Unsplash

We will show how this release supports these new query logging features and how it is now possible to read the log files directly from the server.

Please note that Query Logging is only available in the Enterprise Edition. But that edition is available for free in Neo4j Desktop and for startups.

Quick Intro Query Log Analyzer

The Query Log Analyzer is a tool that allows you to analyze Neo4j Cypher Query logs, that contain runtime information about queries executed by the server (query, runtime, parameters, driver information and more). It allows you to visually render those queries in tables and charts to determine and drill down on problematic queries.

There have been a number of past articles about features of the tool, please find them all here:

Kees Vegter - Medium

You can install the query log analyzer in Neo4j Desktop, it works both with local databases, as well as with log files uploaded to the tool as well as remote databases that have log streaming (via APOC) enabled. You can install the tool from the Graph App Marketplace

Neo4j Developer Tools App Gallery: One-Click Install

Query Log Mode

In Neo4j versions below 4.x we could only switch the query logging on or off
From version 4.x we have an new mode called ‘VERBOSE’. Instead of specifying ‘true’ or ‘false’ for the configuration parameter ‘dbms.logs.query.enabled’ , we must specify, in version 4+, one of the following values ‘VERBOSE’, ’INFO’ and ‘OFF’.

OFF
This will switch off the query logging completely. This is the same as ‘dbms.logs.query.enabled=false’ in previous Neo4j database versions.

INFO
When the value is INFO then the query logging has the same behaviour as we know from the previous Neo4j database versions. Queries which take more time than the given treshold will be logged after they are finished.
Note: a finished query can also be finished with an ‘error’.

VERBOSE
In this mode every query will be logged when started AND will be logged when finished. This setting will ignore the threshold, and is the default setting.

Logging the queries when started makes it possible to track the queries which started but never finished. For instances queries that were running on the server while the server was restarted. This is useful in scenarios when there are bad queries taking all the resources on the server, which could make the server non-responsive. Restart is then sometimes the only option when you don’t have time to wait.

Note that since Neo4j version 4.1 there are more possibilities to control ‘bad’ queries. Before that version there was already the option to set a transaction timeout, which aborted statements that were running longer than a given time.

With 4.1 it is now possible to set the maximum size of memory used in a transaction. When then a query hits the limit, it fails with an error without causing the server to become unresponsive.
The default for a single-instance Neo4j server is ‘no maximum’. The default for a Neo4j Cluster is 2G.
More information about memory limiting can be found here.

Query Log Analyzer 1.0.2

With this new release we support now the VERBOSE mode completely with the possibility to show the ‘unfinished queries’ and also enhance the usage of the tool by making it possible to change the query-log related configuration parameters dynamically and to stream the query log files directly from the server(s).

Unfinished Queries

When the query log is in VERBOSE mode, the start and the finish of the query is logged. In this tab the queries are shown which are not finished when the server is stopped or restarted or at the moment the query log is streamed or copied. This is useful to find problematic statements that don’t finish properly.


Dynamic Query Log Parameters

For a number of versions now, it is possible to change some database configuration parameters while the database is running. The Query Log Analyzer now has the ability to change these dynamic query log parameters in the ‘Query Log Settings’ tab.


If a property is not ‘dynamic’ or the ‘neo4j’ user does not have the proper permissions to change the settings, then the settings are displayed in a read only form.

The Query Log Analyzer now supports cluster configurations with the possibility to set the query log parameters for every cluster member.

Note @1, @2, @3 are referring to the list of connected servers in the left bar.

Log Streaming

Besides the existing functionality of uploading a query log file we added now the “Analyze Current Log” button.

When clicked, a procedure is called on the database to stream the contents of the query log file directly to the analyzer. The possibility to change the query log settings dynamically and the possibility to stream the current query log file makes it easy to temporarily switch on the query logging and analyze the query log directly without restarting the server and copying the query.log file from the server and upload it in the analyzer tool.

The streaming of the log file also supports a cluster setup. Then all the log files of the cluster members are retrieved and analyzed in one step giving the following result in the ‘Query Analysis’ tab.

Note neo4j@1, test@2, neo4j@3 are referring to the database and the list of connected servers in the left bar.

The Analyze Current Log button is only available when there is a connection to the database with the proper permissions and the APOC library installed. The procedure apoc.log.stream() must be available in the server setup.

Note that only the current query.log file is streamed, by default the query.log file follows a rotation plan where 7 log files are kept and the log file rotates when it reaches 20MB.

If inspection of those ‘rotated’ log files is needed, you have to upload them manually in the analyzer tool.

Showing Errors

For a number of Neo4j versions errors are also logged in the query.log file They will now be shown in red in the Query Log Analyzer


Filtering

In the Query Analysis, Query Log and Unfinished Queries tabs it is now possible to filter the content, to quickly find a specific query. Just enter characters in the Filter fields and hit enter to start the search.

Incremental search filters in the query log analyzer

Resources

When you want to have more insight in how to optimise the performance of your Cypher queries take a look at this post.

You can also take the Cypher Query Tuning - Online Training that teaches a lot of important skills in this area.

Cypher Query Tuning in Neo4j 4.0 - Cypher Query Tuning in Neo4j 4.0

The source code for the Query Log Analyzer is on Github at kvegter/query-analyzer-app. There you can read the documentation and report issues.

kvegter/query-analyzer-app

If you have questions regarding the query performance, you can always head to the #help-cypher channel on the Neo4j Users Slack or on the neo4j community.


---

Neo4j 4.x Query Logging Enhancements was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.