Configuring Stackdriver Logging for Neo4j VMs on GCP

[david_allen]

If you’re using Neo4j’s images on GCP, it can be a very useful thing to implement log shipping from the VM host into Google’s Stackdriver monitoring system, which is one of the best ways to do monitoring of services in their cloud.

Shipping logs to Stackdriver lets you see what’s happening on all of your hosts without individually SSH logging into them, and permits you to set up alerts so that you can be notified when certain operations events occur.

This article will describe how to set this up for Neo4j. Because Google is using software called Fluentd to get this done, this approach should also work most anywhere you want to use Fluentd and isn’t limited to Google.

Neo4j Log Shipping to Google Stackdriver

Stackdriver Logging Agent

Google provides a logging agent for Stackdriver, which is based on fluentd. Without going into all of the details, this is simply a small daemon that sits on any host where you install it, and sends logging messages to Stackdriver. You can find the documentation here.

The approach we’ll take to get this done is:

1. Install the logging agent,
2. Authorize it to send logs to Stackdriver
3. Tell the logging agent where to find Neo4j logs

And that’s it!

Install the Logging Agent

First, install the agent as per the documentation linked above:

$ curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
$ sudo bash install-logging-agent.sh
(Lots of DPKG output)
==============================================================================
Installation of google-fluentd complete.Logs from this machine should be visible in the log viewer at:  https://console.cloud.google.com/logs/viewer?project=your-project&resource=gce_instance/instance_id/...113179421906891
A test message has been sent to syslog to help verify proper operation.

Please consult the documentation for troubleshooting advice:  https://cloud.google.com/logging/docs/agent

You can monitor the logging agent's logfile at:  /var/log/google-fluentd/google-fluentd.log
==============================================================================

Give the Logging Agent Permissions

Your GCP VM has a default service account with certain privileges. They may not be sufficient to write to Stackdriver, (by default they usually aren’t) so this is something worth checking. Rather than recapping all of the docs for the agent, here is the quick version of what you need to do:

1. That the service account associated with your VM has the “Logs Writer” role
2. That the google-fluentd agent on your VM is using those service account credentials.

Point #1 you can verify by using the IAM console in Google Cloud. Click on the first IAM tab there, browse by members, locate your service account, and edit it to add the role “Logs Writer” if it is missing.

Point #2 you can verify by generating a JSON key for your service account. In the IAM view, click “Service Accounts”, select yours, and then use the “Generate Key” feature to create and download a JSON key. Place this JSON key on the VM host at the path: /etc/google/auth/application_default_credentials.json

Just to be sure, I restarted the fluentd service like this:

sudo systemctl restart google-fluentd

If you run into any trouble with these steps, there are quite a few troubleshooting steps for the logging agent in the docs.

Test it out!

Before we get to the Neo4j parts, let’s make sure it works.

david_allen@bitbucket-vm:/etc/google/auth$ logger "Is this crazy stackdriver thing working?"
david_allen@bitbucket-vm:/etc/google/auth$

If you then go to the main GCP console, and select Logging -> Logs, you should see your message. This proves that the logging agent works, and that your permissions are good.

Log messages received by StackDriver

You can also confirm that the agent is happy by tailing the logging agent’s logfile:

$ tail -n 1 /var/log/google-fluentd/google-fluentd.log 2019-03-14 12:23:24 +0000 [info]: #0 Successfully sent gRPC to Stackdriver Logging API.

Looking good!

Tell Stackdriver About Neo4j Log Files

Create a single file as /etc/google-fluentd/config.d/neo4j.conf with the following content:

<source>  
   @type tail  
   format none  
   path /var/log/neo4j*/*.log  
   pos_file /var/lib/google-fluentd/pos/neo4j.pos  
   read_from_head true  
   tag neo4j
</source>

This should be pretty straightforward what we’re telling google-fluentd to do. Monitor all log files in that directory, always read from the beginning, keep track of where you are with the specified position file, and tag all of the records with “neo4j”.

One systemctl restart google-fluentd and you should start seeing Neo4j logs flowing through!

If you need the full details on how to configure the agent, to understand how and why this works, they can be found here.

Don’t Forget Systemd Log files!

In certain packaged installations of Neo4j, which include the Neo4j Cloud VMs, the neo4j service runs in systemd, and doesn’t output a regular neo4j.log file to /var/log/neo4j.

Tip: if you have a neo4j.log file, then this step doesn’t apply to you! If you use journalctl to access your neo4j logs, this step does apply to you.

Create a new config file called /etc/google-fluentd/config.d/neo4j-systemd.conf with this content:

<source>
  @type systemd
  path /run/log/journal
  filters [{ "_SYSTEMD_UNIT": "neo4j.service" }]
  pos_file /var/lib/google-fluentd/pos/neo4j-systemd.pos
  read_from_head true
  tag neo4j
</source>

This configuration uses a fluentd plugin for systemd to get logs straight out of journal files.

Tip: Take particular note of the path! This path works with most Neo4j VMs, but different linux distros put systemd journal files in different places.

Restarting the system again, we can see now that systemd logs are flowing through as expected:

Showing highlighted syslog entries coming through. Notice below that you can see an entry with logName neo4j, so we can distinguish between which log stream source is coming through.

Start Using Stackdriver!

Now that you’ve got all of your logs shipped over, you can start to use all of Stackdriver’s features. I’ll leave you with the simplest possible example — looking through logs tagged “neo4j” for the “logged in” message, which from Neo4j’s security.log will give you a complete accounting of who logged in when.

Happy graph hacking.

---

Configuring Stackdriver Logging for Neo4j VMs on GCP was originally published in neo4j on Medium, where people are continuing the conversation by highlighting and responding to this story.