Logstash Output Plugin

[hank]

On the page “Neo4j and Elastic Search” at the bottom there’s a section about Logstash plugins written by Pere Urbon. But the section is empty bullet points. A brief look at Github I didn’t find them. Has anyone thought about using Neo4j as the datastore for the Elastic Stack? Seems like if you have a good semantic model for all the things ES logs, you could get a lot of insights into a cloud application.

[hank]

Ok, here it is:

GitHub

logstash-plugins/logstash-output-neo4j

A Logstash neo4j output. Contribute to logstash-plugins/logstash-output-neo4j development by creating an account on GitHub.

[hank]

Well... I’m in JRuby configuration hell. The plugin uses a few different libraries than the latest packaged with Ubuntu 16.04. If anyone has a recommendation for that, let me know. Also, not sure if the plugin. This project seems to be really comprehensive (although it does not have logger integrations):

GitHub

LendingClub/mercator

produce graph model projections of infrastructure. Contribute to LendingClub/mercator development by creating an account on GitHub.

[michael_hunger]

Interesting idea, I guess so far no one has brought this up. Most work has been in the opposite direction,
also more recent ones like this from @david.allen

Medium – 23 May 19

Neo4j Logging/Monitoring with Elastic Cloud and ELK Stack

Setting up logging and metrics

Reading time: 4 min read

[hank]

When I found that article, I knew that I wasn't alone in the world 😉

Elastic seems to be a terrific solution -- I don't have experience using it in a production environment. But there are a lot of links, ultimately would like to be able to trace those back to commits and lines of source code, configurations, etc.

[szenyo]

hey, I have just found this topic. We have a logstash for hosting data ingestion pipelines. From kafka, files, jdbc to other databases, mainly SQL servers. I was looking for a new ingestion where I would like to load the Neo4j somehow, so I am looking something like this Neo4j output plugin.

@hank do you have any experience with the plugin so far? I am not sure it is maintained properly.

[hank]

Hi Janos,

My brief look at the plugin, and that was a year ago, is that it is limited and it was using an old version of the API. So I lost interest...

That said, I think Logstash has such a wide array of collectors -- if you could push events, then go back and create other relationships. I recall the pattern for time series data is simply one node connected to the next. So I never thought it could, in real time, create the kind of relationships that squash down all the events into something meaningful.

It did not look too difficult to write a plugin for TS data, but it was out of scope if there were not supported tools.

Let me know if you take it up.

Best,
Hank
i/o werx