Ntopng to graylog

[markinhuszn]

Hello,
So I might of figured out a way.
I would send ntopng to MySQL
Elasticsearch would then get the logs from MySQL and forward to Graylog.
Does that sound feasible?

[Totally_Not_A_Robot (Tess)]

Technically speaking you could pipe it directly into ElasticSearch.

https://www.ntop.org/guides/ntopng/advanced_features/flows_dump.html#elasticsearch

If I’m not mistaken, Graylog can still search through the log-data as long as it ends up in one of its indices.

[markinhuszn]

@Totally_Not_A_Robot

g can still search through the log-data as long as it ends up in one of its indices.

Would it not be better to use MySQL if optimization is a requirement?

[Totally_Not_A_Robot (Tess)]

My point was that I believe that Graylog can still query messages that were put into Elasticsearch without the actual help of Graylog itself. @Jan or @Jochen could confirm this.

So I suggested telling nTopNG to talk directly to the ElasticSearch backend and have Graylog query it. Instead of going nTopNG -> MySQL -> Graylog ->Elastic -> Graylog query.

Take that with a tonne of salt; I am only a newb.

[markinhuszn]

What you say sounds reasonable so it’d be good to get a mod to share their knowledge on this.
I am just worried that sending it directly to Elasticsearch it would bring performance issues later on. I am looking to store 1 year worth of logs. Winevt, Cisco & ntopng.

[Totally_Not_A_Robot (Tess)]

@markinhuszn

I am just worried that sending it directly to Elasticsearch it would bring performance issues later on. I am looking to store 1 year worth of logs

If you’re going to store and query logs with Graylog, then you’re going to have to put them into Elastic. I haven’t seen evidence that Graylog2 can query a MySQL database for logging data.

So perhaps I’m misunderstanding your situation and your question.

[markinhuszn]

My plan was to store in MySQL and then use Elasticsearch to query. Mainly because Elasticsearch was never intended to be used for datastoraged.

edit: was thinking of using logstash to get from MySQL to Elasticsearch.

[macko003]

Just some comments.
You do not have to take the advice (from (maybe a) more experienced person). As I read, you guessed it, so do it. Please share your experience.
Graylog add some extra data to the items what stored in elasticsearch (eg. message id, sender address, receiver server id, etc.), take care about it when you import data from mysql. You can check in the graylog’s code what will do the graylog if you don’t use it, I’m not sure.
Try to keep in sync the graylog index retention settings with your import solution, to avoid data loss.

[markinhuszn]

wouldnt I run into issues if I use elasticsearch as my primary datastore? Talking about 1 year of logs here.
Also @Macko003 I hear what you are saying, thought it’s abit hard to comprehend (sorry if that sounded rude, not my intention). Any other input?

[jan (Jan Doberstein)]

wouldnt I run into issues if I use elasticsearch as my primary datastore?

Nope - I know environments aiming for 10 years of storage in Elasticsearch.

[Totally_Not_A_Robot (Tess)]

@macko003

You do not have to take the advice (from (maybe a) more experienced person). As I read, you guessed it, so do it. Please share your experience.

Apologies on my part if my posts sounded too much like I’m trying to steer @markinhuszn’s actions. I do not mean to suggest any authority on the subject of Graylog and its constituent parts. I’m still very much a newbie, quickly learning as much as I can.

[markinhuszn]

Thanks for the confirmation. Just out of curiosity, why would you ever run MySQL with Elasticsearch? I mean what would be the scenario? Thanks

[jan (Jan Doberstein)]

Just out of curiosity, why would you ever run MySQL with Elasticsearch?

Is that possible? What is the role MySQL should have?

[markinhuszn]

The role would’ve just been primary data storage. Then Elasticsearch would query it.

[jan (Jan Doberstein)]

For that purpose you would need to write your own logging software that can do this …

neither Elasticsearch or Graylog are able todo this.

[markinhuszn]

@jan

neither Elasticsearch or Graylog are able todo this.

So Elasticsearch can’t query into MySQL. So, how would one combine both these softwares?

[Totally_Not_A_Robot (Tess)]

@markinhuszn

So, how would one combine both these softwares?

Which are the “both” in this question? Elastic? Graylog? NTopNG? MySQL?

• ElasticSearch is a storage engine, kind of like a backend database. It’s just targeted at a very specific type of use-case and not a generic SQL database.
• MySQL is a generic SQL database
• NTopNG is a network monitoring tool that can send its output to various destinations (incl. both Elastic and MySQL)
• Graylog is the wonderful log aggregation and query tool that this site is about. It’s built on top of Elastic.

MySQL is for the storage of generic application data, it could be whatever. Graylog and Elastic are mostly intended for log management and analysis, though undoubtedly you can shoehorn them into other roles. If you want to analyse MySQL logs, then that’s no problem! If you want to analyse NTop logs, then that’s no problem. But if you want to query data stored in MySQL, that’s not what ES/Graylog are for.

[markinhuszn]

Thanks, but that I already knew.
Perhaps not formulating the question correctly.
Would it be more optimum in terms of performance to use MySQL or ElasticSearch?

[Totally_Not_A_Robot (Tess)]

@markinhuszn

Would it be more optimum in terms of performance to use MySQL or ElasticSearch?

Whose performance? The write performance of NTopNG? The query performance of NTopNG? Something else?

These are questions I feel you are better off asking the NTopNG people because it pertains to their application code and not Graylogs… If you’re going to use the NTopNG GUI to query, then all the more so: ask them, not here.

They have some FAQs but they don’t necessarily cover the performance of Elastic vs MySQL as their storage backend.

EDIT:
Heh, there are people using both at the same time, for separate parts of the NTopNG functionality.

Use MySQL and ElasticSearch together, "all in one" nProbe Template?

EDIT 2:
Right… I’ve gone over your post-history a little bit and apparently you DO want to use Graylog to query the NTopNG data, instead of using the NTopNG GUI. That was not clear to me from just this thread.

So yeah, you are in the right place to ask about Graylog. Your current challenge is that NTopNG can directly output into ElasticSearch, but doing so would bypass the meta-data addition that Graylog usually does (as @macko003 mentioned). That’s not something that is easily solved.

Back in 2015 the NTopNG people made an effort to nicely integrate with ELK though…

Exploring your traffic using ntopng with ElasticSearch+Kibana

So it’s been done, but for different tools.

EDIT 3:
So… NTopNG can send to three different outputs: Elastic, MySQL and LogStash (source).

• Graylog nor Elastic can work with MySQL
• Putting data directly into Elastic leaves it without the Graylog metadata.
• Which leaves LogStash…

There are ways of getting LogStash streams into Graylog (for example). So you could make it work by doing:

NTopNG → LogStash → ??? → Graylog → Elastic

EDIT 4:
Well looky here! The Marketplace has an input for LogStash. And it also notes something important: it has been replaced with the standard BEATS input. Does LogStash speak BEATS by default? Then this could be your saving grace

NTopNG -> LogStash (BEATS) -> Graylog -> Elastic

[markinhuszn]

Yes, I was able to do this using ELK. Never thought about using Logstash to Graylog. Thanks for the effort you put in man, if any questions arises I will edit this.