Heap of different inputs sources and 1 graylog

Graylog Central (peer support)
1

We have 1 graylog instance running in our company. We are currently sending IIS and Windows Event Logs logs to graylog.

But I want to start sending our custom applications logs’ to graylog. The way I plan on doing it is using TCP to send a GELF log to graylog. I like the idea of the _[additional field] (GELF — Graylog 4.0.0 documentation) because it opens up the possibility for powerful analytics of our application.

But I have some doubts about it.

  1. My understanding is that Elasticsearch will create an index for each additional field. What if the number of additional fields being indexed grows large? Will that cause graylog to stop working properly?
  2. What if there is a name clash between two additional fields?

To remedy the two problems above, should I try to minimise the number of additional fields? I could just shove them all into the full_messgae field, rather than each being its own field?

I should also mention that the way I have designed the logging in our application is that a programmer can log whatever key value pair he wants to as part of a log message. But this encourages the 2 problems mentioned above. Perhaps this is a bad idea? I am curious as to what other people do…

2
  1. ElasticSearch won’t create index for addition field, but only field in existing index. You can create new index with own retention in graylog.
    Index model — Graylog 4.0.0 documentation
  2. If you use same field more than once it will work fine. But you can’t use different filetype like string and number. ElasticSearch try to guest best filetype for field on first create. You can still create own mapping if you want:
    Elasticsearch — Graylog 4.0.0 documentation
3

So If I don’t explicitly create an index for an additional fields, what will I be able to do with the additional field?

4

The same as before, or if you use default index. Creating new index for every type of information (not mixing e.g windows, network devices, apps and so on) is best practice, otherwise you will end up will log of list fields from different type of sources, and also you will be able to setup separate retention (e.g remove older data after 1 month) for new index for apps.

To route data to separate index, use streams with function route to index:
https://docs.graylog.org/en/4.0/pages/getting_started/explore.html?highlight=route#streams

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.