How does Graylog work with ES and MongoDB?

Here is the short - when log messages come in via a Graylog input, Graylog uses the settings it has stored in MongoDB to figure out how to handle the message (Settings like inputs, streams, extractors, pipelines, rules, Alerts … all stored in MongoDB) once it has finished processing the message (breaking out fields and other calculations), it sends it out to the Elasticsearch Database to store the message. Dashboards and queries on the data are sent from Graylog to Elasticsearch/Opensearch database and the results are displayed in the Graylog Gui.

The default install has all these products installed on one machine and all meaasges are stored on the Graylog server in it’s Elasticsearch database instance … @gsmith answered a question earlier about where the Elasticsearch database is stored. For how fast it (Elasticsearch) grows daily, it depends on how many messages get sent and what the size of those messages are… for instance a firewall is going to send a lot more messages if internet usage is up because people are in the office (or whatever) and the size of a message may differ on the even and how much data you are sending in There are a few Windows logs that ship in extra paragraphs that simply explain the same event the same way every message and that part could be dropped to save a small amount of space… depending ont he volume coming in of course.

There is a good thread (Discussion and charts) here that explains how a message flows through Graylog that will help to visualize it.

1 Like