Hi folks, I’m a relatively new to Graylog and I searched high and low to learn about how messages flow through Graylog without success. Can someone please let me know how messages are processed and written to ElasticSearch and other outputs? Are processed messages stored in Journal even ElasticSearch can’t keep up with write volume? What are all the components of Graylog that write to and read from the Journal? A detailed diagram would be awesome!
I don’t have a diagram handy, but it’s pretty straightforward.
Messages come in an input, after they are processed by the input they are written to the journal, the message will then stay in the journal written on disk until it has passed through all processing and been written to opensearch successfully. This ensures that even during a power event etc no messages are lost.
As the messages pass through the system it will pass through a series of buffers as they wait for the next step. You can see all these buffers on the node page and they flow from left to right. So you will notice that a backup on the right will start to fill backwards to the left as the buffers overflow.
Hopefully that is helpful.
Thanks for your reply Joel! In addition to writing to ElasticSearch, we also send some of the messages to Splunk. Oftentimes, Splunk becomes unavailable causing Journal to become full. Is there a way to specify ignore Splunk if it’s offline? Are there any docs that go into detail on how messages are processed and written to ElasticSearch and other outputs?
In the open version of Graylog, no there is no way to not have it guarantee delivery of every message to an output, and therefore it will fill the buffers and journal when it’s down.
You can however monitor those with the metrics, and theoretically could take automated actions via the API to disable an output etc.
I don’t know of a single public location that it’s all documented in one place.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.