i am trying to understand a little more about internal workings on graylog. Regarding process chain, is it correct to say that parsing of messages into fields for structure data, pipeline processing, firing of rules etc. happen in input process happens,
my questions is what happens in output buffer before the processed data is passed into elastic search.
Thanks, one more query what does this mean
“we connect to ES servers as an embedded ES node that does not store data [<- specifically this statement], so we look and act like an ES node, and know about configuration data (indexes, shards, etc) for each ES server” When writing to E and when you are not a node, you have to encode and transmit over the wire as HTTP and then json and then decode it. as a node you can send it in native format, and it is fast.
That quote was true until Graylog 2.3.0, which introduced the Elasticsearch HTTP client in order to be less dependent on the actual Elasticsearch version.