Incoming vs Outgoing Messages

Graylog Central (peer support)
1

Hi everyone,

I have had a look around but have not been able to find an answer on this one.

We have just deployed a new Graylog setup (Currently only 1 Graylog server and 1 separate Elasticsearch server) and I would like to know what I should be expecting with the In vs Out messages.

I would expect that they would be roughly the same, however at the moment our incoming number is a lot higher then the outgoing (currently 247in/82 out). I have also seen the Out significantly higher.

So I guess my questions are:

  1. Should the In/Out number of messages be relatively even?
  2. What does a much higher In than Out indicate?
  3. What does a much higher Out than In indicate (slow writing to Elasticsearch?)

Thanks

2

So I guess my questions are:

  • Should the In/Out number of messages be relatively even?

yes - in a well balanced setup that should be. Only peaks might have higher input that output.

  • What does a much higher In than Out indicate?

Higher input means that the writing to Elasticsearch (out) can’t write the same speed as messages are coming in.

  • What does a much higher Out than In indicate (slow writing to Elasticsearch?)

Higher out indicate that the journal (the Graylog buffer) is draining out the messages that are buffer.

3

also if you have pipelines setup to drop certain messages, the output will not match the input

4

Thanks guys. We are now pretty constantly seeing about double the out vs in (247/547). The disk journal utilization is at 1%, Input,Output and Process buffers are sitting at 0%.

Does this seem to indicate a problem?

5

do you have pipelines rules setup to make a copy of messages or some other output setup?

6

No, no pipelines or outputs configured at all. Just a few streams and dashboards…

7

Hej @Steve

it looks like you have two index sets - one default and one additional and messages are routed into two streams that have both different index sets.

Please read the documentation on Index sets:

http://docs.graylog.org/en/2.3/pages/streams.html#storage-requirements

8

Thanks Jan. Actually I somewhat figured this out this afternoon but forgot to update this thread. It clicked in my mind that the only non standard thing we were using was the netflow plugin. I disabled this input and then sure enough the in/out was 50/50.

So I am not sure now if the plugin or netflow itself is the issue. But at least I am narrowing it down and have stopped the double up for now.

Appreciate everyones thoughts/help on this.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.