Disable Message Processing

[janc (Jan Caster)]

Hello, we currently designed our graylog nodes cluster so that we have dedicated web/master and processor nodes (2 web 3 processor).

However, when using kafka global inputs (threads:1) to read from a 3-partition topic the message processor threads get to be assigned to the random nodes including the ones we hope would be dedicated for web only.

Like:
web1 - processing
web2 - processing
processor1 - processing
processor2 - idle
processor3 - idle

Is there any way to completely disable message processing from a graylog node?
We have tried pausing message processing but it actually does not start new threads on the other nodes.

Also, we thought about creating individual inputs for each kafka partition, (e.g. since we have 3 partitions in one topic, we thought about creating three individual inputs with same topic), However, I am worried though that doing so might create three distinct consumer groups and so might create triplicated messages.

Do you have any recommendations in such use case?

Thanks a lot!
Jan

[jan (Jan Doberstein)]

Hej Jan,

you would need to create non global inputs if you want that some nodes did not do any kind of processing.

regards
Jan

[janc (Jan Caster)]

Hello Jan,

Thanks for the suggestion.

We used local inputs instead of global as you recommended and set threads to 1 to utilize all the processing nodes.
And if anyone is interested to know, it does not duplicate the messages.

Also, since we stopped the global inputs, graylog now thinks there might be issues since our web dedicated nodes have no running inputs.
For now we worked around this by creating an HTTP monitor input with long intervals.

Regards,
JanC