Duplicate messages in graylog using winlogbeat

Graylog Central (peer support)
,
1

Hi All,

I have an issue. My winlogbeat sends duplicate and even in one event I found x150 same messages in graylog where in windows event logs there is only 1 message. Anyone is facing the same issue?

Winlogbeat: 5.5.2
Graylog: 2.3.0
Elasticsearch: 5.5.2

Three related error messages (graylog side for beats input):
WARN [AbstractTcpTransport] TLS key file or certificate file does not exist, creating a self-signed certificate for input [Beats/59ba13e50f1d6c6ef11fdac4].

INFO [AbstractTcpTransport] Enabled TLS for input [Beats/59ba13e50f1d6c6ef11fdac4]. key-file="/tmp/keyutil_GraylogIP:null_5406919365084177135.key" cert-file="/tmp/keyutil_GraylogIP:null_936516737847671089.crt"

WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=Windows Events, type=org.graylog.plugins.beats.BeatsInput, nodeId=c6457b59-c9e9-4557-8690-79ec22e7f5bd} should be 1048576 but is 212992.

Winglogbeat settings:
winlogbeat.event_logs:

  • name: ForwardedEvents
    ignore_older: 24h
    output.logstash: # used by Graylog
    hosts: [“graylogIP:5044”]
    ssl.enabled: true
    ssl.verification_mode: none
    compression_level: 4
    worker: 2
2

@jaskis

are you sure, that you have only one winlogbeat running that transfers the message?

Did you have multiple index sets and streams that route the message into index sets and the default?

3

I am confident that only 1 winlogbeat instance is running. Also I have double checked it now - 1 service enabled and 1 process is running. The point is that not all messages gets duplicated. Some does not, some gets quadrupled.

I have only default index with no streams associated to windows events. I had the same experience with nxlog.

4

@jan,

Is there a smart way to debug this?

5

Testing a workaround, played around the settings on winlogbeat. Seems like as if it has fixed the issue. Added batch read size and bulk max size. I have a suspicion that batch read size by default is low and is not coping well with forwardedevents which are not very structure by nature. Will update once more time will pass.

winlogbeat.event_logs:

  • name: ForwardedEvents
    batch_read_size: 512

output.logstash: # used by Graylog
hosts: [“GraylogIP:5044”]
ssl.enabled: true
ssl.verification_mode: none
compression_level: 3
bulk_max_size: 512

1 Like
6

Same issue persists, now I have noticed something else. There can be 100x events with the same “winlogbeat_record_number” however different graylog message code. All those 100x messages are identical. There is a bug wither in winlogbeat or graylog beats plugin.

7

@jan,

Do you have a way to see metrics/debug logs of beats plugin in graylog? I need to debug this in some way as on winlogbeat side seems all good according to logs.

8

when you go to ‘System > Nodes’ you are able to access all metrics of each node. So you are able to find the metrics of you input and so you are able to write them somewhere else.

If that is what you are looking for.
Jan

9

Not entirely. I am looking for a log where I could see how many logs by beats were delivered per batch, whether any ack’s failed, resends, etc.

Could a sidecar help in this case? I mean to manage winlogbeat services.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.