Graylog Sidecar Filebeat dropping messages short_message field is empty

Graylog Central (peer support)
,
1

This used to be fine, not sure when it stopped working but looking at the Graylog server log file I see

> 2021-06-03T11:46:43.346-04:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=e51a981c-c482-11eb-b083-00505688d876, journalOffset=1685303024, codec=gelf, payloadSize=332, timestamp=2021-06-03T15:46:43.345Z, remoteAddress=/1.1.1.1:61164} on input <5f3ab0835abaa1748e473ae7>.
> 2021-06-03T11:46:43.346-04:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=e51a981c-c482-11eb-b083-00505688d876, journalOffset=1685303024, codec=gelf, payloadSize=332, timestamp=2021-06-03T15:46:43.345Z, remoteAddress=/1.1.1.1:61164}
> java.lang.IllegalArgumentException: GELF message <e51a981c-c482-11eb-b083-00505688d876> (received from <1.1.1.1.1:61164>) has empty mandatory "short_message" field.

I’m not sure why it’s missing but I did try to manually add it to the filebeat config file, that made no difference. Below is the filebeat.config (with the manual short_field set)

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: SERVERNAME
fields.gl2_source_collector: 854d8ed5-c285-4e49-b7aa-f2ad1fdf86a9
fields.short_message: SERVERNAME

output.logstash:
   hosts: ["1.1.1.1:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
filebeat.inputs:
- type: log
  encoding: utf-16le-bom
  multiline.type: pattern
  multiline.pattern: '^(.*)$'
  multiline.negate: false
  multiline.match: after
  enabled: true
  paths:
    - 'C:\Windows\Temp\*Error*.log'
    - 'C:\Windows\Temp\*error*.log'

Does anybody have the same issue or a solution?

2

This isn’t required for the filebeat config to work, why do you have it in there? If you want to have the server name in the field short_message you could put it in like this:

...
filebeat.inputs:
- type: log
  encoding: utf-16le-bom
  fields:
    short_message: ${sidecar.nodeName}
  multiline.type: pattern
...
3

In the config pushed down by Graylog that’s what I have, in the post above thats how it shows on the server when deployed.

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
fields.short_message: ${sidecar.nodeName}

output.logstash:
   hosts: ["1.1.1.1:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
filebeat.inputs:
- type: log
  encoding: utf-16le-bom
  multiline.type: pattern
  multiline.pattern: '^(.*)$'
  multiline.negate: false
  multiline.match: after
  enabled: true
  paths:
    - 'C:\Windows\Temp\*Error*.log'
    - 'C:\Windows\Temp\*error*.log'

As far as short_message not being required the Graylog server log is indicating it is a required field. Are you saying my config file should look like this, instead of how it is above?

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["1.1.1.1:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
filebeat.inputs:
- type: log
  encoding: utf-16le-bom
  fields:
    short_message: ${sidecar.nodeName}
  multiline.type: pattern
  multiline.pattern: '^(.*)$'
  multiline.negate: false
  multiline.match: after
  enabled: true
  paths:
    - 'C:\Windows\Temp\*Error*.log'
    - 'C:\Windows\Temp\*error*.log'
4

I am not sure why you have that line in there but it is not required to have a working filebeat. I do not have that line in mine. There may be a dependency that is unique to your setup for it and there may be a server(s) that is not filling out the ${sidecar.nodeName} properly for it’s filebeat…?

5

I only added that line after it stopped working and the Graylog-Server had the error

 2021-06-03T11:46:43.346-04:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=e51a981c-c482-11eb-b083-00505688d876, journalOffset=1685303024, codec=gelf, payloadSize=332, timestamp=2021-06-03T15:46:43.345Z, remoteAddress=/1.1.1.1:61164}
> java.lang.IllegalArgumentException: GELF message <e51a981c-c482-11eb-b083-00505688d876> (received from <1.1.1.1.1:61164>) has empty mandatory "short_message" field.

empty mandatory “short_message” field.

This seems to indicate something has changed in what Graylog is required to process the log. It looks like something changed in a little over the last 30 days.
Because I have notification from 5/18 for the alert tied to this.

6

It’s possible it’s something set up in the extractor?

7

I just blew it away and rebuilt it. Didn’t seem to change anything. Are you aware of any other logs I can review for sidecar or graylog other than

var/log/graylog-server
C:\Program Files\Graylog\sidecar\logs

Because Im not finding any errors other than the short_field

8

Those are the two areas I would check. What is your input type for windows machines? I am asking because the error mentions GELF when the input should be beats

9

The windows is using TCP 12202.
I looked around some more and you are correct it looks like that might be referring to something else. However, that’s the only error I’m seeing besides enterprise plugin stuff. (not an enterprise customer)

10

Looking at the sidecar.log Im now seeing

time="2021-06-04T07:15:11-04:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file." 
time="2021-06-04T07:15:12-04:00" level=info msg="[filebeat] Stopping" 
time="2021-06-04T07:15:12-04:00" level=info msg="[filebeat] Starting (svc driver)" 
time="2021-06-04T07:39:15-04:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file." 
time="2021-06-04T07:39:15-04:00" level=info msg="[filebeat] Stopping" 
time="2021-06-04T07:39:15-04:00" level=info msg="[filebeat] Starting (svc driver)" 
time="2021-06-04T08:32:50-04:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://GRAYLOGSERVERIP:9000/api/sidecars/f62ea292-85f8-4286-bb34-91239d167dc2: dial tcp GRAYLOGSERVERIP:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2021-06-04T08:33:01-04:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://GRAYLOGSERVERIP:9000/api/sidecars/f62ea292-85f8-4286-bb34-91239d167dc2: dial tcp GRAYLOGSERVERIP:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2021-06-04T08:33:12-04:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://GRAYLOGSERVERIP:9000/api/sidecars/f62ea292-85f8-4286-bb34-91239d167dc2: dial tcp GRAYLOGSERVERIP:9000: connectex: No connection could be made because the target machine actively refused it."

I’m not sure why the connection would be refused for sidecar, on the graylog server its reporting normally
image

13

OK…
So…
This has to be a sidecar issue, I removed the filebeat sidecar install.
Then installed the latest client from https://www.elastic.co/downloads/beats/filebeat
Edited the filebeat.yml to match the sidecar filebeat.config
Started the new filebeat service
And…
Wallah we have logs in Graylog and port 5044 shows client connections. I still have nxlog running through sidecar without issue though.

Any ideas on how to resolve that?

14

Tried updating sidecar from 1.0.2 to 1.1.0 this also made no difference.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.