Graylog Sidecar (1.5.1) fails (in fact Filebeat fails)

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

The filebeat agent fails after working approx. 30 minutes or longer with this error:

runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7fbcf68a8dfc m=5 sigcode=18446744073709551610

There several identical servers, only on some we get this error.

2. Describe your environment:

  • OS Information: Server: RHEL 8.10, Client: SLES15SP7

  • Package Version: Graylog Server: 7.0.3, Graylog sidecar: 1.5.1 (RPM Package)

  • Service logs, configurations, and environment variables:

The deployed sidecar configuration is this:

# Required settings

fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
hosts: [“10.2.3.100:5044”]
path:
data: ${sidecar.spoolDir!“/var/lib/graylog-sidecar/collectors/filebeat”}/data
logs: ${sidecar.spoolDir!“/var/lib/graylog-sidecar/collectors/filebeat”}/log

filebeat.inputs:

type: filestream
id: apache-filestream
enabled: true
paths:

/var/log/apache2/access.log

/var/log/apache2/error.log

/var/log/httpd/access_log

/var/log/httpd/error_log
fields_under_root: true
tags: [“linux”,“anwendung”,“loadbalancer”]
fields:
event_source_product: apache_httpd

type: filestream
id: wrapper-filestream
enabled: true
paths:

/apps/anwendung/webviewer/wrapper_prod_ports-default.log
tags: [“linux”,“anwendung”,“webviewer_wrapper”]
fields_under_root: true
multiline.pattern: ‘^\w+\s+|\s+jvm 1\s+| \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} | [\d{4}.*$’
multiline.negate: true
multiline.match: after

type: filestream
id: wrapper-filestream
enabled: true
paths:

/apps/anwendung/viewerservlet/wrapper_prod_ports-default.log
fields_under_root: true
tags: [“linux”,“anwendung”,“viewerservlet_wrapper”]
multiline.pattern: ‘^\w+\s+|\s+jvm 1\s+| \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} | [\d{4}.*$’
multiline.negate: true
multiline.match: after

type: filestream
id: tomcat-filestream
enabled: true
paths:

/apps/anwendung/webviewer/apache-tomcat-9.0.104/logs/localhost_access_log.20*.log

/apps/anwendung/viewerservlet/apache-tomcat-9.0.104/logs/localhost_access_log.20*.log
fields_under_root: true
tags: [“linux”,“anwendung”,“webviewer”]
exclude_lines: [‘./webviewer/rest/ela/user/userinfo.’,
‘./webviewer/rest/info/license.’,
‘./webviewer/VERSION.txt.’,
‘./images/toolbarButton-.’,
‘./webviewer/rest/info/version.’,
‘./webviewer/rest/documents/list.’,
‘./webviewer/rest/ela/annotation/config/show-generic-annotations.’,
‘./webviewer/rest/desk/.’,
‘./webviewer/rest/ela/annotation/templates.’,
‘./webviewer/login.’,
‘./webviewer/webjars/ui/index.html.’,
‘./webviewer/rest/ela/user/register.’,
‘./webviewer/rest/info/commit.’,
‘./webviewer/rest/ela/user/userid?.’,
‘./webviewer/rest/ui/configs?component.’]

Is there a problem in the configuration?

Thansk in advance,

Dietmar

2

@schurd Doing a quick search it could be related to the amount of log files the filebeat instance is having to track and with that thread exhaustion.

You can try increasing the thread usage available to the user running the Sidecar service although it could be an limit at a hardware level.

3

Hello Wine_Merchant,

thanks a lot for the hint. I will try to increase the thread amount. But the number of log files is not high, I would estimate 4 to 8 log files to transfer to graylog.

In the Sidecar configuration I removed the “exclude_lines”. Now the problem occurs less often.

Regards,

Dietmar