Pthread_create failed: Operation not permitted

Graylog Central (peer support)
1

1. Describe your incident:

Try to run filebeat-linux with either journald or log as type , and I got same error everytime:

runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f9782d1f58c m=4 sigcode=18446744073709551610

2. Describe your environment:

  • OS Information:

    • Server
      Docker: 24.0.2
      Graylog: 5.2.2
      Opensearch: 2.11.0
      MongoDB: 6.0.11
      Traefik: 2.10.7

    • Client
      Rocky Linux: 9

  • Package Version:

Filebeat: 7.12.1

  • Service logs, configurations, and environment variables:

Sidecar conf

server_url: "https://graylog.lab.lan/api"
server_api_token: "my_api_token"
node_id: "file:/etc/graylog/sidecar/node-id"
node_name: ""
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files: []
cache_path: "/var/cache/graylog-sidecar"
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10
tags:
 - filebeat-linux

collector_binaries_accesslist:
  - "/usr/lib/graylog-sidecar/filebeat"

Filebeat-linux conf

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["graylog.lab.lan:5044"]
   
path.data: /var/lib/graylog-sidecar/collectors/filebeat/data
path.logs: /var/lib/graylog-sidecar/collectors/filebeat/log
path.module: /var/lib/graylog-sidecar/collectors/filebeat/modules

filebeat.inputs:

- type: journald
  id: winbind
  include_matches:
    - _SYSTEMD_UNIT=winbind.service
  parsers:
    - multiline:
        type: pattern
        pattern: ^[[:space:]]
        match: after
        negate: false
  tags: [ 'winbind' ]

3. What steps have you already taken to try and solve the problem?

I tried different configuration without success.
I’ve already check permissions everything is running under root user and has good permission on files.

4. How can the community help?

Is there anything wrong with my conf ?

See below FULL DETAIL error log:

runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f9782d1f58c m=4 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f9782d1f58c
stack: frame={sp:0x7f975afe6790, fp:0x0} stack=[0x7f975a7e71e8,0x7f975afe6de8)
00007f975afe6690:  0000000000000001  0000000004b6af71 
00007f975afe66a0:  0000000001365d42  0000000000000000 
00007f975afe66b0:  0000000000000000  0000000006141c80 
00007f975afe66c0:  0000000004b6af38  00007f975afe6758 
00007f975afe66d0:  000000000173abdb <runtime.scanframeworker+155>  00007f975afe69f8 
00007f975afe66e0:  00007f975afe6ae0  0000000004256c00 
00007f975afe66f0:  0000000000000000  0000000000000000 
00007f975afe6700:  0000000000000000  0000000000000000 
00007f975afe6710:  0000000000000000  0000000000000000 
00007f975afe6720:  0000000000000000  00000001001e6daf 
00007f975afe6730:  0000000000000000  00007f975afe6870 
00007f975afe6740:  0000000000000000  0000000000000000 
00007f975afe6750:  000000000178bac0 <runtime.goexit+0>  00007f975afe6780 
00007f975afe6760:  00000000017805fd <runtime.scanstack.func1+61>  00007f975afe69f8 
00007f975afe6770:  00007f975afe6ae0  00007f9782d2cc49 
00007f975afe6780:  0000000000000000  00007f9782d1f57e 
00007f975afe6790: <0000000000000130  0000000000000000 
00007f975afe67a0:  0000000000000120  00007f975afe6870 
00007f975afe67b0:  00007f9700000013  00007f975afe68df 
00007f975afe67c0:  00007f9758d33640  0000000000000004 
00007f975afe67d0:  0000003400000013  00007f9782dcd2f1 
00007f975afe67e0:  00007f973bfff640  00007f975afe6aa0 
00007f975afe67f0:  00007f975afe691e  00007f975afe691f 
00007f975afe6800:  00007f973bfff640  00007f9782d1d445 
00007f975afe6810:  00007f9750000020  9f89e2c8cc92d500 
00007f975afe6820:  00007f975afe7640  0000000000000006 
00007f975afe6830:  00007f9750000c90  0000000000000000 
00007f975afe6840:  000000000473d3ec  00007f9782cd2d06 
00007f975afe6850:  00007f9782e79e90  00007f9782ca67f3 
00007f975afe6860:  0000000000000020  0000000000000000 
00007f975afe6870:  0000000000000000  00007f9782dbc9f4 
00007f975afe6880:  00007f973b7ff000  000000000000000d 
runtime: unknown pc 0x7f9782d1f58c
stack: frame={sp:0x7f975afe6790, fp:0x0} stack=[0x7f975a7e71e8,0x7f975afe6de8)
00007f975afe6690:  0000000000000001  0000000004b6af71 
00007f975afe66a0:  0000000001365d42  0000000000000000 
00007f975afe66b0:  0000000000000000  0000000006141c80 
00007f975afe66c0:  0000000004b6af38  00007f975afe6758 
00007f975afe66d0:  000000000173abdb <runtime.scanframeworker+155>  00007f975afe69f8 
00007f975afe66e0:  00007f975afe6ae0  0000000004256c00 
00007f975afe66f0:  0000000000000000  0000000000000000 
00007f975afe6700:  0000000000000000  0000000000000000 
00007f975afe6710:  0000000000000000  0000000000000000 
00007f975afe6720:  0000000000000000  00000001001e6daf 
00007f975afe6730:  0000000000000000  00007f975afe6870 
00007f975afe6740:  0000000000000000  0000000000000000 
00007f975afe6750:  000000000178bac0 <runtime.goexit+0>  00007f975afe6780 
00007f975afe6760:  00000000017805fd <runtime.scanstack.func1+61>  00007f975afe69f8 
00007f975afe6770:  00007f975afe6ae0  00007f9782d2cc49 
00007f975afe6780:  0000000000000000  00007f9782d1f57e 
00007f975afe6790: <0000000000000130  0000000000000000 
00007f975afe67a0:  0000000000000120  00007f975afe6870 
00007f975afe67b0:  00007f9700000013  00007f975afe68df 
00007f975afe67c0:  00007f9758d33640  0000000000000004 
00007f975afe67d0:  0000003400000013  00007f9782dcd2f1 
00007f975afe67e0:  00007f973bfff640  00007f975afe6aa0 
00007f975afe67f0:  00007f975afe691e  00007f975afe691f 
00007f975afe6800:  00007f973bfff640  00007f9782d1d445 
00007f975afe6810:  00007f9750000020  9f89e2c8cc92d500 
00007f975afe6820:  00007f975afe7640  0000000000000006 
00007f975afe6830:  00007f9750000c90  0000000000000000 
00007f975afe6840:  000000000473d3ec  00007f9782cd2d06 
00007f975afe6850:  00007f9782e79e90  00007f9782ca67f3 
00007f975afe6860:  0000000000000020  0000000000000000 
00007f975afe6870:  0000000000000000  00007f9782dbc9f4 
00007f975afe6880:  00007f973b7ff000  000000000000000d 

goroutine 1 [runnable]:
os/signal.signal_enable(0xf)
	/usr/local/go/src/runtime/sigqueue.go:216 +0x72
os/signal.enableSignal(...)
	/usr/local/go/src/os/signal/signal_unix.go:49
os/signal.Notify.func1(0xf)
	/usr/local/go/src/os/signal/signal.go:144 +0x88
os/signal.Notify(0xc000532300, 0xc000b273d0, 0x3, 0x3)
	/usr/local/go/src/os/signal/signal.go:164 +0x162
github.com/elastic/beats/v7/libbeat/service.HandleSignals(0xc0008c1320, 0xc000110210)
	/go/src/github.com/elastic/beats/libbeat/service/service.go:48 +0x125
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc0008b2000, 0x42a595d, 0x8, 0x42a595d, 0x8, 0x0, 0x0, 0x101, 0x0, 0x0, ...)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:461 +0x62d
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0x42a595d, 0x8, 0x42a595d, 0x8, 0x0, 0x0, 0x101, 0xc000b27c20, 0xc000856ad0, 0x0, ...)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:193 +0x5a8
github.com/elastic/beats/v7/libbeat/cmd/instance.Run(0x42a595d, 0x8, 0x42a595d, 0x8, 0x0, 0x0, 0x101, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:194 +0x105
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc0008b22c0, 0xc0008c0100, 0x0, 0x2)
	/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x85
github.com/spf13/cobra.(*Command).execute(0xc0008b22c0, 0xc00004e0a0, 0x2, 0x2, 0xc0008b22c0, 0xc00004e0a0)
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc0008b22c0, 0x0, 0xffffffff, 0xc00009e058)
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x30b
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main()
	/go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22 +0x34

goroutine 12 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000544a50)
	/go/pkg/mod/go.opencensus.io@v0.22.2/stats/view/worker.go:154 +0x105
created by go.opencensus.io/stats/view.init.0
	/go/pkg/mod/go.opencensus.io@v0.22.2/stats/view/worker.go:32 +0x57

goroutine 10 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x637eb80)
	/go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
	/go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:416 +0xd8

goroutine 70 [chan receive]:
github.com/elastic/go-lumber/client/v2.(*AsyncClient).ackLoop(0xc000945590)
	/go/pkg/mod/github.com/elastic/go-lumber@v0.1.0/client/v2/async.go:149 +0xda
created by github.com/elastic/go-lumber/client/v2.(*AsyncClient).startACK
	/go/pkg/mod/github.com/elastic/go-lumber@v0.1.0/client/v2/async.go:123 +0x8e

goroutine 71 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc000418e60)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x1d0
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func1(0xc0008a3d50, 0x47c9b00, 0xc000418e60)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:176 +0x63
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:174 +0x3cc

goroutine 72 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc000271270)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x118
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2(0xc0008a3d50, 0xc000271270)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:180 +0x59
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:178 +0x3fe

goroutine 73 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc0008c11a0, 0x800, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:65 +0xf1
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc000461140, 0x47c9b40, 0xc0008c11a0)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:182 +0x1a2
github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1(0xc000461140, 0x47c9b40, 0xc0008c1180)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:86 +0x6c
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:84 +0x153

goroutine 74 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*retryer).loop(0xc0004612c0)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:135 +0x253
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newRetryer
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:94 +0x15d

goroutine 75 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc000461800)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:127 +0xcb
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x194

goroutine 76 [select]:
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc00094e120)
	/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:128 +0x40a
github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1(0xc00094e120)
	/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:106 +0x50
created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter
	/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:104 +0x15c

goroutine 53 [syscall]:
os/signal.signal_recv(0x4461d50)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:150 +0x45

rax    0x0
rbx    0x7f975afe7640
rcx    0x7f9782d1f58c
rdx    0x6
rdi    0x1341
rsi    0x1344
rbp    0x1344
rsp    0x7f975afe6790
r8     0x7f975afe6860
r9     0x7f9782e2f4e0
r10    0x8
r11    0x246
r12    0x6
r13    0x0
r14    0x473d3ec
r15    0x0
rip    0x7f9782d1f58c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
2

One thing I understand is that with the latest compatible Beats version for Opensearch 2.X which is the 7.12.1.0.

Filebeat 7.12.1.0 does not have the journald inputs as it was introduced in later versions when journalbeat was deprecated.

For now, if I want to get logs from journald, I need to use journalbeat binary and not filebeat.

This info itself does not change the facts it does not work and result in same error if I use input type: log for Filebeat.

3

After investigation, it seems the cause of failure for filebeat 7.12.1.0 is docker engine.
I just deployed the same sidecar conf on another Rocky Linux OS client, this one does not have docker engine installed and the sidecar is running without errors.

I tested then with many filebeat binary version to see which one is working and not with docker engine installed on the host.

  • 7.13.1 → same error than 7.12.1
  • 7.14.1 → same error than 7.12.1 and 7.13.1
  • 7.15.1 → same error than 7.12.1 and 7.13.1 and 7.14.1
  • 7.16.1 → same error than 7.12.1 and 7.13.1 and 7.14.1 and 7.15.1
  • 7.16.2 → old error fixed but new error: Fatal glibc error: rseq registration failed
  • 7.16.3 → same error than 7.16.2
  • 7.17.0 → same error than 7.16.2 and 7.16.3
  • 7.17.1 → same error than 7.16.2 and 7.16.3 and 7.17.0
  • 7.17.2 to 7.17.16 → no errors at all for theses versions

Apparently, the 7.16.2 fixed the error from 7.12.1 to 7.16.1.
It seem to be this issue below to fixe the problem but I’m not sure.

image
image955×170 17.3 KB

The 7.17.2 version fixed the error of the 7.16.2 to 7.17.1

image
image1115×256 35.8 KB

4

So I’m wondering what is the correct version of Filebeat to use?

Opensearch says to use Beats version 7.12.1 for compatibility.
On the other hand, there is the recent vulnerability discovered by a researcher on all Beats agents older than version 7.17.16 or 8.11.
Also there is the problem in Linux when a client uses docker engine.

By using obsolete versions of agents, this will make Graylog unusable, or at least ingestion with the Filebeat agent a big headhache.

For now I will use the latest version (7.17.16) of Filebeat for Linux.
Even if Opensearch says it is not officialy supported.

5

Hey @s0p4L1N

This is probably not excepted but I have used the latest version of Filebeat.
I havent noticed any issue.

6

Yes, it is what I do and no issue either with the 7.17.16, but when Opensearch says that we must use 7.12.1 which is more than 2 years old, it wrong the people that tried to follow first the documentation… as it does not work in some cases.

Anyways with the latest, i’m good :slight_smile:

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.