Graylog v. 5.1.5
Has anyone gotten journelbeat to properly ingest Journald via sidecar? If so, do you mind sharing your sidecar configuration as well as any configuration you had to do client side and server side (excepting the obvious input settings in “System”)?.
Thank you kindly!
EDIT: I am aware of Ingest_journald but I am not bright enough to make sense of that document 
Hey @accidentaladmin
What are you settings for Journalbeat? if you using the sidecar can you show thos settings too?
Hey @gsmith, this is my sidecar setting:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["192.168.128.117:5044"]
path:
data: /var/lib/graylog-sidecar/collectors/journalbeat/data
logs: /var/lib/graylog-sidecar/collectors/journalbeat/log
journalbeat.input:
- paths:
- []
seek: cursor
seccomp:
default_action: allow
syscalls:
- action: allow
names:
- rseq
But then I get:
journalbeat a few seconds ago Failing Collector configuration file is not valid, waiting for the next update.
Exiting: error loading config file: yaml: line 14: did not find expected key
I assume line 14 is 14 within the sidecar config which is, above:
13: seek: cursor
14:
15: seccomp:
thank you!
Hey @accidentaladmin
I did something really simple for testing awhile back.
A the bottom of GL Sidecar configuration file i uncommented these lines
Graylog Sidecar
collector_binaries_whitelist:
#collector_binaries_accesslist:
# - "/usr/bin/filebeat"
# - "/usr/share/filebeat/bin/filebeat"
# - "/usr/bin/packetbeat"
# - "/usr/bin/metricbeat"
# - "/usr/bin/heartbeat"
# - "/usr/bin/auditbeat"
- "/usr/bin/journalbeat"
- "/usr/share/journalbeat/bin/journalbeat"
journalbeat
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["x.x.x.x:5044"]
path:
data: /var/lib/graylog-sidecar/collectors/journalbeat/data
logs: /var/lib/graylog-sidecar/collectors/journalbeat/log
journalbeat.inputs:
- paths: []
seek: head
EDIT I forgot to add this
Ooof! Instituted your suggestions and have a new sidecar error haha
Unable to start collector after 3 tries, giving up!
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f550e4a3d3c m=15 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: unknown pc 0x7f550e4a3d3c
stack: frame={sp:0x7f54afffe900, fp:0x0} stack=[0x7f54af7ff268,0x7f54afffee68)
00007f54afffe800: 0000000000000000 0000000000000000
00007f54afffe810: 0000000000000000 0000000000000000
00007f54afffe820: 0000000000000000 0000000000000000
00007f54afffe830: 0000000000000000 0000000000000002
00007f54afffe840: 000000c00008dfd8 0000000000000000
00007f54afffe850: 0000000000000000 0000000000000000
00007f54afffe860: 0000000000000000 0000000000000000
00007f54afffe870: 0000000000000000 0000000000000000
00007f54afffe880: 0000000000000000 00007f550e52264d
00007f54afffe890: 00007f54adffb6c0 00007f54afffeb10
00007f54afffe8a0: 00007f54afffe9c6 0000000000000000
00007f54afffe8b0: 00007f54afffe9c7 00007f550e4a1c80
00007f54afffe8c0: 00007f54a4000cb0 0000000000020350
00007f54afffe8d0: 00000000003d0f00 00007f550e511174
00007f54afffe8e0: 00007f54adffb990 000000000000000d
00007f54afffe8f0: 0000556ba186c3d6 00007f550e4a3d2e
00007f54afffe900: <00000000007fff80 f16d0998b8ca2d00
00007f54afffe910: 0000000000000006 00007f54affff6c0
00007f54afffe920: 00007f54a4000ca0 0000000000000000
00007f54afffe930: 0000556ba17caf12 00007f550e454f32
00007f54afffe940: 00007f550e5ece70 00007f550e43f472
00007f54afffe950: 0000000000000020 00007f550e5ec703
00007f54afffe960: 0000000000000d68 00007f550e499290
00007f54afffe970: 00007f550e5e85e0 0000000000000001
00007f54afffe980: 000000000000000a 00007f54affff6c0
00007f54afffe990: 0000000000000000 0000556ba17caf12
00007f54afffe9a0: 0000000000000000 00007f550e49aee9
00007f54afffe9b0: 00007f550e5ec680 00007f550e49b2f3
00007f54afffe9c0: 00007f550e5ec680 000000000000000a
00007f54afffe9d0: 00007f54affff6c0 00007f550e49687a
00007f54afffe9e0: 00007f550e5ec840 f16d0998b8ca2d00
00007f54afffe9f0: 00007f550e5ec840 00007f550e5ec840
runtime: unknown pc 0x7f550e4a3d3c
stack: frame={sp:0x7f54afffe900, fp:0x0} stack=[0x7f54af7ff268,0x7f54afffee68)
00007f54afffe800: 0000000000000000 0000000000000000
00007f54afffe810: 0000000000000000 0000000000000000
00007f54afffe820: 0000000000000000 0000000000000000
00007f54afffe830: 0000000000000000 0000000000000002
00007f54afffe840: 000000c00008dfd8 0000000000000000
00007f54afffe850: 0000000000000000 0000000000000000
00007f54afffe860: 0000000000000000 0000000000000000
00007f54afffe870: 0000000000000000 0000000000000000
00007f54afffe880: 0000000000000000 00007f550e52264d
00007f54afffe890: 00007f54adffb6c0 00007f54afffeb10
00007f54afffe8a0: 00007f54afffe9c6 0000000000000000
00007f54afffe8b0: 00007f54afffe9c7 00007f550e4a1c80
00007f54afffe8c0: 00007f54a4000cb0 0000000000020350
00007f54afffe8d0: 00000000003d0f00 00007f550e511174
00007f54afffe8e0: 00007f54adffb990 000000000000000d
00007f54afffe8f0: 0000556ba186c3d6 00007f550e4a3d2e
00007f54afffe900: <00000000007fff80 f16d0998b8ca2d00
00007f54afffe910: 0000000000000006 00007f54affff6c0
00007f54afffe920: 00007f54a4000ca0 0000000000000000
00007f54afffe930: 0000556ba17caf12 00007f550e454f32
00007f54afffe940: 00007f550e5ece70 00007f550e43f472
00007f54afffe950: 0000000000000020 00007f550e5ec703
00007f54afffe960: 0000000000000d68 00007f550e499290
00007f54afffe970: 00007f550e5e85e0 0000000000000001
00007f54afffe980: 000000000000000a 00007f54affff6c0
00007f54afffe990: 0000000000000000 0000556ba17caf12
00007f54afffe9a0: 0000000000000000 00007f550e49aee9
00007f54afffe9b0: 00007f550e5ec680 00007f550e49b2f3
00007f54afffe9c0: 00007f550e5ec680 000000000000000a
00007f54afffe9d0: 00007f54affff6c0 00007f550e49687a
00007f54afffe9e0: 00007f550e5ec840 f16d0998b8ca2d00
00007f54afffe9f0: 00007f550e5ec840 00007f550e5ec840
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc00073f7e8)
/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc00073f7e0)
/usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).Stop(0xc00073f7c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:138 +0x45
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc0009b8a80, 0x556ba13ac58e, 0xb, 0x556ba13ac58e, 0xb, 0x0, 0x0, 0x100, 0x0, 0x0, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:479 +0x830
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0x556ba13ac58e, 0xb, 0x556ba13ac58e, 0xb, 0x0, 0x0, 0x101, 0xc00093fc30, 0x556ba2169ad0, 0x0, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:196 +0x5a8
github.com/elastic/beats/v7/libbeat/cmd/instance.Run(0x556ba13ac58e, 0xb, 0x556ba13ac58e, 0xb, 0x0, 0x0, 0x100, 0x0, 0x0, 0x0, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:197 +0x125
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc0006da000, 0xc000784820, 0x0, 0xa)
/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x85
github.com/spf13/cobra.(*Command).execute(0xc0006da000, 0xc00004e0c0, 0xa, 0xa, 0xc0006da000, 0xc00004e0c0)
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc0006da000, 0x0, 0xffffffff, 0xc0000c4058)
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x30b
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main()
/go/src/github.com/elastic/beats/x-pack/journalbeat/main.go:16 +0x31
goroutine 71 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x556ba3673ea0)
/go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:1131 +0x8d
created by k8s.io/klog/v2.init.0
/go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:416 +0xd8
goroutine 11 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:168 +0xa5
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x46
goroutine 82 [chan receive]:
github.com/elastic/go-lumber/client/v2.(*AsyncClient).ackLoop(0xc000ce3410)
/go/pkg/mod/github.com/elastic/go-lumber@v0.1.0/client/v2/async.go:149 +0xdc
created by github.com/elastic/go-lumber/client/v2.(*AsyncClient).startACK
/go/pkg/mod/github.com/elastic/go-lumber@v0.1.0/client/v2/async.go:123 +0x94
goroutine 83 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc00023e000)
/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x193
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func1(0xc0007615e0, 0x556ba21d59a8, 0xc00023e000)
/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:176 +0x62
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:174 +0x3c6
goroutine 84 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc00073f6d0)
/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0xfa
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2(0xc0007615e0, 0xc00073f6d0)
/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:180 +0x5b
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:178 +0x3f5
goroutine 85 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc0006ac000, 0x800, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:65 +0xd1
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc00080bb60, 0x556ba21d59d0, 0xc0006ac000)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:182 +0x1a2
github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1(0xc00080bb60, 0x556ba21d59d0, 0xc000739fe0)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:86 +0x6e
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:84 +0x153
goroutine 86 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*retryer).loop(0xc00080bf80)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:135 +0x237
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newRetryer
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:94 +0x15d
goroutine 87 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0007861e0)
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:127 +0xb1
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x196
goroutine 89 [runnable]:
syscall.Syscall(0x3, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5
syscall.Close(0x8, 0x5, 0xc0002b8c40)
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:285 +0x45
internal/poll.(*FD).destroy(0xc0006aa420, 0x7f54e451fe90, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:83 +0x45
internal/poll.(*FD).decref(0xc0006aa420, 0x1, 0x0)
/usr/local/go/src/internal/poll/fd_mutex.go:213 +0x5f
internal/poll.(*FD).Close(0xc0006aa420, 0xc0002b8d00, 0x556b9f503d27)
/usr/local/go/src/internal/poll/fd_unix.go:106 +0x51
os.(*file).close(0xc0006aa420, 0x0, 0xc000d3c1e0)
/usr/local/go/src/os/file_unix.go:251 +0x3a
os.(*File).Close(0xc000010098, 0xc0009ea400, 0xc000348238)
/usr/local/go/src/os/file_posix.go:25 +0x35
github.com/elastic/beats/v7/libbeat/metric/system/cgroup/cgcommon.GetPressure(0xc000d3a2c0, 0x40, 0xc0009ea4b0, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/metric/system/cgroup/cgcommon/metrics.go:87 +0x5ad
github.com/elastic/beats/v7/libbeat/metric/system/cgroup/cgv2.(*CPUSubsystem).Get(0xc000d30ea0, 0xc00023a400, 0x33, 0x17, 0xc000d3c018)
/go/src/github.com/elastic/beats/libbeat/metric/system/cgroup/cgv2/cpu.go:67 +0xbd
github.com/elastic/beats/v7/libbeat/metric/system/cgroup.getStatsV2(0xc000057233, 0x25, 0xc00023a400, 0x33, 0x1, 0xc00023a8b4, 0x3, 0xc0009e2600, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/metric/system/cgroup/reader.go:243 +0x227
github.com/elastic/beats/v7/libbeat/metric/system/cgroup.(*Reader).GetV2StatsForProcess(0xc0009e2580, 0xd37c2, 0x556ba1c2a880, 0xc0003481f0, 0x556b9fdd07aa)
/go/src/github.com/elastic/beats/libbeat/metric/system/cgroup/reader.go:219 +0x29c
github.com/elastic/beats/v7/libbeat/cmd/instance/metrics.reportMetricsCGV2(0xd37c2, 0xc0009e2580, 0x556ba2220de0, 0xc0009e20c0)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/metrics/metrics.go:378 +0x4f
github.com/elastic/beats/v7/libbeat/cmd/instance/metrics.reportBeatCgroups(0xc000259d01, 0x556ba2220de0, 0xc0009e20c0)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/metrics/metrics.go:311 +0x44f
github.com/elastic/beats/v7/libbeat/monitoring.(*Func).Visit(0xc00019f618, 0x556ba13a1e01, 0x556ba2220de0, 0xc0009e20c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/metrics.go:219 +0x48
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).doVisit(0xc000506680, 0xc000258701, 0x556ba2220de0, 0xc0009e20c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:83 +0x1c3
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Visit(0xc000506680, 0x556ba139f301, 0x556ba2220de0, 0xc0009e20c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:65 +0x4a
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).doVisit(0xc000506080, 0xc0009e2001, 0x556ba2220de0, 0xc0009e20c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:83 +0x1c3
github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).Visit(...)
/go/src/github.com/elastic/beats/libbeat/monitoring/registry.go:65
github.com/elastic/beats/v7/libbeat/monitoring.CollectFlatSnapshot(0xc000506080, 0xc000ce0001, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/monitoring/snapshot.go:63 +0x158
github.com/elastic/beats/v7/libbeat/monitoring/report/log.makeSnapshot(...)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:198
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop.func1(0xc00073f7c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:147 +0x145
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc00073f7c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:160 +0x6bb
github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1(0xc00073f7c0)
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:131 +0x56
created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter
/go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:129 +0x2f4
goroutine 12 [chan receive]:
github.com/elastic/beats/v7/libbeat/service.HandleSignals.func1(0xc000786fc0, 0xc000ce4b60, 0xc000ce4b50, 0xc0000535b0, 0xc0006ac0a0)
/go/src/github.com/elastic/beats/libbeat/service/service.go:50 +0x54
created by github.com/elastic/beats/v7/libbeat/service.HandleSignals
/go/src/github.com/elastic/beats/libbeat/service/service.go:49 +0x178
rax 0x0
rbx 0xd37d0
rcx 0x7f550e4a3d3c
rdx 0x6
rdi 0xd37c2
rsi 0xd37d0
rbp 0x7f54affff6c0
rsp 0x7f54afffe900
r8 0x0
r9 0x73
r10 0x8
r11 0x246
r12 0x6
r13 0x0
r14 0x556ba17caf12
r15 0x0
rip 0x7f550e4a3d3c
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
Hye @accidentaladmin
I just tested this out like 15 minute ago, I received the same Error and was going to post it.
EDIT: I think it might have something to do where the bin is located and GL-sidecar trying to access it.
This is about as far as i got.
This is my working collector config (Graylog 4.3, Sidecar 1.4.0, Journalbeat 7.x from Elastic Repo, RHEL):
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
journalbeat.inputs:
- paths:
- /run/log/journal
seek: cursor
output.logstash:
hosts: ["${user.BeatsInput}"]
slow_start: true
path:
data: /var/lib/graylog-sidecar/collectors/journalbeat/data
logs: /var/lib/graylog-sidecar/collectors/journalbeat/log
You also have to copy some audit rules and run augenrules --load after that.
src: /usr/share/audit/sample-rules/x.rules
dest: /etc/audit/rules.d/
Log Collector:
Path /usr/share/journalbeat/bin/journalbeat
Parameters -c %s
Appreciate the feedback; given how … vexatious? … this is; do you believe the shipping journald to Graylog outweighs those of shipping syslog on over to Graylog?
A while back I tested the journald input for filebeat. I’m pretty certain journalbeat has been deprecated and replaced with the filebeat input, see Journald input | Filebeat Reference [8.10] | Elastic
When I tested it did work as expected although it had some limitations, such as not supporting multiline which IMO makes it not that useful. If at all possible its recommended to log as close to the source as possible, meaning if the app can send the logs to log management that is highly preferrable.
Point taken … not passing the buck but maybe Graylog should take down the configuration page, then?
https://go2docs.graylog.org/5-0/getting_in_log_data/ingest_journald.html
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
