CEPH to Graylog

Graylog Central (peer support)
1

Hi,

Did anyone have working solution for CEPH logging to Graylog; I set all on CEPH side tcpdump shows packet reach interface, Graylog show that GELF (udp) is receiving but on the end no decoded logs.

Ceph 16.2.3, Graylog 4.1

2
  1. Check and post graylog log, if there is warning about gelf data processing.
  2. It there is no errors, maybe problem is with timestamps. Your logs may be stored with future timestamps, so not visible now, but after some time. Try to check this with Absolute time frame selector, select time frame from past to future (1 or 2 days to future is enough) to see if there are any logs.
    Time frame selector — Graylog 4.0.0 documentation
  3. Usually problem is with wrong timezone in graylog box or device sending logs.
  4. Or maybe there is a bug in ceph?
    [ceph-users] Missing field "host" in logs sent to Graylog
3

Look like malformed data going from CEPH, no “host” field …

2021-07-01 12:16:57,355 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=3ad5c6a1-da66-11eb-a55c-0242ac120005, messageQueueId=2810784, codec=gelf, payloadSize=340, timestamp=2021-07-01T12:16:57.354Z, remoteAddress=/xx.xx.xx.xx:34049}
java.lang.IllegalArgumentException: GELF message <3ad5c6a1-da66-11eb-a55c-0242ac120005> (received from <xx.xx.xx.xx:34049>) has empty mandatory "host" field.
	at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:247) ~[graylog.jar:?]
	at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
4

Bug was already issued in ceph, but no response:
https://tracker.ceph.com/issues/45457

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.