Pipeline Rule doesn't see extracted fields but only a few $message properties like $message.message and $message.level


(Bronius Motekaitis) #1

Thanks to debug(), I can say with more clarity/certainty that I am not seeing my JSON parsed fields in the $message object. I can see things like the raw message with $message.message and $message.level. My custom JSON parsed message fields do, however, show in the Graylog search main page:


See “time, facility, duration, message,” fields etc.

Maybe the reason why is because I have Apache custom logging to syslog as JSON with rsyslogd forwarding these messages to :514 to Graylog. I guess with this workflow, the extractor is acting only on toplevel after all (hence why I can see $message.level, provided by syslog but not $message.duration, provided by Apache/application piping into syslog). I can debug($message.message); and see my raw apache json string. Could this be my issue? How do I get to the json-parsed lower level message fields like $message.time, $message.duration?

With downloaded vm:
Graylog 2.3.1+9f2c6ef on graylog (Oracle Corporation 1.8.0_144 on Linux 4.4.0-93-generic)

Ultimately, my goal is just to overwrite the Graylog timestamp with $message.time as is seen in a couple Graylog Community posts.

Thanks
-Bronius

[edit]
Is this because my input is Syslog UDP (instead of, say, http json or any of a number of other ones)? Should I change my Input or layer extractors … or?
[/edit]

[edit]
Two more clues!
message into BLANK
And also, when I click Details on each JSON extractor, its stats indicate it has never been run… This is out-of-the-box Graylog from a downloaded VM with a new Syslog UDP input. I have not defined any “streams.”
[/edit]


(Jochen) #2

What’s the order of message processors in your Graylog cluster?

Check it on the System / Configurations page.


Searching imported logs by log timestamp, not time Graylog received the log
(Bronius Motekaitis) #3

Ah HA – that was it! Was GeoIP > Pipeline Processor > Message Filter Chain, but on learning of this feature just now, I rearranged it to:
Graylog - Configurations
and now my Timestamp rule applies, and I can see $message.whatever in debug() etc. #awesome

Thanks very much :slight_smile: :slight_smile: Where do I send flowers??


(Jochen) #4

Just keep posting interesting topics and help to improve Graylog. :wink:


(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.