Messages are not being written on index after I modified some extractors

[veninmiha (lau)]

Hi Guys,
I have changed/deleted some extractors and now some of messages are not being parsed and written in indexes even if I add back the extractor, the messages are still being dropped.
Is there any way to help?
Thanks
Laurentiu

[tmacgbay]

Not without more information about:

• what is your environment (OS, graylog/elastic version…etc)
• extractors
• sample messages
• before-after extractors explanation
• expected extraction outcome
• …

[veninmiha (lau)]

HI.
Centos 7, Graylog 4.2.5, elasticsearch-7.16.2.
<86>Jan 10 13:00:32 sshd[31141]: reverse mapping checking getaddrinfo for Hostname [192.168.199.25] failed - POSSIBLE BREAK-IN ATTEMPT!

And if add an extractor to see the source ip (192.168.199.25) the message is not parsed and is not being added to the stream.
The extractor is %{IP:sourceip}, and once is connected to the stream the message is dropped.
Thanks
Laurentiu

[tmacgbay]

Are you able to get it work as desired on creation via “Try against example”? Can I assume that it is a GROK extractor you are using? Are there other extractors against the same message that may be set to cut out what they find? Can you post a screen shot of all of the extractor settings? Is the message actually being dropped or is it just not going to the stream you want it to go to?

An important side note, Graylog only supports up to Elasticsearch 7.10. Since you are at 7.16 you may run into issues that can not be fixed. I don’t think that is related to this posted issue but it’s important that you know.

[veninmiha (lau)]

HI. Thanks for your reply.
Yes, the extractor is Grok, and “try against example” it shows exactly what I’m looking for, the source IP, buts once it’s linked to the stream, everything what has source IP it is not indexed.
Other extractor related to source address is in the below shot screen. But this is for firewall logs and it has “src” field in the message, the above one had not any kind of field related to source, only IP (between the square brackets) and that’s it. But %{IP:sourceip} “Try against example” works only in the “sample”, in production I dont see a new field called “sourceip” and the corresponding source IP.

Thanks
Laurentiu

[tmacgbay]

I still do not see the full picture… It may be that you need to include the square brackets in the search pattern… and you may need to escape them

[%{IP:sourceip}]
or
\[%{IP:sourceip}\]

Or there may be some other extractor that I can’t see that is playing a role in the message in a way that makes the GROK for sourceip not happen.

To be clear, you are saying the message is dropped and not indexed anywhere or just that it doesn’t go to the index you want? If it is being dropped, where is that happening? You would have to actively drop the message…

[veninmiha (lau)]

OK. Let me make it simple. Ho do i extract the IP from the below message.
<86>Jan 10 17:08:09 hostname sshd[23243]: Failed password for invalid user USER from 192.168.224.1 port 56880 ssh2
If I add %{IP:srcip} it will work but will overlaps with other extractor scr=%{IP:blabla}
thanks
Laurentiu

[tmacgbay]

got distracted and didn’t use the </> tool… rookie error! I edited my previous post for clarity.

If there are always square brackets, then further refine by capturing those around the GROK pattern. Make sure to escape - \[

[gsmith]

Hello

I believe this is a Linux log, so you need this. I have this already setup in production.

Results.

hope that helps

EDIT: I forgot to mention I created my own GROK pattern and as you noticed above just the %{IP} default created two fields. To limit to one field here is my custom GROK

Custom Grok

(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])