Finding ip addresses in message fields

Graylog Central (peer support)
1

Hello,

Many moons ago, a friend of mine build me a GrayLog 4.2.13 / elasticsearch 7.17.12 Linux server. I have no clue how the innards work. I just use the machine to search for stuff in syslog messages.
I let my Network switches, Linux Servers and my PfSense Firewall send messages to it. Works well.
I can do searches like showing all blocked traffic by doing:

source:myfirewall.mydomain.local AND message:block

and that shows me all the traffic that got blocked. Messages that PfSense sends, look like this “349,1593001719,lagg0.1,match,block,in,4,0x0,127,63994,0,DF,6,tcp,52,192.168.10.7,173.222.108.210,10163,80,0,S,884491034,64240,mss;nop;wscale;nop;nop;sackOK”

I also get messages like “sshd[28703]: Accepted password for root from 172.16.1.250 port 58405 ssh2” from Linux servers for example.

Being used to other syslog products at work, I thought I could simply look for a specific IP address (regardless if source-ip or dest-ip) by doing:
source:myfirewall.mydomain.local AND message:192.168.10.7
message:192.168.10.7
message:“192.168.10.7”
or even just:
192.168.10.7 or “192.168.10.7”
but then there is no output at all. It finds nothing.

Questions about finding IP addresses inside messages (not indexed, not in fields) have been asked many times before, so I tried things like:
source:myfirewall.mydomain.local AND message:192.168.10.7 with backslashes before each dot (not visible here so i’ll just write it verbally)
and many, many other suggestions (mostly involving regex, escaping characters)
but so far, nothing worked.

I have the impression that the problem lies with the dots in ip addresses because I can find anything else in message fields (they are just plain text), as long as they don’t have dots in them.
I’m lost. How do I search for ip addresses in the “message” field?

2

You are correct that dots aren’t dots. ElasticSearch (and opensearch) are search indexes, they are not SQL databases and are architected very differently. This means that due to the way they function they don’t actually “see” the punctuation in many cases.

This means that the way to solve this kind of thing is in pipelines. Use grok regex etc to find the values and extract them into their own field. Also if it’s an IP address you probably want to set the field type to IP as it will otherwise be stored as text. You can set this now easily in 5.2

3

“Use grok regex etc”

My eyes just glazed over. That is over my head. Is there a “graylog and grok for dummies” somewhere? Googling it only gives me guides that assumes that the reader already has a certain knowledge level. I am not a developer.
The most advanced thing I ever did in Graylog is create a DNS lookup-table thingy so that hostnames of switches etc. appear as such instead of the “source” only being recorded as an ip address. I found a step-by-step how-to which was well-written. This pipeline and grok stuff sounds much more complex though from what i’ve found.

I played around with the idea of somehow extracting the two IP’s in firewall messages, the source and destination IP’s, and creating fields out of them.
So in a message like:
“349,1593001719,lagg0.1,match,block,in,4,0x0,127,63994,0,DF,6,tcp,52,192.168.10.7,173.222.108.210,10163,80,0,S,884491034,64240,mss;nop;wscale;nop;nop;sackOK”
the 192.168.10.7,173.222.108.210 are the source and dest. respectively. I found some info about creating fields out of them so one can make a query like: srcip:192.168.10.7
I would also need a field named something like “ipall” which has both IP-addresses in it because sometimes, you just look for an IP, not yet knowing if it’s the source or dest. IP
Having such fiels (which I guess are indexable) would be better than not be able to search for ip-adresses in plain text at all (my current situation).
I have not found a how-to that starts at the beginning on how to do that so far though.

I tried upgrading my GrayLog to 5.x a while back. Totally ruined my installation (graylog was no longer sending data to elasticssearch, dunno why). Was not able to solve it so I restored from backup and am still running 4.2 as a consequence.

4

This post actually covers it fairly well https://graylog.org/post/graylog-parsing-rules-and-ai-oh-my/

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.