I observe some unclear search issue
I receive proceed Nginx access logs via syslog, this log contains poorly formatted JSON.
Sample: hostname syslogtag: ip=1.2.3.4 localtime=8/Mar/2019:17:07:00 +1000 status=200 response_time=0.027 "POST /API/103/Ping HTTP/1.1" {\x22param1\x22:{\x22param2\x22:\x22value1\x22,\x22value2\x22:\x221\x22,\x22value3\x22:\x22hello\x22,\x22lang\x22:\x22es\x22},\x22node\x22:{\x22nodeId\x22:\x22Big\x22,\x22param4\x22:\x221456\x22},\x22param5\x22:{\x22Code\x22:\x221\x22,\x22net\x22:\x22333\x22,\x22Name\x22:\x22default\x22,\x22param6\x22:\x22-1\x22}} UserAgent="-"
By some reason I can’t find anything using query like: message:"param2"
(also tried message:param2)
I guess Graylog or ES is confused by JSON escape characters, but have no idea how to fix it
Any ideas are welcome
first you would like to split your log based on your key-values. If possible you should change your nxlogs to something that is easier to split …
the “JSON” is not JSON so it will not be recognized as JSON. But you could use the processing pipeline to split that all up and have nice fields you can work with.
If you want to search for param2 regular, you will do message normalization on your logs.
OMG, shame on me, I was always thinking Graylog can search by any substring!
And only now I realized it searches by tokens!
Thank you so much, @jan, now I can do many things better.
Actually my initial idea was “sanitizing” such log and replace \x22 to its actual representation “”
But as I see I can achieve that in 2 ways: pipeline(regex_replace() or replace(), not sure which one I should take) and extractor(“replace with regexp”).
Is there any performance difference between these 2 options?
