Exclude search results by regular expression

I am trying to exclude JSON logs from the results.
My attempts so far:

NOT /^{"/
NOT message:/^{"/
NOT message:{"*

My best guess is that it needs to be escaped differently, but how?

Regular expression where, and what field is the json in, the message field?

The Search:
The text field next to the green magnifying glass button in a Dashboard widget definition.

The field:
“message”

Ah, okay thought so. You cannot use regex on the message field, the message field is a full-text searchable field (and you cannot change the data type of that special field). What that means really simply is that it isn’t stored as a string of text, each word is broken apart, some punctuation is ignored etc. (so that you can do things like fuzzy search, find words that aren’t next to each other, all the normal things you would expect from a search engine.

You can run regex of the other text field type which is “aggregatable”, that stores text as a simple string. So there would be two things, either identify if its json in a pipeline rule and write a field called is_json=yes and filter on that, or clone the message field to a second field (again in pipelines) and set that second field as aggregatable text, but that would double the storage size.

Okay, this is confusing, why does this:
AND message:/[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/

work then?

I also can’t get string comparisons that contain { and " for example to work. Which again I assume to be escape sequence problems.

Have you tried:

NOT message:/.*{".*/

Warnings:

  • The message field is treated differently than other fields because it’s a text type in Elastic/Opensearch and not a keyword
  • This kind of search is resources consuming
2 Likes

I did not exactly add the backslashes for fun. Without them I get:
“Query parsing error: Cannot parse query, cause: integer expected at position 3.”

I just posted that variant for completeness sake.
But I tried your suggestion, with and without. It does not change the result. (Well besides the not escaped sequence also not working of course with the same error.)

Also that suggestion is not equivalent to my understanding. as .* would match anything and any length. ^ is the start of sequence. So explicitly not “anything”.

Search regex are different from Stream/EventDef regex.
With Stream/EventDef you don’t need to match the whole pattern, for example

field user match regex toto

It will match foototo, toto, totobar, foototobar…
But this regex in Search will only match toto.
If you want to match foototo in Search you need the regex
.*toto
and for totobar
toto.*
So with you example indeed you only need:

NOT message:/\{".*/

But as I said message is different from other fields and it may not work.
Maybe it can work without a regex:

NOT message:\{\"*