Field not analyzing

[MalinkinSA (Sergey Malinkin)]

Hello,
We have a graylog 4.0.7 and for parsing we use Grok patterns extractor. But all field what was created by this grok - not analyzed. And we can’t search by the custom field without * or fulltext searching.
How we can change it? I tried change index template, but after index rotation all my changes lost.

[MalinkinSA (Sergey Malinkin)]

Hm, i think i found solution. I must add type: “text” and analyzer: “standard” for the interesting me field in this custom template and merge it? And this field have a maping for all index where they found?

[gsmith]

Hello,

@MalinkinSA

And we can’t search by the custom field without * or fulltext searching.
How we can change it?

Could you show your configurations and to be honest, I have no clue why this is happing to you.

@MalinkinSA

But all field what was created by this grok - not analyzed.

Could you explain in greater detail about this?

@MalinkinSA

I tried change index template, but after index rotation all my changes lost.

There might be some information in here about indices.

Upgrading to Elasticsearch 7.10 / problem with custom index template

@H2Cyber @zoulja The Graylog documentation prescribes this as a template: { "template": "graylog_*", "mappings" : { "message" : { "properties" : { "http_method" : { "type" : "keyword" }, "http_response_code" : { "type" : "long" }, "ingest_time" : { "type" : "date", "format": "strict_date_time" }, "took_ms" : { "type" : "long" } } } } } With ES 6.8 this wo…

[MalinkinSA (Sergey Malinkin)]

Hello, @gsmith
I find solution in custom template mapping, but one monet baffles me. If a want add a few mapping in different indexes i must create this filed in all of this indexes?

@jan
It’s correct that i can’t describe few different templates in graylog-custom-mapping-7x.json?

[gsmith]

Instead of GROK can you use regex instead?

[MalinkinSA (Sergey Malinkin)]

I try both, nothing changes, default type - keyword

[gsmith]

Can you show what configuration you have made or wanted to make such as Templates and your Grok patterns extractor. In my lab I have created a Grok patterns extractor then rotated my indices and my Grok patterns extractor still created fields I needed. Im not see the same issue as you are. Not knowning how you implemented this it hard to troubleshoot.

[MalinkinSA (Sergey Malinkin)]

No, i speak about another issue.
The field is not lost after rotation. For example i use Grok %{NUMBER:src_port} for extract src_port from event. In elastic, this field was created with type - keyword. And after rotation this field was with keyword type.
But in my case me need switch type of field from keyword to integer, and this i can make only with custom template. But in my cluster i have a few indexes where this field exist and indexes where not exist. And therefore, through custom mapping, when I have to do this field for all my indices

[gsmith]

Ok so you modified your default INDEX template and when it rotates you lose that configured mapping keyword --> integer.

I have been working with something like this and the following links helped out a lot.

Create or update index template API | Elasticsearch Guide [7.14] | Elastic

Then I had to read this to fine turn my knowledge and understand of Elasticsearch.

Custom index mappings

You might have missed a step.
Hope this helps

[MalinkinSA (Sergey Malinkin)]

Yep, but all fields must describe in graylog-custom-mapping with specified index_pattern.
It’s work, but in my case i must use template as * and all filed created in all new indexes

But it’s better than nothing