Specific field not indexed correctly and search not working

Graylog Central (peer support)
1

Hi all,

Since we migrated from Graylog 2 to Graylog 3, search is no longer working for us.

We used to create search strings based on custom fields that we send through gelf to Graylog, after the migration, we still get the messages and these are saved correctly in Elasticsearch.

Search seems to work for message , however our custom field params doesn’t.

How can we ensure that this custom field is indexed correctly to get search back to work?

Best regards.

2
  1. Check the server logs first, to make sure there is no any obvious error.
  2. Check and share extractor/pipeline configuration
3

Hi,

We don’t see any relevant errors in the logs, this is discarded because I can see the message in Graylog, but if I try to search for any string within the params field, I don’t get any results.

We don’t use extractors, it’s exactly the same configuration we had in Graylog 2. We’ve tried to use them, but it adds a huge overhead to CPU, causing excessive load in the node, we ingest around 3000 msg/s, this wasn’t a problem when using the old version though.

Thanks

4

Maybe relevant to you:

Also note that message , full_message , and source are the only fields that are being analyzed by default. While wildcard searches (using * and ? ) work on all indexed fields, analyzed fields will behave a little bit different. See wildcard and regexp queries for details.

1 Like
5

Is it possible to analyze other fields?

Wildcard searches add a huge overhead when it comes to CPU load on elasticsearch cluster, and they’re not recommended because of this fact.

6

I’ve tried to update the template mappings for one of our indexes, however, when this is rotated, the fields I added to the template are gone, what is overwritting this and where is that configuration?

{
"order": -1,
"index_patterns": [
    "events_*"
],
"settings": {
    "index": {
    "analysis": {
        "analyzer": {
        "analyzer_keyword": {
            "filter": "lowercase",
            "tokenizer": "keyword"
        }
        }
    }
    }
},
"mappings": {
    "message": {
    "_source": {
        "enabled": true
    },
    "dynamic_templates": [
        {
        "internal_fields": {
            "mapping": {
            "type": "keyword"
            },
            "match_mapping_type": "string",
            "match": "gl2_*"
        }
        },
        {
        "store_generic": {
            "mapping": {
            "type": "keyword"
            },
            "match_mapping_type": "string"
        }
        }
    ],
    "properties": {
        "gl2_processing_timestamp": {
            "format": "yyyy-MM-dd HH:mm:ss.SSS",
            "type": "date"
        },
        "gl2_accounted_message_size": {
            "type": "long"
        },
        "gl2_receive_timestamp": {
            "format": "yyyy-MM-dd HH:mm:ss.SSS",
            "type": "date"
        },
        "full_message": {
            "fielddata": false,
            "analyzer": "standard",
            "type": "text"
        },
        "streams": {
            "type": "keyword"
        },
        "source": {
            "fielddata": true,
            "analyzer": "analyzer_keyword",
            "type": "text"
        },
        "message": {
            "fielddata": false,
            "analyzer": "standard",
            "type": "text"
        },
        "meta": {
            "fielddata": false,
            "analyzer": "standard",
            "type": "text"
        },
        "params": {
            "fielddata": false,
            "analyzer": "standard",
            "type": "text"
        },
        "timestamp": {
        "format": "yyyy-MM-dd HH:mm:ss.SSS",
        "type": "date"
        }
    }
    }
},
"aliases": {}
}

How can we persist these changes?

7

After digging a bit in the documentation, I found how to add custom mappings (https://docs.graylog.org/en/3.2/pages/configuration/elasticsearch.html#custom-index-mappings), I applied this to the indexes used and now the fields are searchable as it was in Graylog 2.

Thanks for your help guys and see you around!

8

Please post small example for other user, how did you resolve your problem…

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.