Hi,
I have to save some long log messages (from an iPad synchronisation process) in my Graylog server.
The limit appears to be defined to 32 kb.
Is it possible to increase easily this value (to 64 kb) ? from /etc/graylog/server/server.conf ?
I have to keep the message as a searchable field.
I’m using Graylog 2.1.2+50e449a (Oracle Corporation 1.8.0_111 on Linux 3.14.32-xxxx-grs-ipv6-64)
Thx
No, that’s not possible.
If I set doc_values to false, the message field keeps to be searchable ?
Kind of, but that doesn’t help with the 32kb field limit of Lucene.
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/doc-values.html
I save the log in full_message.
In mapping:
"full_message" : {
"analyzer" : "standard",
"index" : "analyzed",
"type" : "string"
},
Can I set “index”: “no” without burn all the configuration ?
You can create a custom index template which overrides the default index template of Graylog via the “order” attribute.
See http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings for details.
graylog-custom-mapping.json:
{
"graylog-internal" : {
"order" : 0,
"template" : "graylog_*",
"settings" : {
"index" : {
"analysis" : {
"analyzer" : {
"analyzer_keyword" : {
"filter" : "lowercase",
"tokenizer" : "keyword"
}
}
}
}
},
"mappings" : {
"message" : {
"_source" : {
"enabled" : true
},
"dynamic_templates" : [ {
"internal_fields" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string"
},
"match" : "gl2_*"
}
}, {
"store_generic" : {
"mapping" : {
"index" : "not_analyzed"
},
"match" : "*"
}
} ],
"properties" : {
"full_message" : {
"analyzer" : "standard",
"index" : "no",
"type" : "string"
},
"streams" : {
"index" : "not_analyzed",
"type" : "string"
},
"source" : {
"analyzer" : "analyzer_keyword",
"index" : "analyzed",
"type" : "string"
},
"message" : {
"analyzer" : "standard",
"index" : "analyzed",
"type" : "string"
},
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss.SSS",
"type" : "date"
}
}
}
},
"aliases" : { }
}
}
And I load the index mapping into Elasticsearch:
$ curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
Is the good practice ?
Thx
You’re not supposed to overwrite the graylog-internal template. Choose a different name.
You also don’t need to redefine all of the existing fields, just the ones you want to modify.
{
"graylog-internal-unindex-fullmessage" : {
"order" : 0,
"template" : "graylog_*",
"mappings" : {
"message" : {
"properties" : {
"full_message" : {
"analyzer" : "standard",
"index" : "no",
"type" : "string"
}
}
}
}
}
}I have the following error with the previous json
{
"error" : {
"root_cause" : [ {
"type" : "action_request_validation_exception",
"reason" : "Validation Failed: 1: template is missing;"
} ],
"type" : "action_request_validation_exception",
"reason" : "Validation Failed: 1: template is missing;"
},
"status" : 400
}
template value is defined
Please post the complete command, the complete template, and the complete output.
graylog-custom-mapping.json
{
"graylog-internal-unindex-fullmessage" : {
"order" : 0,
"template" : "graylog_*",
"mappings" : {
"message" : {
"properties" : {
"full_message" : {
"analyzer" : "standard",
"index" : "no",
"type" : "string"
}
}
}
}
}
}
command:
curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
output:
{
"error" : {
"root_cause" : [ {
"type" : "action_request_validation_exception",
"reason" : "Validation Failed: 1: template is missing;"
} ],
"type" : "action_request_validation_exception",
"reason" : "Validation Failed: 1: template is missing;"
},
"status" : 400
}Please refer to http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#creating-a-new-index-template or the Elasticsearch documentation for a description of the HTTP request format for creating an index template…
curl -X GET 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
I have this:
{
"graylog-custom-mapping" : {
"order" : 0,
"template" : "graylog_*",
"settings" : { },
"mappings" : {
"message" : {
"properties" : {
"full_message" : {
"analyzer" : "standard",
"index" : "no",
"type" : "string",
"doc_values" : false
}
}
}
},
"aliases" : { }
}
}
but, in Graylog, the full messages are still truncated.
Is elasticSearch needs to be restarted ?
No, but you need to create a new index so that the new mapping is being applied.
full_message is still truncated even with the new index. How can I be sure that the index: “no” is took into consideration ?
If I search a value full_message: "2018" it returns messages.
Thx
Check the mapping of the index the message was stored in.
graylog-custom-mapping.json:
{
"template": "graylog_*",
"mappings": {
"message": {
"properties": {
"full_message": {
"index": "no",
"doc_values": false,
"type": "string"
}
}
}
}
}
curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
{
"acknowledged" : true
}
Create a new Graylog index (graylog_5).
curl -X GET 'http://localhost:9200/graylog_5/_mapping/message'
...
"full_message": {
"type": "string",
"analyzer": "standard"
},
...
What’s wrong ?
Thx
What templates are active in your Elasticsearch cluster?
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-templates.html#getting
{
"graylog-internal": {
"order": -2147483648,
"template": "graylog_*",
"settings": {
"index": {
"analysis": {
"analyzer": {
"analyzer_keyword": {
"filter": "lowercase",
"tokenizer": "keyword"
}
}
}
}
},
"mappings": {
"message": {
"_source": {
"enabled": true
},
"dynamic_templates": [
{
"internal_fields": {
"mapping": {
"index": "not_analyzed",
"type": "string"
},
"match": "gl2_*"
}
},
{
"store_generic": {
"mapping": {
"index": "not_analyzed"
},
"match": "*"
}
}
],
"properties": {
"full_message": {
"analyzer": "standard",
"index": "analyzed",
"type": "string"
},
"streams": {
"index": "not_analyzed",
"type": "string"
},
"source": {
"analyzer": "analyzer_keyword",
"index": "analyzed",
"type": "string"
},
"message": {
"analyzer": "standard",
"index": "analyzed",
"type": "string"
},
"timestamp": {
"format": "yyyy-MM-dd HH:mm:ss.SSS",
"type": "date"
}
}
}
},
"aliases": {}
},
"graylog-custom-mapping": {
"order": 0,
"template": "graylog_*",
"settings": {},
"mappings": {
"message": {
"properties": {
"full_message": {
"index": "no",
"type": "string",
"doc_values": false
}
}
}
},
"aliases": {}
}
}