Elastic warnings in graylog server.log

[tmacgbay]

Recently upgraded to Graylog 4.0 (then to 4.0.1) and upgraded Elastic to 7.10. I have Kibana 7.10 running on a separate machine and while it is pointed to elastic… I am not really doing anything in there, figured I would set it up and see what data I could read out of elastic and/or get a better GUI understanding about the indices. I have a strong bet my problem is related to the kibana connection but I have not found a way to resolve it.

With this set up I am seeing repeated entries in my graylog server.log for each index I have in Graylog here is an example:

2020-12-01T14:34:32.749-05:00 WARN [RestClient] request [GET http://ELASTIC_SVR:9200/trop_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-7.10.0-51e9d6f22758d0374a0f3f5c6e8f3a7997850f96 "this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .security, .kibana], but in a future major version, directaccess to system indices and their aliases will not be allowed"]

I have spent a day + trying to track down turning this off on either side (short of uninstalling kibana and trying to figure out which kibana only indices to remove) The thing is, if I run the query manually I don’t get any system indices that they warn about [.kibana_task_manager, .security, .kibana]

Command:
curl -X GET --netrc "http://ELASTIC_SVR:9200/trop_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false&pretty"

Result:
{
“trop_43” : {
“aliases” : { }
},
“trop_55” : {
“aliases” : {
“trop_deflector” : { }
}
},
“trop_40” : {
“aliases” : { }
},
“trop_50” : {
“aliases” : { }
},
“trop_49” : {
“aliases” : { }
},
“trop_42” : {
“aliases” : { }
},
“trop_47” : {
“aliases” : { }
},
“trop_45” : {
“aliases” : { }
},
“trop_41” : {
“aliases” : { }
},
“trop_53” : {
“aliases” : { }
},
“trop_44” : {
“aliases” : { }
},
“trop_48” : {
“aliases” : { }
},
“trop_52” : {
“aliases” : { }
},
“trop_46” : {
“aliases” : { }
},
“trop_51” : {
“aliases” : { }
},
“trop_54” : {
“aliases” : { }
}
}

I am hoping that it is something more esoteric than just a server.conf setting that @aaronsachs kindly pointed out that I should have read… If I should not have Kibana connected to the elastic DB, I am fine with shutting all that down (not time to play there as of late anyway) but if you suggest that could you also suggest which kibana only indices I need to remove from elastic too?

  Thanks!

[tmacgbay]

A little hunting around and I have found a solution to turning off the warnings on the elasticsearch side. Here is the detail in case anyone is going through something similar and is searching.

1 - Made the curl command more verbose with -v (of course) and received back:

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.10.10.110:9200...
* TCP_NODELAY set
* Connected to ELASTIC_SVR (10.10.10.110) port 9200 (#0)
* Server auth using Basic with user 'graylogcalling'
> GET /trop_*/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false&pretty HTTP/1.1
> Host: ELASTIC_SVR:9200
> Authorization: Basic ZWxhc3RpYzpKYWNNNNrJDBuNQ==
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 692
<

2 - It clicked — examine Elastic http settings… (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html)

3 - This lead to putting the following line in the /etc/elasticsearch/elasticsearch.yml config:
http.max_warning_header_count: 0

I am never wild about turning off warnings but blathering warnings get my attention.

yay me!