Graylog tries to create deflector aliases as indexes on restart


(Johannes Dagemark) #1

Hi

We are running
Graylog: 2.4.6+ceaa7e4
JVM: Oracle Corporation 1.8.0_181 on Linux 3.10.0-862.11.6.el7.x86_64
Elastic: 5.6.11-1

After each restart of elastics we get messages similar to this one in the graylog server.log

2018-12-03T13:41:11.195+01:00 WARN [Messages] Failed to index message: index=<del3month_deflector> id= error=<{“type”:“invalid_index_name_exception”,“reason”:“Invalid index name [del3month_deflector], already exists as alias”,“index_uuid”:“na”,“index”:“del3month_deflector”}>

It’s like graylog tries to create the deflector aliases as indexes but fails due to the fact that they already exists as aliases.

Have anyone seen this problem before?

Best regards
Johannes Dagemark


(Jan Doberstein) #2

I’m only aware of this: http://docs.graylog.org/en/2.4/pages/faq.html#how-do-i-fix-the-deflector-exists-as-an-index-and-is-not-an-alias-error-message


(Johannes Dagemark) #3

Thanks Jan, but it seems my problem is the other way around… :frowning:

I have the deflector aliases for all my indexes. Just can’t figure out why I get these error messages.

Best
Johannes


(Jan Doberstein) #4

Did you have cerebro installed to manage your elasticsearch?

With that it would be easy to solve that, remove all aliases and create the correct one.


(Johannes Dagemark) #5

Yes I do have cerebro installed :slight_smile:

Just to be sure before I start deleting, I have 3 indices with index prefix
default
del3month
del6month

In cerebro I can see that I have the following aliases and assignments

default_deflector assigned to index default_211
del3month_deflector assigned to index del3month_211
del6month_deflector assigned to index del6month_210

I’m not sure about the deflector alias naming convention but it sure sounds alright to me.

Would this be a good way forward perhaps?
1, stop graylog
2, remove the aliases
3, start graylog and hope that “correct” aliases are created by graylog.

If you have any idea on how to move forward I’m most grateful.

Best
Johannes


(Jan Doberstein) #6

That might work - but I would not make my life depending on that.


(Johannes Dagemark) #7

Did try

1, stop graylog
2, remove the aliases
3, start graylog and hope that “correct” aliases are created by graylog.

The aliases was recreated but the error messages persists. They appear every time I restart the elastic cluster. I don’t think that it’s related to the alias anymore because when the cluster comes back I get a “Bulk indexing finally successful” and the indexing failures stop.

2018-12-07T13:25:26.963+01:00 WARN [Messages] Failed to index message: index=<del3month_deflector> id=<24b6543c-fa1b-11e8-9597-98f2b326b090> error=<{“type”:“invalid_index_name_exceptio
n”,“reason”:“Invalid index name [del3month_deflector], already exists as alias”,“index_uuid”:“na”,“index”:“del3month_deflector”}>
2018-12-07T13:25:26.963+01:00 WARN [Messages] Failed to index message: index=<del6month_deflector> id=<24b6543f-fa1b-11e8-9597-98f2b326b090> error=<{“type”:“invalid_index_name_exceptio
n”,“reason”:“Invalid index name [del6month_deflector], already exists as alias”,“index_uuid”:“na”,“index”:“del6month_deflector”}>
2018-12-07T13:25:26.963+01:00 ERROR [Messages] Failed to index [2000] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in
the Bulk request failed, check BulkResult.getItems() for more information.
2018-12-07T13:25:28.424+01:00 INFO [Messages] Bulk indexing finally successful (attempt #14).
2018-12-07T13:25:28.782+01:00 INFO [Messages] Bulk indexing finally successful (attempt #15).

Could it be a case of an unclear log message? Perhaps “Failed to index message” should be more like “could not find index xxxx, perhaps elasticsearch is not operational, check your elasticsearch server log”.

Best regards
Johannes Dagemark


(Jan Doberstein) #8

Do you mind opening a bug report over https://github.com/Graylog2/graylog2-server/issues

thx


(Johannes Dagemark) #9

Sure, just posted it here https://github.com/Graylog2/graylog2-server/issues/5393

Thanks for the help so far.

Best
Johannes