I have a 3 node cluster , all 3 running both graylog and elastrcsearch.Version is 2.2.1
Till week ago I had only about 100 msg/second and onyl the default stream and Default index set.
A week ago I connected a new input with about 5000 msg/sec and for it I set up a new stream SRZ, where only stream rule is simple:
“field source must contain SRZ-1200”
and it is connected with new SRZ index set, and with this option selected:
“Remove matches from ‘All messages’ stream”
(because I wanted all SRZ messages to go to only SRZ stream and to be stored only in new SRZ index set)
It seems to work for streams, as I see large number of msgs in SRZ stream and only around 50 msg/sec in Defauls stream.
But problem is when looking at the indices windows. Both SRZ index and also the Default index seem to be filling approximately at the same pace…many 1000 of msgs per second. It looks like the SRZ messages are stored in both indexes. But when I click on few SRZ messages, they are alwasy show as "stored in srz_),
In troubleshooting this, I changed the destination of SRZ stream to be the Default Index set. This works fine, as now only Default Index set is growing. But I dont want that, I want the SRZ messages to be stored in SRZ index only.
Please elaborate on your issue.
From your description, it seems that everything is fine and working as intended.
I see the main problem in the stream option and Checkbox " “Remove matches from ‘All messages’ stream”
Here I will copy and paste the text from window with name : Editing Stream
Messages that match this stream will be written to the configured index set.
CHECKED BOX Remove matches from ‘All messages’ stream
Remove messages that match this stream from the ‘All messages’ stream which is assigned to every message by default.
My understanding of what this Check Box should do is the following:
New message arrives from source SRZ-1200. Based on the rule it is directed int the SRZ stream.
Because the box (Remove matches from…)is checked, the message is removed from Default msgs stream.
writing into index, msg should be written ONLY into the SRZ index (because we checked the box to remove it from default).
And it also works as expected, for example on my test system.
On the production this is not always the case:
CORRECT INDEX ALOCATION: In production, some messages can be seen in GUI as
"Routed into stream": SRZ (correct) and then it say
Stored in index: srz_98 (correct)
WRONG INDEX ALOCATION: Lots of messages in GUI have such description:
“Routed into stream”: SRZ (correct)and then it say
Stored in index: graylog_234 (WRONG, as it is defaul index)
Why are some Msgs from stream SRZ written in the Defaul index set ? And many are still writeninthe correct index srz… I dont think is correct, all messages from stream SRZ should be stored into srz index, to my understanding. (the text in the described window says: “Messages that match this stream will be written to the configured index set.”).
I hope this describes the problem better.
I tried to describe the problem even better, because it works fine in test .Then I found out that only msgs that are received by ONE of the Three nodes see this problem.
It looks like that node is running older version 2.1.2 , and when Openonig window Edit stream, there is no option to “Remove matches from ‘All messages’ stream”.
After upgrade of that node this should start working.
Sorry for this, nothing wrong with graylog, just my admin error.