Hello, i just upgrade from 5.0.9 to 5.1.3 (in docker), all went fine, but one event definition don’t work with error:
" Aggregation search failed
Event definition Total IP counter (637209c90320b33a82b4d4f0) failed: OpenSearch exception [type=too_many_buckets_exception, reason=Trying to create too many buckets. Must be less than or equal to: [100000] but was [100001]. This limit can be set by changing the [search.max_buckets] cluster level setting.].
"
This definition just get all messages from web-server for last 5 minutes, group it by remote_addr and calculate Count, if Count of some ip address more that limit, then HTTP notification happens.
It definition works normally on previous versions of GL, and i don’t changed my openserver version ( 2.8.0)
Default value of search.max_buckets is 65535, i try to increase it to 100000, but it don’t help. BTW, i don’t cleary understand, is it safely to increase it at all.
NOT source: myhostname1 AND NOT remote_addr:_IP_ AND NOT remote_addr: _IP_ AND NOT remote_addr:_IP_ AND NOT remote_addr:_IP_ AND NOT remote_addr:_IP_ AND NOT remote_addr:_IP_AND NOT remote_addr:_IP_ AND NOT remote_addr:_IP_ AND NOT remote_addr:_IP_ AND NOT source:myhostname2 AND NOT source: myhostname3 AND NOT _exists_: nginx_error
IP - is different ip addresses that i want to remove from search
myhostnameX - is hostnames of another whitelisted servers
Oh, then
Group by Field(s)
remote_addr
Create Events if
count(remote_addr) **>** 600
Your aggregation is too complex, which results in too many buckets.
If you cannot simplify the logic, I would try creating a pipeline rule which performs that check and marks each message with a boolean flag. Then aggregate based on the value of that flag.
I understand, that something wrong with my query. Strange that the same query works normal in manual search.
I don’t ever think that exclude some documents by value of one-two field would be too complex ^(
And if i understand right, the issue on the opensearch side?
