Hi Team, I need help in how to reduce the time required for making my search queries run faster, and reduce the required run time. Context: I have over 5 streams and the amount of time taken for each stream is quite some. Over 200ms. So I need the query time to be reduced. Can anyone help?

[image] Srijoni: Context: I have over 5 streams and the amount of time taken for each stream is quite some. I could be wrong, but streams and queries are not directly linked are they? If I understand correctly, streams shift and move incoming data into various locations based on a ruleset. Q…

nope @Totally_Not_A_Robot when queries on streams take long - add more ressources to elasticsearch …

What does your ES cluster look like? Generally speaking, slow query response means your ES cluster is at capacity and will need to have additional data nodes added to it. 200ms isn’t actually bad, mind you… To illustrate, our setup currently runs with 19 data nodes, 3 masters, 3 routing nodes, 6.1b…

[image] jan: when queries on streams take long I don’t understand that sentence though. How do you query a stream?? You query against indices, right? Or are we mixing two discussions and are you referring to queries that are part of a stream configuration? [image] benvanstaveren: Additio…

OFF @benvanstaveren I have some questions about your cluster How did you start with this cluster? Do you planned it, or just increase when it needed? Do you have performance experiences before and after you added the master and route nodes? Do you use the 3 replicas only for the fast search? H…

To answer the questions: 1: We had an ELK cluster (6.x) consisting of 3 masters and 9 data nodes, that was the initial sizing for the cluster. We then decided to move to Graylog, and at the time we did that Graylog didn’t support ES 6.x so we set up 9 data nodes with 5.6.x to mirror our existing se…

It elucidate a lot of things, we run our servers on virtualized hardware, and I get “disk” from dedicated storage. At the last 2 years I loose a node for few hours when we had a problem in a DC. So I store only 1 replica, and do backup (also for archiving) Thanks for the tips, I will do some resea…

I’ll add a disclaimer that this setup is probably pretty particular to our needs, so it may not be the best way to do it. But it works for us :slight_smile: We basically don’t back up our ES data nodes, on account of there being 3 replicas, so chances of 4 nodes all going down at just the right tim…

I don’t understand that sentence though. How do you query a stream?? You query against indices, right? Or are we mixing two discussions and are you referring to queries that are part of a stream configuration? I guess I was to short in this sentence. A query in a stream was refered to a search i…

How to make search queries run faster?

Graylog Add-ons

macko003 January 24, 2019, 11:25am 11

if @benvanstaveren wish Users feedbacks / Guides for heavy load graylog Cluster

Topic		Replies	Views
Help in Better performance of my Elasticsearch-Graylog Setup Graylog Central (peer support)	10	4262	February 27, 2019
Enhance Graylog search performance by adding new Elastic nodes? Graylog Central (peer support)	2	1678	August 10, 2017
Terrible search performance Graylog Central (peer support)	11	2013	May 27, 2019
High load on the elasticsearch data nodes Graylog Central (peer support)	7	2968	September 13, 2019
How many nodes needed for >5TB logs in a day? Graylog Central (peer support) pipeline-rules , route-to-streampl	28	4775	February 22, 2019

How to make search queries run faster?

Related topics