Help in Better performance of my Elasticsearch-Graylog Setup

Graylog Central (peer support)
1

image
image.png1854×752 92.3 KB

Hi Team,

I have a setup of 3 Graylog Nodes and 5 ES nodes. The search queries in my setup take about 400ms- 500ms to execute. Can you help with speeding up the execution time?

The ES config file looks like:

curl -XPUT "https://graylog-es-cluster.service.%%RUNTIMEDC%%.coi:9200/_cluster/settings" -d '{
  "persistent" : {
    "indices.recovery.max_bytes_per_sec": "100mb",
    "cluster.routing.allocation.cluster_concurrent_rebalance": 10,
    "cluster.routing.allocation.node_concurrent_recoveries": 10 
  }
}'

It’ll be a great help if anyone could give a solution to speed up the queries to a shorter time span?

Thanks,
Srijoni

2

For legibility, could you please go back and edit your post? Select the copy/pasted CURL-stuff and hit that </> button. This makes for much nicer formatting.

I have a setup of 3 Graylog Nodes and 5 ES nodes. The search queries in my setup take about 400ms- 500ms to execute. Can you help with speeding up the execution time?

The amount of nodes you have is only one fraction of the whole puzzle :slight_smile:

  • How big is your whole data set?
  • How many indexes do you have?
  • What is your sharding/replication strategy in Elastic?
  • Are you querying against the full data set or one index?

There are so many factors that we need to know :slight_smile: If I’m querying a few TB of data for a complicated pattern with many ANDs and ORs, I’d be happy with 400-500ms :slight_smile:

3
  • How much memory and heap does the ES have?
  • How much is the heap usage?
  • Do you see any io wait on your ES servers?
  • Any cpu/load problem?

monitor,monitor, monitor all necessary parameters. After you do it, monitor all unnecessary parameter too! It can be helpful when you need to debug…

//I think the 4-500 ms query time not a big problem.

4

Then again, you are one of those folks I was referring to, with the humongous datasets :slight_smile:

5

You may want to add a few data nodes to your Elastic cluster :slight_smile:

I’ll have you know most of our queries complete in <200ms … super long time range ones still come in under 500ms so I’m with @macko003 there, 500ms isn’t bad :smiley:

1 Like
6

image

Shards: 4
Replicas :1
Index rotation strategy: Index Size
Max Index size: 5.0GB
Querying against indexes

7

On 5 ES nodes, even searching across that entire index set should complete way faster, in a few dozen ms even. Looks like your ES nodes are either all on the same server, or hilariously underpowered. Got specs?

1 Like
8

Graylog:
CPU: 5 cores
Memory: 10 GiB
ES:
CPU: 3 cores
Memory: 30 GiB

Anyother specs you looking for in particular?

9

Okay, so for ES, you want at least 64Gb of RAM with 32Gb allocated to ES, and at least 8 cores - anything below that will cause slow performance.

10

Okaaay! If with my current specs, I implement the following tuning, will they cause any significant changes?
Solutions:

  • refresh_interval to 30 seconds in ES
  • Graylog server.conf : increase output_batch_size outputbuffer_processors
  • Adding a node to Graylog.

Wanted to check if I could extend the current setup to the max performance.

11

None of this will make any difference to be honest, you need to tune Elasticsearch, not Graylog - so bigger CPU on Elastic, bigger memory as well. Otherwise everything else isn’t worth it :slight_smile:

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.