Graylog search is slow

Graylog Central (peer support)
1

1. Describe your incident:

graylog search is slow and returning this error
Error!

Executing search failed: SyntaxError: Unexpected token ‘i’, “timeout” is not valid JSON

2. Describe your environment:

we are using graylog 6.3.1 running on AWS ecs fargate with 1 master node and 1 slave node. DB is aws opensearch 2.19 running on 6 r7g.large.search

3. What steps have you already taken to try and solve the problem?

initially, we are running aws opensearch 3 r7g.medium.search which we have already upgraded and increased the node to 6 but still slow. we have also updated graylog to latest and opensearch from 2.17 to 2.19 but still getting the same error.

2

Please provide more context from your server.log. Or where did you see this error?

3

came from graylog console.

4

image
image1586×714 97 KB

5

Hi, thank you for supplying more info. You write “graylog search is slow” - but the image shows exceptions regarding bulk indexing - which has nothing to do with searching.
Also, the errors shown mention a NullPointerException not a Syntax error from your first post.

Can you please clarify your initial question and provide additional info?

6

before I upgrade aws opensearch from 2.17 to 2.19, the logs are not displaying on the graylog console and it returns this error.
Executing search failed: SyntaxError: Unexpected token ‘i’, “timeout” is not valid JSON

after upgrading to aws opensearch 2.19, I am now getting the error attached on the screenshot.

image
image1910×870 34.5 KB

7

Can we go back to your upgrade process? From what version(s) did you start? How long has this been running without problems?
I assume it worked with your old version of Graylog and OpenSearch 2.17? Am I correct? And after the Graylog update to 6.3.1 the problems started? And then you updated to 2.19?

How does the “search is slow” come into play? That was already with the old, running versions?

8

more than 2 years already without any problem. even prior to to upgrade, we already have the slow issue. now even the default search with 5 mins time frame is not displaying.

9

Hello @glennfernandez83,

Another user who experienced the same error contained within your screenshot found that removing the below option from the opensearch.yml file fixed the issue.

compatibility.override_main_response_version: true

10

i am using aws opensearch service

11

When you initially upgraded to OS 2.17, what version were you on? Moving forward you should use this compatibility matrix to make sure you are upgrading to a supported version.

Check the option is enabled by running the below.

curl -X GET https://user:password@URL:443/_cluster/settings?include_defaults=true

If it is you can use this next command to disable

curl -X PUT https://user:password@URL:443/_cluster/settings -H 'Content-Type: application/json' -d '
{
  "persistent" : {
    "compatibility.override_main_response_version" : false
  }
}'
12

it was 2.17 before we upgraded to 2.19.

13

this worked and we can now see our logs. however there’s some notifications

image
image1405×512 31.6 KB

14

Check the server.conf of you Graylog nodes for the is_leader option, it should be true on one node and false on all others. Make the change and restart the Graylog service.

15

image
i have that option on 1 node and false on other node.

16

That’s good.

If you close the notifications, do they reappear?

17

yes, still come back.

18

The No leader node error can occur when resource utilisation is high on the Mongo/Graylog node but the multiple leader issue is less clear.

At this point it would be a good idea to paw through the Graylog server logs to see if that gives a clearer picture of what is occurring.

19

I checked this today and can see that opensearch 2.19 is already supported. since we are running aws opensearch 2.19, I decided to set this to true

curl -X PUT https://user:password@URL:443/_cluster/settings -H ‘Content-Type: application/json’ -d ’
{
“persistent” : {
“compatibility.override_main_response_version” : false
}
}’

however, I got this error instead.
The Indexer version which is currently running (Elasticsearch:7.10.2) has a different major version than the one the leader node was started with (OpenSearch:2.19.0). This will most probably result in errors during indexing or searching. A full restart is required after a major Indexer version upgrade.

I restarted my graylog master and node but still error persist and logs were not showing again.

20

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.