Index error after one ES node crashed

Graylog Central (peer support)
1

When clicking on the Search link in graylog, it is now giving the following error:

While retrieving data for this widget, the following error(s) occurred:

  • Elasticsearch exception [type=index_not_found_exception, reason=no such index ].

This occurred after one of the ES nodes crashed. It was restarted and Graylog now reports:

Elasticsearch cluster is green. Shards: 136 active, 0 initializing, 0 relocating, 0 unassigned,

There is a backlog of messages currently that its working through, however the process buffer and the output buffer are both at 100%. There are 3 x Graylog nodes and 3 X ES nodes.

Any ideas on what could be the issue?

many thanks
Darrin

2

Hello,

When the ES nodes crash I’m sure that your journal was filling up quick.
If the process buffer and the output buffer are both at 100% give Elasticsearch a few minutes to ingest logs from your journal. The amount of time depends on how many messages are left in the journal.

EDIT @darrinh
What do you see when executing this command?

curl -XGET http://localhost:9200/_cat/indices

What do you see in graylog, Elasticsearch Logs?

3

Hi there,

that query returns the following:

green open graylog_47 YfHKg0B7QdmybDhtbH4-Bw 4 0 20009885 0 15.7gb 15.7gb
green open graylog_46 JlOPoAt5RIWubu-cgUqvQg 4 0 20034290 0 15.3gb 15.3gb
green open graylog_49 NsswIRCrRJOH3B6w6-H2dQ 4 0 20010584 0 14.4gb 14.4gb
green open graylog_48 9hjMD3vFSgewL-_PSdzpFA 4 0 20000459 0 15.1gb 15.1gb
green open gl-system-events_2 BlVfXN_RTNGcU2DK-bQG5g 4 0 0 0 832b 832b
green open graylog_50 FVVo22JRQXCBHbfxvsNDiA 4 0 20019164 0 15.3gb 15.3gb
green open gl-system-events_3 TKR0Iw7aR06I21xYv9Sx6Q 4 0 0 0 832b 832b
green open gl-system-events_0 0DUHH3kASTKABL7l8szC9w 4 0 0 0 832b 832b
green open graylog_52 OouRanfrR5-dSRqT5BSszg 4 0 20003684 0 15.5gb 15.5gb
green open graylog_51 lpSFTKmDQxuIxVmV9zqLjw 4 0 20010395 0 15.2gb 15.2gb
green open gl-system-events_1 GgiD8fXgRfqR39nnzsQW3A 4 0 0 0 832b 832b
green open gl-system-events_6 ezkgzHQQSkOX_mTno1UJug 4 0 0 0 832b 832b
green open graylog_54 mr_qLlIERXW905wqNruvAw 4 0 20016804 0 14.5gb 14.5gb
green open graylog_53 gy3wn5pMReugjjYCw0fFqQ 4 0 20002959 0 14.8gb 14.8gb
green open gl-system-events_4 Wy4hPLUoQWGORUR20VO6pA 4 0 0 0 832b 832b
green open graylog_56 5HUYnTqBSFGxjR-YI1AxWg 4 0 20012099 0 15.7gb 15.7gb
green open gl-system-events_5 tez_k_QvTcymoemi3Z5wog 4 0 0 0 832b 832b
green open graylog_55 MXHD0iM5QAWRFcsGzldYyA 4 0 20015437 0 15.4gb 15.4gb
green open graylog_58 ov1BRy6-S4Kcs93zYZAN_g 4 0 20001513 0 14.9gb 14.9gb
green open graylog_57 _OUGQ2LTQ2Ckn4tIx3neZg 4 0 20004554 0 15.7gb 15.7gb
green open graylog_59 Jh057X9KRKerY0tAL27Raw 4 0 20022569 0 14.3gb 14.3gb
green open gl-events_6 xGu9O0EVQbm_ZX_08ebdDg 4 0 0 0 832b 832b
green open gl-events_1 GMuZeMPGRZqMPxpq-2H2iw 4 0 242 0 131.9kb 131.9kb
green open gl-events_0 CieUCs6fS_SoXcGadNUTrg 4 0 24 0 55.9kb 55.9kb
green open gl-events_5 SLG4ONFmQCKERuOLPE7Bow 4 0 959 0 297.6kb 297.6kb
green open gl-events_4 ef3nW638Q8yC9hhyeJbQ8w 4 0 8209 0 1.7mb 1.7mb
green open gl-events_3 X3HsZY81TbOWG0LYMf7oHQ 4 0 10660 0 2.3mb 2.3mb
green open gl-events_2 OT6SPV9fRcqkUO1sYWdCOQ 4 0 10657 0 2.3mb 2.3mb
green open graylog_61 mGeWUTlUSaGC8oz4RhriaA 4 0 20010363 0 15.7gb 15.7gb
green open graylog_60 TP0KzwLrRqaCZXEsldtFeQ 4 0 20015648 0 15.5gb 15.5gb
green open graylog_63 AR7Hq7nJRsu0CPVxTrfaEQ 4 0 62955669 135 64.4gb 64.4gb
green open graylog_62 qyZSteyHRaGPkStMokoY_g 4 0 135143037 64 104.8gb 104.8gb
green open graylog_45 rjqnwl3STkyXbTw_3k2Z2g 4 0 20009220 2469 15.5gb 15.5gb
green open graylog_44 c45ShOTYTAG2FctN2DlUNg 4 0 20012229 0 13.7gb 13.7gb

The process and output buffers have gotten back to 0% utilisation now, still getting that same error from the search.

many thanks
darrin

4

Do you see anything in ES logs they may point to what index that Graylog cant find. See if there is anything they may pertain to this issue.

5

I can’t see anything obvious, graylog shows this in the logs:

2021-10-05T15:49:19.588+11:00 ERROR [PivotAggregationSearch] Aggregation search query returned an error: Elasticsearch exception [type=index_not_found_exception, reason=no such index ].

If i set the search time to 5 days ago, I can see data, but any time after that gives the no index error.

6

ok, I did System > indices > [index name] > maintenance > on all of the indices and the search function has been restored.

thanks for your assistance.

cheers
darrin

7

:laughing: you beat me by seconds, I was going to ask you about that Glad it was resolved.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.