Lost all stream access

[mntbighker (Mark M)]

edit: For reasons I can’t explain after multiple restarts graylog just decided to start working again. #shrug

1. Describe your incident:
Lost access to all streams, logs and user account settings

2. Describe your environment:

• OS Information: RedHat 8.8

• Package Version: Graylog 4.3.14

I’m seeing:

Loading component failed: Loading chunk 5f73c643-169 failed. (error: https://graylog.foo.com/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/5f73c643-169.1397ed61dfe654a0a6f4.js)

Is this the expected behavior if the Ent lic is exceeded? It seems to keep digesting logs but all access to them is blocked? We added a lot of clients today, but I have never seen violation happen in one day. There was no reported violation this morning.

I’m also seeing these:

[235]: index [graylog_73], type [_doc], id [af9dbe6b-f5e5-11ed-82c4-00155dcc7560], message [Elastics
earchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total field
s [1000] has been exceeded]]]
[379]: index [graylog_73], type [_doc], id [af9f1df4-f5e5-11ed-82c4-00155dcc7560], message [Elastics
earchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total field
s [1000] has been exceeded]]]

So maybe one of the added clients has thrown a wooden shoe into the mechanism?

[ihe]

even if it is working, you should check this post: ElasticsearchException ... Limit of total fields [1000] has been exceeded
It will help you to reduce your fields to a limit below 1000.

[mntbighker (Mark M)]

I believe the RedHat 8.8 update caused SELinux to start killing part of Graylog. I made SEL permissive and at some point after that GL just started working again.

[chris.black-gl (Chris Black)]

Glad to hear it, but you still need to address the field limit issue in the logs. You are dropping some number of logs until you deal with it.

Review the doc @ihe sent and use your streams and index routing to break your traffic into separate index sets, so you have fewer streams per index.

[mntbighker (Mark M)]

Thanks, I will look into field limits next week.

I was able to make SEL enforcing again after relabeling a shed load of files, including a LOT of Elastic files. Not sure what RH8.8 did, but that seems to have been the catalyst.