Web interface flickers because of SSL certificate

ivan_miller · May 11, 2023, 9:15am

Hello all,

I have the following problem:

A few weeks ago my Graylog web interface started flickering and I could not start any input, although I still received logs.

After trying around several times, I thought it might be the Elasticsearch and decided to set up a new Graylog instance on an Ubuntu 22.04.2 LTS with Opensearch.

The new Graylog instance ran fine until I added the SSL certificate.
The SSL certificate was recognized by Graylog without any problems and I also reach the webinterface via HTTPS but the webinterface starts flickering again and for a few seconds a message appears that the server is currently unavailable and then the webinterface is displayed again. And this repeats itself every few seconds.

As soon as I comment out TLS and the certificate in the server.conf file, it works fine again and the webinterface doesn’t flicker anymore and doesn’t lose the connection.

I hope you can help me.

Many greetings

drewmiranda-gl · May 11, 2023, 8:13pm

This sounds like the graylog-server service is possible starting, encountering an error, then stopping. Are you able to post your server.log or at least the most recent relevant log lines from it?

I coincidentally just published an article about HTTPS and Graylog Web: How-To Guide: Securing Graylog with TLS
Give that a read and see if there me be anything you are missing.

ivan_miller · May 12, 2023, 7:03am

2023-05-12T00:44:28.835Z ERROR [MessagesAdapterOS2] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[14]: index [graylog_5], id [25dd5e10-f05e-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
[179]: index [graylog_5], id [2676b7e1-f05e-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T00:44:30.830Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[0]: index [graylog_5], id [270f4e60-f05e-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T00:44:50.838Z ERROR [MessagesAdapterOS2] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[285]: index [graylog_5], id [33d87810-f05e-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
[287]: index [graylog_5], id [33d8c632-f05e-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T01:01:39.829Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[253]: index [graylog_5], id [8d29ffe0-f060-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T01:16:50.829Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[149]: index [graylog_5], id [ac343fc2-f062-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T01:40:23.833Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[165]: index [graylog_5], id [f5eda9f0-f065-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T01:40:30.824Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[8]: index [graylog_5], id [fa40fe81-f065-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T01:40:41.839Z ERROR [MessagesAdapterOS2] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[281]: index [graylog_5], id [013dc833-f066-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
[282]: index [graylog_5], id [013dc835-f066-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T01:56:08.833Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[210]: index [graylog_5], id [2996b102-f068-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T03:21:42.826Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[32]: index [graylog_5], id [1d13e8b1-f074-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T03:37:33.827Z ERROR [MessagesAdapterOS2] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[122]: index [graylog_5], id [53e0ec11-f076-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T05:39:44.833Z ERROR [MessagesAdapterOS2] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[131]: index [graylog_5], id [65e5bec2-f087-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
[136]: index [graylog_5], id [65e60ce3-f087-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
2023-05-12T05:40:27.844Z ERROR [MessagesAdapterOS2] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:$
[317]: index [graylog_5], id [7febbb80-f087-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$
[321]: index [graylog_5], id [7febe293-f087-11ed-a4bc-0050568eabcd], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]$

drewmiranda-gl · May 12, 2023, 2:29pm

Those errors appear to all be related to the default 1000 field limit with Elasticsearch/OpenSearch. You’ll need to increase the field limit, even if temporarily.

I recommend you separate different types of log sources into their own streams and indexes (via stream routing and configuring the stream to use its own index set).

For stream routing you can use either stream rules or pipeline rules.

ivan_miller · May 15, 2023, 12:00pm

I have now looked at the docs and it fails on the first command to figure out how many fields I need.

I enter the command “curl -XGET ‘localhost:9200/graylog_2/?pretty’ | grep type | wc -l” and get the following output:

wc: invalid option – ‘�’
Try ‘wc --help’ for more information.
% Total % Received % Xferd Average Speed Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 --:–:-- --:–:–:–:–:-- 0curl: (6) Could not resolve host: xn–localhost-499d

I also tried to replace graylog_2 with graylog_7 but without success.

And one more thing. I didn’t install Java at all, because it wasn’t necessary according to the Graylog documentation, so I didn’t make an entry in the Java keystore.

drewmiranda-gl · May 15, 2023, 6:19pm

Looks like wordpress did some character replacing with invalid chars

i had to replace the single quotes (') and dashes (-):

curl -XGET 'localhost:9200/graylog_2/?pretty' | grep type | wc -l

You’ll also need to replace graylog_2 with whatever index is giving you the 1000 field error, though it likely is the most recent (high number) graylog_ index.

To give you some idea of what the output looks like, here is what happens when i run it:

drew@hplap:~$ curl -XGET 'localhost:9200/graylog_2/?pretty' | grep type | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14326  100 14326    0     0  2924k      0 --:--:-- --:--:-- --:--:-- 3497k
167

Forgot to mention java, so you can still use the javakey store that is bundled with the java version bundled with graylog. It should be covered in the blog post i linked to about graylog and TLS.

Hope that helps!

ivan_miller · May 25, 2023, 1:41pm

I looked through your documentation on SSL and didn’t find anything noticeably wrong other than not adding anything to the JAVA keystore because I didn’t install JAVA in the first place.

Was not necessary according to the Graylog doc.

I have now updated my Graylog instance to 5.1.1 with the hope that it will solve the problem, but it still didn’t work.

It can’t be Opensearch, because as soon as I disable TLS everything works fine again.

And as soon as I enable TLS again, the web interface flickers again.

ivan_miller · June 7, 2023, 11:59am

I suspect that I have solved the problem, but unfortunately I cannot explain why it happened that way.

I opened the Graylog web interface today on an Ubuntu client machine using the Firefox browser and noticed that nothing flickers anymore.

Then I was confused because on my Windows machine I also use the Firefox browser. However, I decided to try another browser on my Windows machine and I chose the Microsoft Edge browser.

I can’t explain why the Graylog web interface flickers and loses connection with the Firefox browser, but I found that it runs smoothly without flickering with the Microsoft Edge browser.

Maybe someone has the same problem as me and it could help them.

drewmiranda-gl · June 7, 2023, 2:06pm

Are you able to get a video that shows what you are describing? I’m very curious!

system · June 21, 2023, 2:07pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog Web interface doesnt open with HTTPS Graylog Central (peer support)	2	355	August 5, 2020
Unable to start graphical interface after enabled TLS in Graylog 3.0 Graylog Central (peer support)	11	1938	April 5, 2019
Unable to start any input after setting up web interface as SSL Graylog Central (peer support)	5	663	June 17, 2019
HTTPS is going to destroy my whole project :( Graylog Central (peer support)	19	1436	February 26, 2021
SSL web interface Graylog Central (peer support)	8	2721	June 19, 2018

Web interface flickers because of SSL certificate

Related Topics