Web interface stops responding intermittently

To check use top/htop see if you can see the added CPU’s. If you do you should be all good. I’ve been looking into your issue all day, and re-reading what’s been posted here. It could be a couple things but I want to take one step at a time. The first being resources if its not that maybe move on to configurations.

Htop screenshot below. Still can’t access the web interface though.

Thanks gsmith, I really appreciate you help on this by the way.

No problem, glad to help. Give me a few I want to do research. If you get a chance those log files would help, but make sure you cut/blur out personal info.

Thanks :slight_smile: What log files do you need and how do I get them?

Graylog Logs since were dealing with the Web UI.

One more idea, it has to do with Network Port Conflicts. I’ve seen this before when two or more services are using the same port. For example port 9000. Just to make sure your Graylog web UI and Elasticsearch are using different ports.

You can find out by using something like this.

sudo lsof -i -P -n | grep LISTEN

Results,

If those are all good perhaps your Graylog configuration file.
Also check you firewall , selinux/apparmor if there enabled.

I have probably misled you regrading the default port for ES, I think that is still 9200. The web interface is on port 9000.

systemd-r  797 systemd-resolve   13u  IPv4  19826      0t0  TCP 127.0.0.53:53 (LISTEN)
mongod     945         mongodb   11u  IPv4  20270      0t0  TCP 127.0.0.1:27017 (LISTEN)
java       999   elasticsearch  124u  IPv6  27295      0t0  TCP [::1]:9300 (LISTEN)
java       999   elasticsearch  126u  IPv6  26217      0t0  TCP 127.0.0.1:9300 (LISTEN)
java       999   elasticsearch  151u  IPv6  26242      0t0  TCP 127.0.0.1:9200 (LISTEN)
java       999   elasticsearch  152u  IPv6  26241      0t0  TCP [::1]:9200 (LISTEN)
sshd      1023            root    3u  IPv4  22797      0t0  TCP *:22 (LISTEN)
sshd      1023            root    4u  IPv6  22799      0t0  TCP *:22 (LISTEN)
nginx     1066            root    8u  IPv4  23022      0t0  TCP *:80 (LISTEN)
nginx     1066            root    9u  IPv6  23023      0t0  TCP *:80 (LISTEN)
nginx     1066            root   10u  IPv4  23024      0t0  TCP *:443 (LISTEN)
nginx     1067        www-data    8u  IPv4  23022      0t0  TCP *:80 (LISTEN)
nginx     1067        www-data    9u  IPv6  23023      0t0  TCP *:80 (LISTEN)
nginx     1067        www-data   10u  IPv4  23024      0t0  TCP *:443 (LISTEN)
nginx     1068        www-data    8u  IPv4  23022      0t0  TCP *:80 (LISTEN)
nginx     1068        www-data    9u  IPv6  23023      0t0  TCP *:80 (LISTEN)
nginx     1068        www-data   10u  IPv4  23024      0t0  TCP *:443 (LISTEN)
nginx     1070        www-data    8u  IPv4  23022      0t0  TCP *:80 (LISTEN)
nginx     1070        www-data    9u  IPv6  23023      0t0  TCP *:80 (LISTEN)
nginx     1070        www-data   10u  IPv4  23024      0t0  TCP *:443 (LISTEN)
nginx     1071        www-data    8u  IPv4  23022      0t0  TCP *:80 (LISTEN)
nginx     1071        www-data    9u  IPv6  23023      0t0  TCP *:80 (LISTEN)
nginx     1071        www-data   10u  IPv4  23024      0t0  TCP *:443 (LISTEN)
java      1182         graylog   92u  IPv6  28853      0t0  TCP x.x.x.x:9000 (LISTEN)

Oh, you have Nginx installed and I assume your using Nginx as a proxy?

I would look into those logs also. A while back a community member had a similar issue like yourself, it ended up being something with Nginx not sure if they fixed it. Ill try to dig it up maybe something in there might help.

EDIT: Check your firewall , selinux/apparmor if there enabled.

Here is my Lab GL Server configuration. Maybe something in there might help.

[root@graylog elasticsearch]# cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = 8-6-7-5-3-0-9_something_something
root_password_sha2 =
root_email = "greg.smith@domain.com"
root_timezone = America/Chicago
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.10.10.10:9000
http_enable_cors = true
elasticsearch_hosts = http://10.10.10.10:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 5000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 12gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://mongo_admin:password@localhost:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = localhost
transport_email_port = 25
transport_email_subject_prefix = [graylog]
transport_email_from_email = root@domain.com
transport_email_web_interface_url = http://10.10.10.10:9000
http_connect_timeout = 10s
proxied_requests_thread_pool_size = 32
prometheus_exporter_enabled = true
prometheus_exporter_bind_address = graylog.domain.com:9833

Nginx is loaded but both the access and error logs are empty.

Selinux is not running but apparmor is.

Here’s the result of your cat command on my GL config file.

root@SystemsLoggingGraylog-Live:/etc/graylog/server# cat /etc/graylog/server/server.conf | egrep -v “^\s*(#|$)”
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = ***********************
root_password_sha2 = ********************
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.10.0.131:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 6gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

There are a few lines missing, and some differences in values that I’ve highlighted.

Thanks

Hello,

Couple things I noticed in you config file. With the amount of CPU’s you have compared to the configurations you showed above. I would assume you have 10 physical CPU cores. Shown below that I quoted each one of those configuration I do believe creates a thread. It is recommended that processbuffer_processors, outputbuffer_processors and inputbuffer_processors are no greater then your physical CPU cores. So in other words you should have 10 physical cores or more.

And

If your receiving that many messages I would defiantly kick it up a notch. One of my small Graylog server in production is running 14 cpu, 10 GB mem, a TB storage. Its receiving about 30 GB logs a day and my Graylog server is in a happy place. Using the following configuration. If you add them up = 12 and I reserved 2 cores for the server.

processbuffer_processors = 7
outputbuffer_processors = 3
inputbuffer_processors = 2

I’m not 100% sure but you may need more CPU cores then 4.

Hey gsmith, happy New Year, I hope you had a good one?

This server is an EC2 t2.xlarge instance with 4 vCPUs. Pardon my lack of knowledge about AWS, but how does that relate to physical CPUs? I’ve upgraded the instance to a t2.2xlarge with 8 vCPUs but I still can’t get the web interface to respond.

Thanks

Thank you , I hope you had a good one also :slight_smile:

Couple things to try.

  • Check your services
    Systemctl status graylog-server
  • Check the Graylog log file. This would be a good place to start to find out Why and What went wrong.

If you don’t see anything in the log that may pertain to the issue try restarting Graylog service and tail graylog log file. If I understand you correct now you can not get the Web UI to respond? Can you log into the web ui?

Is your Graylog configuration still the same as shown above? If not can you repost your new config here?
Normally when I add CPU or Memory to my Virtual machine sometimes a reboot helps.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.