GrayLog Web GUI Stuck on Restarting


(Scott Smail) #1

I’m a new graylog users and have just deployed the ova in my vmware environment. I completed the configuration, opened a browser to the IP and was greeted with a message that graylog was restarting and to wait a couple minutes. I’ve done that, but the issue persists. Nothing in the logs and the services are all running. Any ideas?


(Jochen) #2

Is there anything in the logs of Graylog or Elasticsearch at all?
:arrow_right: http://docs.graylog.org/en/2.2/pages/configuration/file_location.html#omnibus-package

What’s the configuration of your virtual machine?
Which commands did you execute to configure the OVA and what was their output?


(Scott Smail) #3

After the OVA booted up, I configured the network and ran the following commands from the documentation:
sudo apt-get install -y open-vm-tools
graylog-ctl set-email-config "my SMTP"
graylog-ctl set-admin-password "my password"
graylog-ctl set-timezone CST
graylog-ctl reconfigure

After the commands, I tried going to the web GUI and encountered the GRAYLOG is restarting issue. I’m sure I’m missing something…

If I go to /var/log/graylog/elasticsearch I see the following

-rw-r--r-- 1 root    root    2508 Jun 27 15:59 @4000000059528110199daf94.u
-rw-r--r-- 1 root    root      30 Jun 27 15:36 config
-rw-r--r-- 1 root    root    2218 Jun 27 16:00 current
-rw------- 1 graylog graylog    0 Jun 27 15:36 graylog_deprecation.log
-rw------- 1 graylog graylog    0 Jun 27 15:36 graylog_index_indexing_slowlog.log
-rw------- 1 graylog graylog    0 Jun 27 15:36 graylog_index_search_slowlog.log
-rw------- 1 graylog graylog 5009 Jun 27 16:00 graylog.log
-rw------- 1 root    root       0 Jun 27 15:36 lock

If I look at graylog.log, I see…

[2017-06-27 15:36:36,489][INFO ][node                     ] [Bela] version[2.4.2], pid[1882], build[161c65a/2016-11-17T11:51:03Z]
[2017-06-27 15:36:36,494][INFO ][node                     ] [Bela] initializing ...
[2017-06-27 15:36:40,697][INFO ][plugins                  ] [Bela] modules [reindex, lang-expression, lang-groovy], plugins [kopf], sites [kopf]
[2017-06-27 15:36:41,037][INFO ][env                      ] [Bela] using [1] data paths, mounts [[/ (/dev/mapper/graylog--vg-root)]], net usable_space [11.4gb], net total_space [14.9gb], spins? [possibly], types [ext4]
[2017-06-27 15:36:41,037][INFO ][env                      ] [Bela] heap size [2.3gb], compressed ordinary object pointers [true]
[2017-06-27 15:36:41,041][WARN ][env                      ] [Bela] max file descriptors [64000] for elasticsearch process likely too low, consider increasing to at least [65536]
[2017-06-27 15:36:47,860][INFO ][node                     ] [Savage Steel] version[2.4.2], pid[2085], build[161c65a/2016-11-17T11:51:03Z]
[2017-06-27 15:36:47,861][INFO ][node                     ] [Savage Steel] initializing ...
[2017-06-27 15:36:49,361][INFO ][plugins                  ] [Savage Steel] modules [reindex, lang-expression, lang-groovy], plugins [kopf], sites [kopf]
[2017-06-27 15:36:49,422][INFO ][env                      ] [Savage Steel] using [1] data paths, mounts [[/ (/dev/mapper/graylog--vg-root)]], net usable_space [11.4gb], net total_space [14.9gb], spins? [possibly], types [ext4]
[2017-06-27 15:36:49,432][INFO ][env                      ] [Savage Steel] heap size [2.3gb], compressed ordinary object pointers [true]
[2017-06-27 15:36:49,436][WARN ][env                      ] [Savage Steel] max file descriptors [64000] for elasticsearch process likely too low, consider increasing to at least [65536]
[2017-06-27 15:36:54,311][INFO ][node                     ] [Savage Steel] initialized
[2017-06-27 15:36:54,313][INFO ][node                     ] [Savage Steel] starting ...
[2017-06-27 15:36:54,524][INFO ][transport                ] [Savage Steel] publish_address {10.240.1.120:9300}, bound_addresses {10.240.1.120:9300}
[2017-06-27 15:36:54,535][INFO ][discovery                ] [Savage Steel] graylog/YN5lqdomQbGHlEirSlpCVA
[2017-06-27 15:37:04,596][INFO ][cluster.service          ] [Savage Steel] new_master {Savage Steel}{YN5lqdomQbGHlEirSlpCVA}{10.240.1.120}{10.240.1.120:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-06-27 15:37:04,682][INFO ][http                     ] [Savage Steel] publish_address {10.240.1.120:9200}, bound_addresses {10.240.1.120:9200}
[2017-06-27 15:37:04,684][INFO ][node                     ] [Savage Steel] started
[2017-06-27 15:37:04,737][INFO ][gateway                  ] [Savage Steel] recovered [0] indices into cluster_state
[2017-06-27 15:59:45,545][INFO ][node                     ] [Savage Steel] stopping ...
[2017-06-27 15:59:45,604][INFO ][node                     ] [Savage Steel] stopped
[2017-06-27 15:59:45,604][INFO ][node                     ] [Savage Steel] closing ...
[2017-06-27 15:59:45,611][INFO ][node                     ] [Savage Steel] closed
[2017-06-27 16:00:17,470][INFO ][node                     ] [Tethlam] version[2.4.2], pid[826], build[161c65a/2016-11-17T11:51:03Z]
[2017-06-27 16:00:17,472][INFO ][node                     ] [Tethlam] initializing ...
[2017-06-27 16:00:19,180][INFO ][plugins                  ] [Tethlam] modules [reindex, lang-expression, lang-groovy], plugins [kopf], sites [kopf]
[2017-06-27 16:00:19,287][INFO ][env                      ] [Tethlam] using [1] data paths, mounts [[/ (/dev/mapper/graylog--vg-root)]], net usable_space [11.4gb], net total_space [14.9gb], spins? [possibly], types [ext4]
[2017-06-27 16:00:19,295][INFO ][env                      ] [Tethlam] heap size [2.3gb], compressed ordinary object pointers [true]
[2017-06-27 16:00:19,297][WARN ][env                      ] [Tethlam] max file descriptors [64000] for elasticsearch process likely too low, consider increasing to at least [65536]
[2017-06-27 16:00:25,339][INFO ][node                     ] [Tethlam] initialized
[2017-06-27 16:00:25,340][INFO ][node                     ] [Tethlam] starting ...
[2017-06-27 16:00:25,582][INFO ][transport                ] [Tethlam] publish_address {10.240.1.120:9300}, bound_addresses {10.240.1.120:9300}
[2017-06-27 16:00:25,596][INFO ][discovery                ] [Tethlam] graylog/M0JYvr0kSSWNMJXKP0rqfQ
[2017-06-27 16:00:35,661][INFO ][cluster.service          ] [Tethlam] new_master {Tethlam}{M0JYvr0kSSWNMJXKP0rqfQ}{10.240.1.120}{10.240.1.120:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-06-27 16:00:35,712][INFO ][http                     ] [Tethlam] publish_address {10.240.1.120:9200}, bound_addresses {10.240.1.120:9200}
[2017-06-27 16:00:35,714][INFO ][node                     ] [Tethlam] started
[2017-06-27 16:00:35,740][INFO ][gateway                  ] [Tethlam] recovered [0] indices into cluster_state

Current shows the following…

2017-06-27_16:00:17.47114 [2017-06-27 16:00:17,470][INFO ][node                     ] [Tethlam] version[2.4.2], pid[826], build[161c65a/2016-11-17T11:51:03Z]
2017-06-27_16:00:17.47292 [2017-06-27 16:00:17,472][INFO ][node                     ] [Tethlam] initializing ...
2017-06-27_16:00:19.18049 [2017-06-27 16:00:19,180][INFO ][plugins                  ] [Tethlam] modules [reindex, lang-expression, lang-groovy], plugins [kopf], sites [kopf]
2017-06-27_16:00:19.28783 [2017-06-27 16:00:19,287][INFO ][env                      ] [Tethlam] using [1] data paths, mounts [[/ (/dev/mapper/graylog--vg-root)]], net usable_space [11.4gb], net total_space [14.9gb], spins? [possibly], types [ext4]
2017-06-27_16:00:19.29540 [2017-06-27 16:00:19,295][INFO ][env                      ] [Tethlam] heap size [2.3gb], compressed ordinary object pointers [true]
2017-06-27_16:00:19.29726 [2017-06-27 16:00:19,297][WARN ][env                      ] [Tethlam] max file descriptors [64000] for elasticsearch process likely too low, consider increasing to at least [65536]
2017-06-27_16:00:25.33986 [2017-06-27 16:00:25,339][INFO ][node                     ] [Tethlam] initialized
2017-06-27_16:00:25.34057 [2017-06-27 16:00:25,340][INFO ][node                     ] [Tethlam] starting ...
2017-06-27_16:00:25.58312 [2017-06-27 16:00:25,582][INFO ][transport                ] [Tethlam] publish_address {10.240.1.120:9300}, bound_addresses {10.240.1.120:9300}
2017-06-27_16:00:25.59670 [2017-06-27 16:00:25,596][INFO ][discovery                ] [Tethlam] graylog/M0JYvr0kSSWNMJXKP0rqfQ
2017-06-27_16:00:35.66148 [2017-06-27 16:00:35,661][INFO ][cluster.service          ] [Tethlam] new_master {Tethlam}{M0JYvr0kSSWNMJXKP0rqfQ}{10.240.1.120}{10.240.1.120:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
2017-06-27_16:00:35.71247 [2017-06-27 16:00:35,712][INFO ][http                     ] [Tethlam] publish_address {10.240.1.120:9200}, bound_addresses {10.240.1.120:9200}
2017-06-27_16:00:35.71424 [2017-06-27 16:00:35,714][INFO ][node                     ] [Tethlam] started
2017-06-27_16:00:35.74020 [2017-06-27 16:00:35,740][INFO ][gateway                  ] [Tethlam] recovered [0] indices into cluster_state
~

(Jochen) #4

What is/was the output of those commands?

These are the logs of Elasticsearch. What’s in the logs of Graylog?


(Scott Smail) #5

When I ran the install command it went through a normal installation process. I don’t recall the output for the remaining commands. isn’t the location on the graylog logs /var/log/graylog-server? If so, that directory doesn’t exist. If not, where are the logs for graylog? Here’s the listing under /var/log/graylog.

ubuntu@graylog:/var/log$ cd /var/log/graylog
ubuntu@graylog:/var/log/graylog$ ls -l
total 20
drwx------ 2 graylog root 4096 Jun 28 16:00 elasticsearch
drwx------ 2 graylog root 4096 Jun 28 16:00 etcd
drwx------ 2 graylog root 4096 Jun 28 16:00 mongodb
drwx------ 2 root    root 4096 Jun 28 16:00 nginx
drwx------ 2 graylog root 4096 Jun 28 16:00 server
ubuntu@graylog:/var/log/graylog$

(Jochen) #6

No, see http://docs.graylog.org/en/2.2/pages/configuration/file_location.html#omnibus-package.


(Scott Smail) #7

Here’s what is repeating over and over in the log… it’s referencing timezone. Here’s the command I used to set the timezone: sudo graylog-ctl set-timezone America/Chicago, but when I run the date command it shows time is off by 5 hours.

2017-06-29_12:44:12.87077 It looks like you are trying to access MongoDB over HTTP on the native driver port.
2017-06-29_12:44:16.69130 2017-06-29 12:44:16,689 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-06-29_12:44:16.69310 2017-06-29 12:44:16,692 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-06-29_12:44:16.69514 2017-06-29 12:44:16,694 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-06-29_12:44:16.69700 2017-06-29 12:44:16,696 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-06-29_12:44:16.70923 2017-06-29 12:44:16,709 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-06-29_12:44:16.71155 2017-06-29 12:44:16,711 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-06-29_12:44:16.73624 2017-06-29 12:44:16,732 ERROR: org.graylog2.bootstrap.CmdLineTool - Invalid configuration
2017-06-29_12:44:16.73658 com.github.joschi.jadconfig.ParameterException: Couldn't convert value for parameter "root_timezone"
2017-06-29_12:44:16.73728       at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:141) ~[graylog.jar:?]
2017-06-29_12:44:16.73809       at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:99) ~[graylog.jar:?]
2017-06-29_12:44:16.73849       at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351) [graylog.jar:?]
2017-06-29_12:44:16.73924       at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344) [graylog.jar:?]
2017-06-29_12:44:16.73967       at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:177) [graylog.jar:?]
2017-06-29_12:44:16.74048       at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2017-06-29_12:44:16.74092 Caused by: com.github.joschi.jadconfig.ParameterException: Couldn't convert value "US" to DateTimeZone.
2017-06-29_12:44:16.74143       at com.github.joschi.jadconfig.jodatime.converters.DateTimeZoneConverter.convertFrom(DateTimeZoneConverter.java:26) ~[graylog.jar:?]
2017-06-29_12:44:16.74187       at com.github.joschi.jadconfig.jodatime.converters.DateTimeZoneConverter.convertFrom(DateTimeZoneConverter.java:12) ~[graylog.jar:?]
2017-06-29_12:44:16.74271       at com.github.joschi.jadconfig.JadConfig.convertStringValue(JadConfig.java:167) ~[graylog.jar:?]
2017-06-29_12:44:16.74373       at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:139) ~[graylog.jar:?]
2017-06-29_12:44:16.74440       ... 5 more
2017-06-29_12:44:16.74504 Caused by: java.lang.IllegalArgumentException: The datetime zone id 'US' is not recognised
2017-06-29_12:44:16.74542       at org.joda.time.DateTimeZone.forID(DateTimeZone.java:229) ~[graylog.jar:?]
2017-06-29_12:44:16.74572       at com.github.joschi.jadconfig.jodatime.converters.DateTimeZoneConverter.convertFrom(DateTimeZoneConverter.java:24) ~[graylog.jar:?]
2017-06-29_12:44:16.74630       at com.github.joschi.jadconfig.jodatime.converters.DateTimeZoneConverter.convertFrom(DateTimeZoneConverter.java:12) ~[graylog.jar:?]
2017-06-29_12:44:16.74672       at com.github.joschi.jadconfig.JadConfig.convertStringValue(JadConfig.java:167) ~[graylog.jar:?]
2017-06-29_12:44:16.74736       at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:139) ~[graylog.jar:?]
2017-06-29_12:44:16.74778       ... 5 more

(Scott Smail) #8

I just resolved the timezone, but still experiencing the same problem.

Graylog is restarting…

There is no Graylog web application running at the moment, please reload this page in a minute. It can take up to 1-2 minutes until all services are running properly. In case this is a permanent error, check the following:

Check if all services are running - sudo graylog-ctl status shows an overview of all running services
Check for errors in log files - Relevant services write log files here: /var/log/graylog/*/current
Ask for help - If there is no way to fix the issue ask for help: https://github.com/Graylog2/graylog2-images/issues


(Jochen) #9

This graylog-ctl only configures the time zone used in Graylog, not the system timezone in the virtual machine.


(Jochen) #10

Is it the same error message in the logs or are there other messages?


(system) #11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.