Graylog upgraded to 3.0 - Blank screen

Graylog Central (peer support)
1

I just upgraded us from v2.5 to v3.0 and now Graylog only displays a blank white page when I load it up in the browser.

Logs don’t really reveal anything, no errors. I am sure I have messed up one of my new config settings - but which one? Below is my server.conf, if anyone can point me in the right direction I’d appreciate it.

## Global
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = <removed> 
root_password_sha2 = <removed>
root_email = <removed>
root_timezone = EST
plugin_dir = /usr/share/graylog-server/plugin
rules_file = /etc/graylog/server/rules.drl
lb_recognition_period_seconds = 3
ldap_connection_timeout = 5000
alert_check_interval = 45

## REST API & Web
rest_listen_uri = http://0.0.0.0:12900/
rest_transport_uri = http://192.168.48.158:12900/
http_external_uri = http://192.168.48.158/
http_bind_address = 127.0.0.1:9000
http_publish_uri = http://192.168.48.158/

## Search
allow_leading_wildcard_searches = true
allow_highlighting = false

## Elasticsearch
elasticsearch_shards = 1
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog2
elasticsearch_cluster_name = locker2
elasticsearch_discovery_zen_ping_multicast_enabled = false
elasticsearch_discovery_zen_ping_unicast_hosts = 192.168.48.158:9300
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

##### ADDED IN v2.3
elasticsearch_hosts = http://192.168.48.158:9200
elasticsearch_connect_timeout = 10s
# elasticsearch_idle_timeout = 999
elasticsearch_max_total_connections = 20
elasticsearch_max_total_connections_per_route = 2
elasticsearch_max_retries = 2
elasticsearch_socket_timeout = 60s
elasticsearch_discovery_enabled = false
elasticsearch_discovery_filter =
elasticsearch_discovery_frequency = 30s

## Message Journal
message_journal_enabled = true
message_journal_dir = /mnt/store1/graylog-server/journal/
message_journal_max_age = 72h
message_journal_max_size = 150gb
message_journal_flush_age = 1m
message_journal_flush_interval = 1000000
message_journal_segment_age = 1h
message_journal_segment_size = 400mb

## MongoDB
mongodb_uri = mongodb://<removed>@127.0.0.1:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5

# Email transport
transport_email_enabled = true
transport_email_hostname = <removed>
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = [Graylog]
transport_email_from_email = <removed>
transport_email_web_interface_url = <removed>

# Content Packs
content_packs_loader_enabled = false
# content_packs_dir = data/contentpacks
# content_packs_auto_load = grok-patterns.json

# Plugins
usage_statistics_enabled = false
dns_resolver_enabled = true
dns_resolver_run_before_extractors = true
2 
## REST API & Web
rest_listen_uri = http://0.0.0.0:12900/
rest_transport_uri = http://192.168.48.158:12900/
http_external_uri = http://192.168.48.158/
http_bind_address = 127.0.0.1:9000
http_publish_uri = http://192.168.48.158/

You should re-read the documentation and the migration guide.

rest_* settings are no longer present in 3.x they are replaced by http_* settings.

Read for that 3 settings carefully. if you bind to localhost ( http_bind_address = 127.0.0.1:9000) on port 9000 it is very unlikely that your Graylog can reach itself on an IP ( http_publish_uri = http://192.168.48.158/ ) port 80.
It is also unlikely that your Browser can reach Graylog on the same IP on Port 80 ( http_external_uri = http://192.168.48.158/ ).

Make the above match your Setup and it will work. It is very likely that adjusting http_bind_adress to 0.0.0.0:9000 and adding :9000 to the both other settings will solve your problems.

3

Thanks Jan… I think I accidentally re-added the REST entries when troubleshooting, I will take them out again.

The problem is that even after reading documentation it still is not entirely clear to me what each of these settings does (I’m not a developer). I understand the server needs to talk to itself on port 9000, and I know I normally talk to it on 443 https.

To put it simply: My IP is 192.168.48.158. What should my settings be? I have them like this now based on your last message:

http_bind_address = 0.0.0.0:9000
http_external_uri = http://192.168.48.158:9000/
http_publish_uri = http://192.168.48.158:9000/

But the server fails to start, logs say something about it being unhealthy and ‘cannot assign requested address’… can you point me in the right direction again?

Thanks for your patience!

edit: From the documentation I should only have to set http_bind_address and the other two will set themselves based on that… right? But I’ve tried every address I can think of for http_bind_address without luck:

0.0.0.0:9000
127.0.0.1:9000
192.168.48.158:9000
192.168.48.158
192.168.48.158:80
etc
etc

I’ve been doing trail-and-error all morning trying different things. All I can get is a white blank screen, and no good log entries explaining whats going on… heeelp

4

the question is:

  • what IP is available on your Graylog server?
  • did you had a proxy running before Graylog?
  • what is the exact content of your Graylog log when you startup Graylog?
5

Hi Jan,

Here is a full log of me starting the service then stopping the service:
https://pastebin.com/1sWxfW3J

Currently these are my settings (see earlier post for full server.conf):

 ## REST API & Web
 http_bind_address = 0.0.0.0:9000
 http_publish_uri = http://192.168.48.158:9000/
 http_external_uri = http://192.168.48.158:9000/

Attached is a screenshot of Firefox when I load the page - it says “Graylog Web Interface” in the tab, but the page is entirely blank. Similar results in Chrome.

Graylog%20Web%20Interface
Graylog%20Web%20Interface764×563 12.9 KB

6

you entered the IP with HTTPS what defaults to port 443.

You should enter HTTP and port 9000 according to your given configuration.

That is why i have asked if you had a proxy or similar before.

7

The server ran on 443 before the upgrade (no proxies). If it was responding on 9000 I’d settle for that - but it’s not.

I get a timeout on http://192.168.48.158:9000, and no entries in the logs. When I hit https://192.168.48.158:443 it at least responds with a white page and a “Graylog Web Interface” tab. Can you point me in the direction?

8

so you changed the owner and run Graylog as root before?

I did not understand your architecture. You connect via HTTPS but never have that mentioned anywhere that you use that.

Sorry I’m out I can’t imagine what your setup looks lik.e

9

Jan :frowning: I understand your frustration. Please consider:

  • Our setup is extremely simple - we have one node, no mods, nothing strange. This is the easiest possible setup you could have (next to running the OVA). We are an average Graylog user - you can’t help us?

  • We don’t care if Graylog runs on http or https. We don’t care what port we run on. We are just trying to understand how to upgrade from 2.3 to 3.1 (we’re doing this on a virtualized cloned copy of our production system - just to make sure it is do-able).

  • This system was fully functioning at v2.5. Essentially all I did was run one rpm -uvh command and edit three lines of the config file - as instructed to get us to 3.0.

I will keep trying to figure out your software with trial and error if you won’t help us but I would really appreciate some more advice on how to troubleshoot this.

10

the problem is that all given and provided information does not make any sense. So the picture is like a grotesque surreal one but nothing that would work in a world that follow known rules.

Did you have the original server.conf from your 2.5 version? if yes, please post your rest_* and web_* parameters from that.

11

we have a similar simple setup, now at 3.1… no proxy. We only have the bind address uncommented and specified… so:

 ## REST API & Web
 http_bind_address = 192.168.72.100:9000

 #http_publish_uri =  http://192.168.1.1:9000/
 #http_external_uri =

And the web browser URL:

http://192.168.72.100:9000

Make sure you restart your browser (and perhaps clear cache) before testing your configuration - sometimes the browser will hold on to old data. Unless you are explicitly trying for (and set up for) a secure connection, don’t use https

12

Thanks tmacgbay.

Apologies Jan - I’ve dug around more and discovered that we are using Apache to proxy port 9000 http into port 443 https. I inherited this system from another administrator who has left the company and I’ve been learning as I go. Do you know why v3.0 would break it? I don’t think I saw anything in the change logs about that.

Anyways today I am working on how to fix the proxy (or remove it if I have to)… I’m getting us to v3.1 slowly but surely…

Thanks again

13

leave http_* all to defaults will make Graylog listen on localhost 9000, you are then able to use the proxy:

http://docs.graylog.org/en/3.1/pages/configuration/web_interface.html#apache-httpd-2-x

The important part - in pre 3.0 version the line for RequestHeader set X-Graylog-Server-URL did contain a /api/ at the end. Starting from 3.0 you need to remove that. The complete line looks like:

        RequestHeader set X-Graylog-Server-URL "https://graylog.example.org/"

where https://graylog.example.org/ should be the virtualhostname you have configured in apache. (that also might be the IP)

I guess the solution is - restore the defaults for Graylog and remove the api/ from that single line will make you successful.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.