Server currently unavailable for Ubuntu 16.04

Graylog Central (peer support)
1

Background

I have an Ubuntu 16.04 clean VM, and I am trying to install the latest version of graylog on it. Unfortunately, it is failing.

What did I do?

First I created a clean VM with the afore mentioned OS.
Then I followed the official instructions step by step:

http://docs.graylog.org/en/2.4/pages/installation/os/ubuntu.html

I installed the prerequisites, mongo DB, ElasticSearch and last but not least, Graylog 2.4.

I also configured /etc/elasticsearch/elasticsearch.yml to have cluster.name: graylog ( before installing graylog ).

Graylog conifg

Now, we both know graylog won’t start out of the box. You need to do some config. My configuration for /etc/graylog/server/server.conf is really simple:

rest_listen_uri = http://0.0.0.0:9000/api/
web_listen_uri = http://0.0.0.0:9000/

That’s it.
I also created the necessary keys and passwords as usual.

What’s the problem

When I connect to my server via its public IP with port 9000, I get the following error message:

We are experiencing problems connecting to the Graylog server running on http://10.0.0.11:9000/api/. Please verify that the server is healthy and working correctly.

You will be automatically redirected to the previous page once we can connect to the server.

Do you need a hand? We can help you.

Following are the details:

Error message
Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.
Original Request
GET http://10.0.0.11:9000/api/system/sessions
Status code
undefined
Full error message
Error: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.

I can access the web page, but I immediately see the given error. I have another instance configured the same way but with graylog 2.2 and it never gave me any trouble.

What did you try to solve the problem?

My first idea was to check the health of the services. When I run sudo systemctl status XXXX, replacing XXXX with mongod, elasticsearch and graylog-server they all are active and running.

I also checked for similar errors in this forum but they are for different OSs so I am kinda lost here.

GIve me some specs!

Following are the specs of my OS ( lsb_release -a):

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

And the browser I am using:

Firefox Quantum
60.0 ( 64-bit )

Questions

  1. Did I miss come config step?
  2. Why am I getting this error and how can I fix it?
2

What happens when you try to open http://10.0.0.11:9000/api/ in your web browser?

3

I get a “The connection has timed out” message from Firefox :S

4

How are you accessing the Graylog web interface in your web browser?

5

I paste the public IP directly in the address bar: http://104.40.243.143:9000. I honestly don’t know where that http://10.0.0.11:9000/api/ is comming from.

I do it like this for our test server with graylog 2.2 and it works fine. Should I change some config files?

6

If you tell Graylog to listen on all network interfaces, it’s using the first public network interface (and its IP address) to communicate with the Graylog REST API.

You need to customize web_endpoint_uri accordingly, see also http://docs.graylog.org/en/2.4/pages/configuration/web_interface.html.

7

The entire stack is on the same machine. We have no NGINX nor anything.
I also tried:

rest_listen_uri = http://104.40.243.143:9000/api/
web_listen_uri = http://104.40.243.143:9000/

But if I do that it just doesn’t connect.
What baffles me is that this used to work in previous versions of GRaylog 2 but now it doesn’t :S
I have read and re-read the page you have given me before but I found nothing that can help me.

8

Time to check the logs of your Graylog node.
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html#graylog

9

Ok, I checked the /var/log/graylog-server/server.log file and I found the following ERROR:

2018-05-15T10:23:27.380Z ERROR [LookupDataAdapter] Couldn’t start data adapter spamhaus-drop/5afaa6ec0d04fd376582c8a3/@1a3cddbc
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:68) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-05-15T10:23:27.391Z ERROR [LookupDataAdapter] Couldn’t start data adapter abuse-ch-ransomware-domains/5afaa6ec0d04fd376582c8a4/@13a0401c
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-05-15T10:23:27.392Z ERROR [LookupDataAdapter] Couldn’t start data adapter tor-exit-node/5afaa6ec0d04fd376582c8a6/@fa8ac27
org.graylog.plugins.threatintel.tools.AdapterDisabledException: TOR service is disabled, not starting TOR exit addresses adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.tor.TorExitNodeDataAdapter.doStart(TorExitNodeDataAdapter.java:73) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-05-15T10:23:27.384Z ERROR [LookupDataAdapter] Couldn’t start data adapter abuse-ch-ransomware-ip/5afaa6ec0d04fd376582c8a8/@510f1006
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-05-15T10:23:27.403Z INFO [LookupTableService] Data Adapter spamhaus-drop/5afaa6ec0d04fd376582c8a3 [@1a3cddbc] RUNNING
2018-05-15T10:23:27.409Z INFO [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5afaa6ec0d04fd376582c8a8 [@510f1006] RUNNING

Something related to Spamhaus-drop, I have no idea what that is.

10

Please provide the complete log file (see https://community.graylog.org/faq#format-markdown for formatting hints).

You can use https://0bin.net/ or https://gist.github.com/ to share if it’s too big to post here.

11

Ok, here are the full logs !

https://0bin.net/paste/-uNsuB3ybbA80e1y#Ds49fwJhO7ZC9byYVZgVXrt-hbEX2OUululJNi96sZo

12 
2018-05-15T10:23:54.821Z INFO  [JerseyService] Started REST API at <http://localhost:9000/api/>
2018-05-15T10:23:54.821Z INFO  [JerseyService] Started Web Interface at <http://localhost:9000/>

You’ve configured Graylog to listen on the loopback interface (localhost/127.0.0.1), so that only clients on the very same machine can access it.

13

Yes, I have tried that after trying with 0.0.0.0 and with the server public IP. Nothing seems to work :S

What IP should I put there? (given that placing the server’s public IP doesn’t work! )?

14

It seems as if you’ve just ignored my previous post in this topic.

15

Oh @jochen, forgive me if my actions gave you that impression, but nothing could be further from the truth.
I did read the article you mentioned, and I did it more than once.

The problem is that I am new at devops as well as to graylog, so I don’t understand how the documentation is supposed to help me.

For example, I managed to access everything using the following config:

rest_listen_uri = http://0.0.0.0:9000/api/
web_listen_uri = http://0.0.0.0:9000/
#web_endpoint_uri =
rest_transport_uri = http://104.40.243.143:9000/api/

Now, I know that since web_endpoint_uri is commented, it uses rest_transport_uri. So this is equivalent to the following:

rest_listen_uri = http://0.0.0.0:9000/api/
web_listen_uri = http://0.0.0.0:9000/
web_endpoint_uri = http://104.40.243.143:9000/api/

Which also works. But the part I really can’t understand is why this works ( passing the public IP in web_endpoint_uri ) but it doesn’t when I put this public IP in rest_listen_uri and web_listen_uri respectively.

I know that 0.0.0.0 loops back to a local network interface, but I don’t understand how this helps me nor why it makes it work.

I am also not even trying to use proxies nor load balancers, this is all just a barebones installation.

I understand that my questions may seem a little bit off, but that is only because of my inexperience, not because I think your advice is not good enough. It most definitely is, it’s just that I still lack the knowledge to infer end results from it, as all newcomers lack.

16

This also means, that Graylog itself (and other Graylog nodes you might add in the future) will use the public network interface to access the Graylog REST API instead of the private network interface.

Has the “public IP” been set-up on the machine running Graylog, i. e. is it listed in the output of one of the following commands?

# ip addr show
# ifconfig -a

If not, that’s the problem. You can only listen on network interfaces (or IP addresses) which have been set up on the machine running Graylog.

No, it doesn’t. It’s quite the opposite and you’re mixing it up with 127.0.0.1 (localhost).

17

Currently, I fail to see how this will affect our Azure stack. I will surely keep it mind!

Has the “public IP” been set-up on the machine running Graylog, i. e. is it listed in the output of one of the following commands?

Well, no it doesn’t!
This is somehow confusing to me because the Azure panel specifically shows 104.40.243.143 as the public IP for the machine. Azure support was also very surprised to see 10.0.0.11 claiming this IP shouldn’t exist, when after all, it is in eth0.

The more you know!

So, if I understand correctly, the correct configuration should be:

rest_listen_uri = http://10.0.0.11:9000/api/
web_listen_uri = http://10.0.0.11:9000/
web_endpoint_uri = http://104.40.243.143:9000/api/ #we leave this one as is, right?

I assume this is somehow preferable to using 0.0.0.0, right ?

And thank you for clearing out that 127.0.0.1 vs 0.0.0.0 confusion!
This is what I get for reading random blogs about DevOps. Somehow, the wiki is more reliable :stuck_out_tongue:

Thanks for all the help!

18

Please refer to the following articles:

1 Like
19

Thanks!
Will have a look on those articles.

If by the end I still have questions on the proper way to configure graylog I will post them here just to be sure!

20

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.