Inputs appear as not running


(Andy) #1

Hi there,

I am facing a strange problem. Well at least it’s strange to me.
I have a Graylog 2.2.3 server running inside of a Docker Container. With some tinkering I managed to get it work with https. There is a Gelf input with utilizes TLS for a secure connection and it works like a charm. It did not work in the beginning but this was due to the missing cert in the keystore.
But here is the problem. As soon as I activate TLS in the graylog-config file my inputs appear as not running, but they are. I can tell because a) the nxlog logfiles tell me that a secure connection was made for the client to the graylog server and b) because the is new data inside the search section of my graylog server sent over from said clients. The message count in the top right corner also shows 0 messages, which is also strange.
If I try to activate the “not running” input (just for the fun of it) I get an error message.


But again, my logs aren’t showing any related errors.

Am I doing something wrong here? Did I miss someting? I can provide further information or config files if needed.
Any help would be great, thanks in advance.
Cheers, Andy


(Jochen) #2

What’s the configuration of Graylog?
What’s the configuration of the Docker container?
How did you set up and configure HTTPS?


(Andy) #3

Hi,

I used the official how-to to setup HTTPS http://docs.graylog.org/en/2.2/pages/configuration/https.html
The containers were build according to this how-to:
http://docs.graylog.org/en/2.2/pages/installation/docker.html

this is the only part of the config standard config file which was altered.

rest_listen_uri = https://0.0.0.0:9000/api/
rest_enable_tls = true
rest_tls_cert_file = /usr/share/graylog/data/config/certs/cert.pem
rest_tls_key_file = /usr/share/graylog/data/config/certs/pkcs8-plain.pem
web_listen_uri = https://0.0.0.0:9000/
web_enable_tls = true
web_tls_cert_file = /usr/share/graylog/data/config/certs/cert.pem
web_tls_key_file = /usr/share/graylog/data/config/certs/pkcs8-plain.pem

My docker compose-file looks like this

version: '2'
services:
graylog-mongo:
image: "some/mongo3"
volumes:
- /data/graylog/data/mongo:/data/db
restart: always
graylog-elasticsearch:
image: "some/elasticsearch2"
command: "elasticsearch -Des.cluster.name='graylog'"
volumes:
- /data/graylog/data/elasticsearch:/usr/share/elasticsearch/data
restart: always
graylog:
image: "some/graylog2"
volumes:
- /data/graylog/data/journal:/usr/share/graylog/data/journal
- /data/graylog/data/contentpacks/:/usr/share/graylog/data/contentpacks
- /data/graylog/config:/usr/share/graylog/data/config
environment:
GRAYLOG_PASSWORD_SECRET: somepasswordpepperman
GRAYLOG_ROOT_PASSWORD_SHA2: 285e24a9290ba3572d1692be1213233e8644682bbfc4600d420ccb207e28a23f22aa
GRAYLOG_WEB_ENDPOINT_URI: https://10.8.0.43:9001/api
links:
- graylog-mongo:mongo
- graylog-elasticsearch:elasticsearch
ports:
- "9001:9000"
- "11111:11111"
- "5140:5140"
restart: always

Just in case, here is the Dockerfile of the graylog container:

FROM openjdk:8-jre
MAINTAINER Graylog, Inc. <hello@graylog.com>

# Build-time metadata as defined at http://label-schema.org
ARG BUILD_DATE
ARG VCS_REF
ARG GRAYLOG_VERSION

LABEL org.label-schema.build-date=$BUILD_DATE \
      org.label-schema.name="Graylog Docker Image" \
      org.label-schema.description="Official Graylog Docker image" \
      org.label-schema.url="https://www.graylog.org/" \
      org.label-schema.vcs-ref=$VCS_REF \
      org.label-schema.vcs-url="https://github.com/Graylog2/graylog-docker" \
      org.label-schema.vendor="Graylog, Inc." \
      org.label-schema.version=$GRAYLOG_VERSION \
      org.label-schema.schema-version="1.0" \
      com.microscaling.docker.dockerfile="/Dockerfile" \
      com.microscaling.license="Apache 2.0"


ENV GOSU_VERSION 1.10
RUN set -ex \
  && wget -nv -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture)" \
  && wget -nv -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture).asc" \
  && export GNUPGHOME="$(mktemp -d)" \
  && gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
  && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
  && rm -r "$GNUPGHOME" /usr/local/bin/gosu.asc \
  && chmod +x /usr/local/bin/gosu \
  && gosu nobody true

ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/jre
RUN set -ex \
  && addgroup --gid 1100 graylog \
  && adduser --disabled-password --disabled-login --gecos '' --uid 1100 --gid 1100 graylog \
  && mkdir /usr/share/graylog \
  && wget -nv -O /usr/share/graylog.tgz "https://packages.graylog2.org/releases/graylog/graylog-${GRAYLOG_VERSION}.tgz" \
  && tar xfz /usr/share/graylog.tgz --strip-components=1 -C /usr/share/graylog \
  && chown -R graylog:graylog /usr/share/graylog \
  && rm /usr/share/graylog.tgz \
  && setcap 'cap_net_bind_service=+ep' $JAVA_HOME/bin/java

ENV GRAYLOG_SERVER_JAVA_OPTS "-Xms1g -Xmx2g -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow"
ENV PATH /usr/share/graylog/bin:$PATH
WORKDIR /usr/share/graylog

RUN set -ex \
  && for path in \
    ./data/journal \
    ./data/log \
    ./data/config \
  ; do \
    mkdir -p "$path"; \
  done

RUN mkdir -p /usr/share/graylog/data/config/certs/

COPY resources/cert.pem /usr/share/graylog/data/config/certs/cert.pem
COPY resources/pkcs8-plain.pem /usr/share/graylog/data/config/certs/pkcs8-plain.pem
COPY resources/graylog-ca.pem /usr/share/graylog/data/config/certs/graylog-ca.pem
COPY resources/graylog-ca.key.pem /usr/share/graylog/data/config/certs/graylog-ca.key.pem

RUN chown 1100:1100 -R /usr/share/graylog/data/config/certs/

RUN keytool -importcert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit -alias graylog-self-signed -file /usr/share/graylog/data/config/certs/cert.pem -noprompt
RUN keytool -importcert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit -alias graylog-self-signed-input -file /usr/share/graylog/data/config/certs/graylog-ca.pem -noprompt

COPY config ./data/config

VOLUME /usr/share/graylog/data

COPY docker-entrypoint.sh /

EXPOSE 9000
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["graylog"]

(Jochen) #4

Are these self-signed certificates or something “official” from a well-known CA?
If they’re self-signed, you need to add them to the JVM’s trust store: http://docs.graylog.org/en/2.2/pages/configuration/https.html#adding-a-self-signed-certificate-to-the-jvm-trust-store

Also, what interface is Graylog binding the REST API to? The complete URL is printed on startup.


(Andy) #5

Yes they are self signed, at least for now. And I did add them to the truststore. This is done at build time (see Dockerfile).

These are the ports that are opened inside of the container right now.


5140 and 11111 are Inputs.


(Jochen) #6

Please also answer this question.


(Andy) #7

This is what the logs show on startup

2017-06-28 11:49:34,141 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <https://0.0.0.0:9000/api/>
2017-06-28 11:49:34,141 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <https://0.0.0.0:9000/>

Port 9300 and 9350 are Elasticsearch Ports.
Port 36403 isn’t mentioned in the log file.


(Jochen) #8

Try setting rest_transport_uri and web_endpoint_uri to the correct host name, i. e. whatever is used as CN in your certificates.

For example, if your certificates have example.com as Common Name (CN), use https://example.com:9000/api/as URI.


(Andy) #9

Well I did not specify a CN in the certificate. Could that really be the issue? I mean everything else is running smoothly.


(Jochen) #10

Since the Common Name (CN) is mandatory, I wonder how you’ve created the certificates.

Yes, definitely.


(Andy) #11

Strange, I just used this command from the mentioned how-to:
sudo openssl req -x509 -days 365 -nodes -newkey rsa:2048 -keyout pkcs5-plain.pem -out cert.pem
openssl is version OpenSSL 1.0.2g

It did ask for a CN, but it wasn’t mandatory. Should I try generating new ones and proceed as I did with the current ones?


(Jochen) #12

Create new certificates with the proper Common Name (CN) you want to use (it should be the domain name part of the URL you want to use for Graylog).


(Andy) #13

So I created some new certs and tested with them. If I don’t change my config it is still the same as before.
So then I changed:
web_listen_uri = https://0.0.0.0:9000/ to https://graylog.example.com:9000/
rest_transport_uri = https://0.0.0.0:9000/api/ to https://graylog.example.com:9000/api
I ended up with this error:
2017-06-30 14:39:40,450 ERROR: org.graylog2.bootstrap.CmdLineTool - Invalid configuration com.github.joschi.jadconfig.ValidationException: Wildcard IP addresses cannot be used if the Graylog REST API and web interface listen on the same port.

So I changed rest_listen_uri = https://0.0.0.0:9000/api/ to https://graylog.example.com:9000/api

After that I got several errors including:
2017-06-30 14:42:30,929 ERROR: org.graylog2.bootstrap.ServerBootstrap - Graylog startup failed. Exiting. Exception was: java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]} at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:742) ~[graylog.jar:?] at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:555) ~[graylog.jar:?] at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:304) ~[graylog.jar:?] at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:147) [graylog.jar:?] at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?] at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]

I must say that rest_transport_uri was uncommented before these tests.
I changed everything back to 0.0.0.0:9000, now it runs as it did before but with some minor errors: 2017-06-30 14:53:25,614 WARN : org.graylog2.plugin.BaseConfiguration - "https://0.0.0.0:9000/api/" is not a valid setting for "rest_transport_uri". Using default [https://172.21.0.4:9000/api/].

2017-06-30 14:49:57,169 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://172.21.0.4:9000/api/system/metrics/multiple on node <2e5b7b01-4bbb-4c31-9d58-609a26bcf698> javax.net.ssl.SSLPeerUnverifiedException: Hostname 172.21.0.4 not verified:

The last one sounds plausible because the cn is graylog.example.com and not the internal container IP 172.21.0.4

I am kinda running out of ideas.

Thanks for your help so far.


(Jochen) #14

Try the following settings:

# Listen on all network interfaces
rest_listen_uri = https://0.0.0.0:9000/api/
web_listen_uri = https://0.0.0.0:9000/

# Publish the correct URI of the Graylog REST API of this Graylog node
rest_transport_uri = https://graylog.example.com:9000/api/

(Andy) #15

Hi,

I changed the config as suggested. It is still the same as before, Inputs appear as if they are not running and the Input counter shows 0 incoming messages, although there are messages incoming. But now I only get two WARN messages in the logs

2017-07-03 08:02:57,438 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog.example.com:9000/api/system/inputstates on node <2e5b7b01-4bbb-4c31-9d58-609a26bcf698>
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?`
`2017-07-03 08:02:57,150 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog.example.com:9000/api/system/metrics/multiple on node <2e5b7b01-4bbb-4c31-9d58-609a26bcf698>
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

(Jochen) #16

What’s the output of the following command:

curl -k -v -i https://graylog.example.com:9000/api/

On a side note: Please use proper formatting for your text snippets. This will make reading your posts much easier on the eye:

```
Text
```

(Andy) #17

This is the output. I think that something is off with the TLS config.

$ curl -k -v -i https://graylog.example.com:9000/api/
*   Trying 10.8.0.43...
* Connected to graylog.example.com (10.8.0.43) port 9000 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 694 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: An unexpected TLS packet was received.
* Closing connection 0
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

(Jochen) #18

So obviously the HTTP interface isn’t running with TLS (HTTPS).

Maybe you forgot the web_enable_tls and rest_enable_tls settings?


(Andy) #19

Nope, both settings are activated.
Could this setting from the docker-compose file come into play?

GRAYLOG_WEB_ENDPOINT_URI: https://10.8.0.43:9001/api

I have to do the remapping ot the port since there is another container running which already uses port 9000.


(Jochen) #20

What’s the (complete) log output of Graylog when you’re starting the container?

Maybe you should’ve run the curl command against the correct port then…