Graylog https access issue

Hi folks,

I have followed below article for https configuration. I was able to generate and apply the certificates, however when i restart the graylog service, i get the errors and i am not able to access graylog portal.

i am able to resolve the graylog fqdn both locally and remotely, port 443 is open on the firewall too.

Any insight what i might be doing wrong

certificate
https://github.com/3vi1john/graylog

sudo openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -keyout pkcs5-plain.pem -out cert.pem

sudo openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

sudo openssl pkcs8 -in pkcs5-plain.pem -topk8 -v2 des3 -out pkcs8-encrypted.pem -passout pass:'changeit'

sudo keytool -list -v -keystore keystore.jks -alias graylog.seacoglobal.com

sudo keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12

sudo openssl pkcs12 -in keystore.p12 -nokeys -out graylog-certificate.pem

sudo openssl pkcs12 -in keystore.p12 -nocerts -out graylog-pkcs5.pem

sudo openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem
2018-03-07T03:44:28.656Z INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-03-07T03:44:28.657Z INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.3 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-03-07T03:44:28.667Z INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-03-07T03:44:28.668Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-03-07T03:44:28.860Z ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Couldn't run validator method
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:227) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:100) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:177) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        ... 5 more
Caused by: java.lang.RuntimeException: Unable to resolve hostnames of rest/web listen uris:
        at org.graylog2.plugin.BaseConfiguration.isRestAndWebOnSamePort(BaseConfiguration.java:385) ~[graylog.jar:?]
        at org.graylog2.plugin.BaseConfiguration.validateWebTlsConfig(BaseConfiguration.java:454) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        ... 5 more
Caused by: java.net.UnknownHostException: graylog.seacolgobal.com: Name or service not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_161]
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[?:1.8.0_161]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
        at java.net.InetAddress.getByName(InetAddress.java:1076) ~[?:1.8.0_161]
        at org.graylog2.plugin.BaseConfiguration.isRestAndWebOnSamePort(BaseConfiguration.java:381) ~[graylog.jar:?]
        at org.graylog2.plugin.BaseConfiguration.validateWebTlsConfig(BaseConfiguration.java:454) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        ... 5 more

This should be quite self-explaining. Make sure that graylog.seacolgobal.com on the machine(s) running Graylog, if you want to use it for rest_listen_uri or web_listen_uri.

yes, the fqdn was defined in the server.cng file, also i added the fqdn in local host file. On the centos machine, i can locally ping the name and it resolves to IP.

i have followed the doc, and defined both api and web, or shall i just try first with the web. and if i try with the api, then do i need to add the cert to nxlog nodes?

What’s the result of the following commands on the machine running Graylog?

# host graylog.seacolgobal.com
# dig graylog.seacolgobal.com
# nslookup graylog.seacolgobal.com

All looks ok to me.
not sure, why it’s complaining about not able to get the host resolved.

Have you been trying with this host name? Unfortunately you decided to blur out the host name from your screenshot.

yes, that’s the host name. but you can see it’s getting resolved.

Are you sure that there’s not typo in your configuration?

Because if the system can resolve the host name, so can Graylog running on that system.

i am going to give it one more shot and try again. Also, i have followed the doc, and defined both api and web, or shall i just try first with the web. and if i try with the api, then do i need to add the cert to nxlog nodes?

so when i access the gray log, i would use https, instead of http, even though, the web listener uri is set to http ?

hey jochen, the https worked like a charm now. sometimes you just need to be working as team to get things moving :slight_smile:

and now, collectors disappeared,

2018-03-13T07:41:33.379Z WARN  [ProxiedResource] Unable to call https://graylog.domain.com:9000/api/system/metrics/multiple on node <760cc826-3cbd-49a6-95f3-8a701af1c70a>
javax.net.ssl.SSLPeerUnverifiedException: Hostname graylog.domain.com not verified:
    certificate: sha256/yH9nR/MT3Rohi1zFGkDIadLFfzyLQJdFlse5D0tgxPo=
    DN: CN=Navdeep Singh, OU=IT, O=company, L=Singapore, ST=Singapore, C=SG
    subjectAltNames: []
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:308) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:268) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160) ~[graylog.jar:?]
        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256) ~[graylog.jar:?]
        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134) ~[graylog.jar:?]
        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
        at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

The Common Name (CN) and subjectAltNames have to match your host name(s), as described in our documentation:
http://docs.graylog.org/en/2.4/pages/configuration/https.html#creating-a-self-signed-private-key-certificate

ok, do we need to make changes else where in the configuration for nodes running nxlog to connect to graylog server once tls is enabled

Please refer to the Graylog documentation:

I checked those documents, they don’t mention anything specifically for collector/nxlog config.

@jochen, is there any relation ship between enabling https on graylog and nxlog on plain tcp? Does both have to be ssl enabled or can i leave nxlog to run without on im_tcp instead of adding ssl.

based on my tests, once i enable ssl on graylog, graylog no longer sees nxlog nodes.

If you’re requiring TLS on your inputs, NXLOG also has to send its data over TLS.
If you’re using HTTPS for the Graylog REST API, the Graylog Collector Sidecar also has to use HTTPS to communicate with the Graylog REST API.

So i see, nxlog and sidecar are 2 components. Is it correct that i need to enable TLS for sidecar collector and optional for nxlog?

where can i find some info specific to sidecar tls?

looking at the documentation

http://docs.graylog.org/en/2.4/pages/collector_sidecar.html

It seems like sidecar and nxlog both have to be configured for TLS

Secure Sidecar Communication

The Communication between Sidecar and Graylog will be secured if your API use SSL. [we can’t selectively enable web tls and leave out rest api. as per the documentation this would cause issues]

To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input. Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support and Insecure TLS connection. After this is saved, the communication between Beats and Graylog will use TLS.

If you prefer NXLog you need to mark Allow untrusted certificate in the NXLog Outputs configuration and Enable TLS for your GELF Input.