Graylog https access issue

v_2nas · March 12, 2018, 2:12am

Hi folks,

I have followed below article for https configuration. I was able to generate and apply the certificates, however when i restart the graylog service, i get the errors and i am not able to access graylog portal.

i am able to resolve the graylog fqdn both locally and remotely, port 443 is open on the firewall too.

Any insight what i might be doing wrong

certificate
https://github.com/3vi1john/graylog

sudo openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -keyout pkcs5-plain.pem -out cert.pem

sudo openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

sudo openssl pkcs8 -in pkcs5-plain.pem -topk8 -v2 des3 -out pkcs8-encrypted.pem -passout pass:'changeit'

sudo keytool -list -v -keystore keystore.jks -alias graylog.seacoglobal.com

sudo keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12

sudo openssl pkcs12 -in keystore.p12 -nokeys -out graylog-certificate.pem

sudo openssl pkcs12 -in keystore.p12 -nocerts -out graylog-pkcs5.pem

sudo openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

2018-03-07T03:44:28.656Z INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-03-07T03:44:28.657Z INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.3 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-03-07T03:44:28.667Z INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-03-07T03:44:28.668Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-03-07T03:44:28.860Z ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Couldn't run validator method
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:227) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:100) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:177) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        ... 5 more
Caused by: java.lang.RuntimeException: Unable to resolve hostnames of rest/web listen uris:
        at org.graylog2.plugin.BaseConfiguration.isRestAndWebOnSamePort(BaseConfiguration.java:385) ~[graylog.jar:?]
        at org.graylog2.plugin.BaseConfiguration.validateWebTlsConfig(BaseConfiguration.java:454) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        ... 5 more
Caused by: java.net.UnknownHostException: graylog.seacolgobal.com: Name or service not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_161]
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[?:1.8.0_161]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
        at java.net.InetAddress.getByName(InetAddress.java:1076) ~[?:1.8.0_161]
        at org.graylog2.plugin.BaseConfiguration.isRestAndWebOnSamePort(BaseConfiguration.java:381) ~[graylog.jar:?]
        at org.graylog2.plugin.BaseConfiguration.validateWebTlsConfig(BaseConfiguration.java:454) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        ... 5 more

jochen · March 12, 2018, 7:48am

This should be quite self-explaining. Make sure that graylog.seacolgobal.com on the machine(s) running Graylog, if you want to use it for rest_listen_uri or web_listen_uri.

v_2nas · March 12, 2018, 8:08am

yes, the fqdn was defined in the server.cng file, also i added the fqdn in local host file. On the centos machine, i can locally ping the name and it resolves to IP.

i have followed the doc, and defined both api and web, or shall i just try first with the web. and if i try with the api, then do i need to add the cert to nxlog nodes?

jochen · March 12, 2018, 8:26am

What’s the result of the following commands on the machine running Graylog?

# host graylog.seacolgobal.com
# dig graylog.seacolgobal.com
# nslookup graylog.seacolgobal.com

v_2nas · March 13, 2018, 1:10am

All looks ok to me.
not sure, why it’s complaining about not able to get the host resolved.

jochen · March 13, 2018, 7:16am

Have you been trying with this host name? Unfortunately you decided to blur out the host name from your screenshot.

v_2nas · March 13, 2018, 7:20am

yes, that’s the host name. but you can see it’s getting resolved.

jochen · March 13, 2018, 7:23am

Are you sure that there’s not typo in your configuration?

Because if the system can resolve the host name, so can Graylog running on that system.

v_2nas · March 13, 2018, 7:29am

i am going to give it one more shot and try again. Also, i have followed the doc, and defined both api and web, or shall i just try first with the web. and if i try with the api, then do i need to add the cert to nxlog nodes?

v_2nas · March 13, 2018, 7:36am

so when i access the gray log, i would use https, instead of http, even though, the web listener uri is set to http ?

v_2nas · March 13, 2018, 7:40am

hey jochen, the https worked like a charm now. sometimes you just need to be working as team to get things moving

v_2nas · March 13, 2018, 7:42am

and now, collectors disappeared,

2018-03-13T07:41:33.379Z WARN  [ProxiedResource] Unable to call https://graylog.domain.com:9000/api/system/metrics/multiple on node <760cc826-3cbd-49a6-95f3-8a701af1c70a>
javax.net.ssl.SSLPeerUnverifiedException: Hostname graylog.domain.com not verified:
    certificate: sha256/yH9nR/MT3Rohi1zFGkDIadLFfzyLQJdFlse5D0tgxPo=
    DN: CN=Navdeep Singh, OU=IT, O=company, L=Singapore, ST=Singapore, C=SG
    subjectAltNames: []
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:308) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:268) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160) ~[graylog.jar:?]
        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256) ~[graylog.jar:?]
        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134) ~[graylog.jar:?]
        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
        at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

jochen · March 13, 2018, 8:15am

v_2nas:

javax.net.ssl.SSLPeerUnverifiedException: Hostname graylog.domain.com not verified:
    certificate: sha256/yH9nR/MT3Rohi1zFGkDIadLFfzyLQJdFlse5D0tgxPo=
    DN: CN=Navdeep Singh, OU=IT, O=company, L=Singapore, ST=Singapore, C=SG
    subjectAltNames: []

The Common Name (CN) and subjectAltNames have to match your host name(s), as described in our documentation:
http://docs.graylog.org/en/2.4/pages/configuration/https.html#creating-a-self-signed-private-key-certificate

v_2nas · March 13, 2018, 8:17am

ok, do we need to make changes else where in the configuration for nodes running nxlog to connect to graylog server once tls is enabled

jochen · March 13, 2018, 8:19am

Please refer to the Graylog documentation:

v_2nas · March 13, 2018, 8:24am

I checked those documents, they don’t mention anything specifically for collector/nxlog config.

v_2nas · March 14, 2018, 3:06am

@jochen, is there any relation ship between enabling https on graylog and nxlog on plain tcp? Does both have to be ssl enabled or can i leave nxlog to run without on im_tcp instead of adding ssl.

based on my tests, once i enable ssl on graylog, graylog no longer sees nxlog nodes.

jochen · March 14, 2018, 8:15am

If you’re requiring TLS on your inputs, NXLOG also has to send its data over TLS.
If you’re using HTTPS for the Graylog REST API, the Graylog Collector Sidecar also has to use HTTPS to communicate with the Graylog REST API.

v_2nas · March 14, 2018, 8:56am

So i see, nxlog and sidecar are 2 components. Is it correct that i need to enable TLS for sidecar collector and optional for nxlog?

where can i find some info specific to sidecar tls?

v_2nas · March 14, 2018, 9:03am

looking at the documentation

http://docs.graylog.org/en/2.4/pages/collector_sidecar.html

It seems like sidecar and nxlog both have to be configured for TLS

Secure Sidecar Communication

The Communication between Sidecar and Graylog will be secured if your API use SSL. [we can’t selectively enable web tls and leave out rest api. as per the documentation this would cause issues]

To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input. Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support and Insecure TLS connection. After this is saved, the communication between Beats and Graylog will use TLS.

If you prefer NXLog you need to mark Allow untrusted certificate in the NXLog Outputs configuration and Enable TLS for your GELF Input.

Topic		Replies	Views
Graylog 3.2 HTTPS private key issue Graylog Central (peer support)	4	2167	May 29, 2020
Configuration HTTPS Graylog Central (peer support)	17	4180	April 26, 2018
Help please with using HTTPS on clustered Graylog Graylog Central (peer support)	6	4115	April 4, 2019
Secure Multi-node environment Graylog Central (peer support)	2	685	December 2, 2020
Graylog 3.0.2 https Graylog Central (peer support) sidecar	11	4352	September 2, 2019

Graylog https access issue

Related topics