I have followed below article for https configuration. I was able to generate and apply the certificates, however when i restart the graylog service, i get the errors and i am not able to access graylog portal.
i am able to resolve the graylog fqdn both locally and remotely, port 443 is open on the firewall too.
2018-03-07T03:44:28.656Z INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-03-07T03:44:28.657Z INFO [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.3 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-03-07T03:44:28.667Z INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-03-07T03:44:28.668Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-03-07T03:44:28.860Z ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Couldn't run validator method
at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:227) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:100) ~[graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:177) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
... 5 more
Caused by: java.lang.RuntimeException: Unable to resolve hostnames of rest/web listen uris:
at org.graylog2.plugin.BaseConfiguration.isRestAndWebOnSamePort(BaseConfiguration.java:385) ~[graylog.jar:?]
at org.graylog2.plugin.BaseConfiguration.validateWebTlsConfig(BaseConfiguration.java:454) ~[graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
... 5 more
Caused by: java.net.UnknownHostException: graylog.seacolgobal.com: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_161]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[?:1.8.0_161]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[?:1.8.0_161]
at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[?:1.8.0_161]
at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
at java.net.InetAddress.getByName(InetAddress.java:1076) ~[?:1.8.0_161]
at org.graylog2.plugin.BaseConfiguration.isRestAndWebOnSamePort(BaseConfiguration.java:381) ~[graylog.jar:?]
at org.graylog2.plugin.BaseConfiguration.validateWebTlsConfig(BaseConfiguration.java:454) ~[graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
... 5 more
This should be quite self-explaining. Make sure that graylog.seacolgobal.com on the machine(s) running Graylog, if you want to use it for rest_listen_uri or web_listen_uri.
yes, the fqdn was defined in the server.cng file, also i added the fqdn in local host file. On the centos machine, i can locally ping the name and it resolves to IP.
i have followed the doc, and defined both api and web, or shall i just try first with the web. and if i try with the api, then do i need to add the cert to nxlog nodes?
i am going to give it one more shot and try again. Also, i have followed the doc, and defined both api and web, or shall i just try first with the web. and if i try with the api, then do i need to add the cert to nxlog nodes?
2018-03-13T07:41:33.379Z WARN [ProxiedResource] Unable to call https://graylog.domain.com:9000/api/system/metrics/multiple on node <760cc826-3cbd-49a6-95f3-8a701af1c70a>
javax.net.ssl.SSLPeerUnverifiedException: Hostname graylog.domain.com not verified:
certificate: sha256/yH9nR/MT3Rohi1zFGkDIadLFfzyLQJdFlse5D0tgxPo=
DN: CN=Navdeep Singh, OU=IT, O=company, L=Singapore, ST=Singapore, C=SG
subjectAltNames: []
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:308) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:268) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:160) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
@jochen, is there any relation ship between enabling https on graylog and nxlog on plain tcp? Does both have to be ssl enabled or can i leave nxlog to run without on im_tcp instead of adding ssl.
based on my tests, once i enable ssl on graylog, graylog no longer sees nxlog nodes.
If you’re requiring TLS on your inputs, NXLOG also has to send its data over TLS.
If you’re using HTTPS for the Graylog REST API, the Graylog Collector Sidecar also has to use HTTPS to communicate with the Graylog REST API.
It seems like sidecar and nxlog both have to be configured for TLS
Secure Sidecar Communication
The Communication between Sidecar and Graylog will be secured if your API use SSL. [we can’t selectively enable web tls and leave out rest api. as per the documentation this would cause issues]
To secure the communication between the Collector and Graylog you just need to mark Enable TLS in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input. Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support and Insecure TLS connection. After this is saved, the communication between Beats and Graylog will use TLS.
If you prefer NXLog you need to mark Allow untrusted certificate in the NXLog Outputs configuration and Enable TLS for your GELF Input.