Hello,
I have an issue that I could not find anywhere else.
Below is a message I receive for a stacktrace. It originates from a docker log output that is forwarded to syslog.
The syslog is in its turn forwarded by filebeat to Graylog.
That all works fine, the issue arises when trying to extract the message.
First, filebeat reads the syslog file and uses the document_type: syslog. This did not seem to change anything.
Second, I tried to extract the fields using GROK and here it falls flat.
%{SYSLOGBASE} %{GREEDYDATA:syslog_message_content}
will only return the first line. The syslog fields exctract fine but I’m interested in the message here.
[2017-04-27 12:16:01.418] ERROR consul updater System.err com.orbitz.consul.ConsulException: Consul request failed
(?s)%{SYSLOGBASE} %{GREEDYDATA:syslog_message_content}
will return multiple lines but only extracts from the first line. It also removes all newlines resulting in an unreadable blob of text.
[2017-04-27 12:16:01.418] ERROR consul updater System.err com.orbitz.consul.ConsulException: Consul request failed [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.util.Http.extractConsulResponse(Http.java:54) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.HealthClient.getChecksByState(HealthClient.java:241) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.HealthClient.getChecksByState(HealthClient.java:225) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at be.test.testapp.consul.ConsulBasedApplicationM$Updater.run(ConsulBasedApplicationM.java:85) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at java.lang.Thread.run(Thread.java:745) [2017-04-27 12:16:01.418] ERROR consul updater System.err Caused by: java.net.SocketTimeoutException: timeout [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.Okio$3.newTimeoutException(Okio.java:212) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout.exit(AsyncTimeout.java:288) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:242) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:184) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:125) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:775) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.access$200(HttpEngine.java:86) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:760) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.readResponse(HttpEngine.java:613) [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.getResponse(RealCall.java:244) [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201) [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163) [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.execute(RealCall.java:57) [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.retrofit.OkHttpCall.execute(OkHttpCall.java:174) [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.consul.util.Http.extractConsulResponse(Http.java:52) [2017-04-27 12:16:01.419] ERROR consul updater System.err #011... 4 more [2017-04-27 12:16:01.420] ERROR consul updater System.err Caused by: java.net.SocketException: Socket closed [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at java.net.SocketInputStream.read(SocketInputStream.java:204) [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at java.net.SocketInputStream.read(SocketInputStream.java:141) [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at com.orbitz.okio.Okio$2.read(Okio.java:140) [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:238) [2017-04-27 12:16:01.420] ERROR consul updater System.err #011... 19 more
Here is the original message:
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err com.orbitz.consul.ConsulException: Consul request failed
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.util.Http.extractConsulResponse(Http.java:54)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.HealthClient.getChecksByState(HealthClient.java:241)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.HealthClient.getChecksByState(HealthClient.java:225)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at be.test.testapp.consul.ConsulBasedApplicationM$Updater.run(ConsulBasedApplicationM.java:85)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at java.lang.Thread.run(Thread.java:745)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err Caused by: java.net.SocketTimeoutException: timeout
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.Okio$3.newTimeoutException(Okio.java:212)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout.exit(AsyncTimeout.java:288)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:242)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:325)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:314)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:184)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:125)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:775)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.access$200(HttpEngine.java:86)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:760)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.readResponse(HttpEngine.java:613)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.getResponse(RealCall.java:244)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.execute(RealCall.java:57)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.retrofit.OkHttpCall.execute(OkHttpCall.java:174)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.consul.util.Http.extractConsulResponse(Http.java:52)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.419] ERROR consul updater System.err #011... 4 more
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.420] ERROR consul updater System.err Caused by: java.net.SocketException: Socket closed
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at java.net.SocketInputStream.read(SocketInputStream.java:204)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at java.net.SocketInputStream.read(SocketInputStream.java:141)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at com.orbitz.okio.Okio$2.read(Okio.java:140)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.420] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:238)
Apr 27 12:16:01 p4 testsystem[1618]: [2017-04-27 12:16:01.420] ERROR consul updater System.err #011... 19 more
Expected output:
[2017-04-27 12:16:01.418] ERROR consul updater System.err com.orbitz.consul.ConsulException: Consul request failed
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.util.Http.extractConsulResponse(Http.java:54)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.HealthClient.getChecksByState(HealthClient.java:241)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.consul.HealthClient.getChecksByState(HealthClient.java:225)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at be.test.testapp.consul.ConsulBasedApplicationM$Updater.run(ConsulBasedApplicationM.java:85)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at java.lang.Thread.run(Thread.java:745)
[2017-04-27 12:16:01.418] ERROR consul updater System.err Caused by: java.net.SocketTimeoutException: timeout
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.Okio$3.newTimeoutException(Okio.java:212)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout.exit(AsyncTimeout.java:288)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:242)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:325)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.indexOf(RealBufferedSource.java:314)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:184)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:125)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:775)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.access$200(HttpEngine.java:86)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:760)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.internal.http.HttpEngine.readResponse(HttpEngine.java:613)
[2017-04-27 12:16:01.418] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.getResponse(RealCall.java:244)
[2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201)
[2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
[2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.okhttp3.RealCall.execute(RealCall.java:57)
[2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.retrofit.OkHttpCall.execute(OkHttpCall.java:174)
[2017-04-27 12:16:01.419] ERROR consul updater System.err #011at com.orbitz.consul.util.Http.extractConsulResponse(Http.java:52)
[2017-04-27 12:16:01.419] ERROR consul updater System.err #011… 4 more
[2017-04-27 12:16:01.420] ERROR consul updater System.err Caused by: java.net.SocketException: Socket closed
[2017-04-27 12:16:01.420] ERROR consul updater System.err #011at java.net.SocketInputStream.read(SocketInputStream.java:204)
[2017-04-27 12:16:01.420] ERROR consul updater System.err #011at java.net.SocketInputStream.read(SocketInputStream.java:141)
[2017-04-27 12:16:01.420] ERROR consul updater System.err #011at com.orbitz.okio.Okio$2.read(Okio.java:140)
[2017-04-27 12:16:01.420] ERROR consul updater System.err #011at com.orbitz.okio.AsyncTimeout$2.read(AsyncTimeout.java:238)
[2017-04-27 12:16:01.420] ERROR consul updater System.err #011… 19 more