Hi I’m new to graylog but have been trying hard to get it working first on Ubuntu 18.04 which constantly fails to install graylog 2.4.6. I will log this separately. Now I am on Centos 7 and everything has installed without problem. Except our syslog input from a Sophos firewall shows as continuously Loading in the Search window. I believe I should be seeing the raw text as no extractors have been applied. At one point I was getting an address error FIELDDATA data is too large and I have upped the “indices.breaker.fielddata.limit” to 95%. Now the errors are as below:
I would be grateful if someone can make sense of this as I need to get this working.
/var/log/elasticsearch/graylog.log
[2018-12-20 09:22:41,096][DEBUG][action.search ] [Solo] All shards failed for phase: [query]
RemoteTransportException[[Solo][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: IllegalArgumentException[Expected numeric type on field [timestamp], but got [string]];
/var/log/graylog-server/Graylog-Server.log
2018-12-20T09:32:56.260Z ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=3b64d930-043a-11e9-b8d7-00505685f117, journalOffset=50186527, codec=syslog, payloadSize=787, timestamp=2018-12-20T09:32:56.259Z, remoteAddress=/192.168.225.225:61142} on input <5c08006854723c5946dd7071>.
2018-12-20T09:32:56.260Z ERROR [DecodingProcessor] Error processing message RawMessage{id=3b64d930-043a-11e9-b8d7-00505685f117, journalOffset=50186527, codec=syslog, payloadSize=787, timestamp=2018-12-20T09:32:56.259Z, remoteAddress=/192.168.225.225:61142}
java.lang.IllegalArgumentException: Invalid format: “2018:12:20-09:32:56” is malformed at “:12:20-09:32:56”
at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945) ~[graylog.jar:?]
at org.joda.time.DateTime.parse(DateTime.java:160) ~[graylog.jar:?]
at org.joda.time.DateTime.parse(DateTime.java:149) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.parseDate(SyslogServerEvent.java:108) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.parsePriority(SyslogServerEvent.java:136) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.parse(SyslogServerEvent.java:152) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.(SyslogServerEvent.java:50) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.SyslogCodec.parse(SyslogCodec.java:132) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.SyslogCodec.decode(SyslogCodec.java:96) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
I should mention I am now on the alpha version 3 of Graylog having tried 2.4.6 and got the same results.
Invalid format: “2018:12:20-09:32:56” is malformed at “:12:20-09:32:56”
The firewall does not send valid timestamps for a syslog message. Change the input to RAW and parse the message yourself or fix the timestamps on the firewall device.
The current stable version is Graylog 2.5.1 (at time of writing this).
Thanks I did find something online about using raw input but I’m not sure I have set it as syslog anywhere. Is there a setting on the edit input screen? I will revert back to 2.5.1 but was getting desperate I should have tried this first.
Thanks I recreated the input as raw and it does look healthier I get an equal number of messages in and out but the search screen is still displaying loading?
[2018-12-20 11:37:04,502][DEBUG][action.search ] [N’Garai] All shards failed for phase: [query]
RemoteTransportException[[N’Garai][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: IllegalArgumentException[Expected numeric type on field [timestamp], but got [string]];
Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [timestamp], but got [string]
at org.elasticsearch.search.aggregations.support.AggregationContext.numericField(AggregationContext.java:161)
at org.elasticsearch.search.aggregations.support.AggregationContext.originalValuesSource(AggregationContext.java:145)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:85)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:60)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:87)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:85)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The server.log in the graylog-server directory seems happy.
I should have tried this first.