Received Messages continuously loading

martinwoodger · 2018-12-20 09:48:32 UTC

Hi I’m new to graylog but have been trying hard to get it working first on Ubuntu 18.04 which constantly fails to install graylog 2.4.6. I will log this separately. Now I am on Centos 7 and everything has installed without problem. Except our syslog input from a Sophos firewall shows as continuously Loading in the Search window. I believe I should be seeing the raw text as no extractors have been applied. At one point I was getting an address error FIELDDATA data is too large and I have upped the “indices.breaker.fielddata.limit” to 95%. Now the errors are as below:

I would be grateful if someone can make sense of this as I need to get this working.

/var/log/elasticsearch/graylog.log

[2018-12-20 09:22:41,096][DEBUG][action.search ] [Solo] All shards failed for phase: [query]
RemoteTransportException[[Solo][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: IllegalArgumentException[Expected numeric type on field [timestamp], but got [string]];

/var/log/graylog-server/Graylog-Server.log

2018-12-20T09:32:56.260Z ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=3b64d930-043a-11e9-b8d7-00505685f117, journalOffset=50186527, codec=syslog, payloadSize=787, timestamp=2018-12-20T09:32:56.259Z, remoteAddress=/192.168.225.225:61142} on input <5c08006854723c5946dd7071>.
2018-12-20T09:32:56.260Z ERROR [DecodingProcessor] Error processing message RawMessage{id=3b64d930-043a-11e9-b8d7-00505685f117, journalOffset=50186527, codec=syslog, payloadSize=787, timestamp=2018-12-20T09:32:56.259Z, remoteAddress=/192.168.225.225:61142}
java.lang.IllegalArgumentException: Invalid format: “2018:12:20-09:32:56” is malformed at “:12:20-09:32:56”
at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945) ~[graylog.jar:?]
at org.joda.time.DateTime.parse(DateTime.java:160) ~[graylog.jar:?]
at org.joda.time.DateTime.parse(DateTime.java:149) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.parseDate(SyslogServerEvent.java:108) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.parsePriority(SyslogServerEvent.java:136) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.parse(SyslogServerEvent.java:152) ~[graylog.jar:?]
at org.graylog2.syslog4j.server.impl.event.SyslogServerEvent.(SyslogServerEvent.java:50) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.SyslogCodec.parse(SyslogCodec.java:132) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.SyslogCodec.decode(SyslogCodec.java:96) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

I should mention I am now on the alpha version 3 of Graylog having tried 2.4.6 and got the same results.

jan · 2018-12-20 10:08:17 UTC

it looks likes you have multiple issues.

Invalid format: “2018:12:20-09:32:56” is malformed at “:12:20-09:32:56”

The firewall does not send valid timestamps for a syslog message. Change the input to RAW and parse the message yourself or fix the timestamps on the firewall device.

The current stable version is Graylog 2.5.1 (at time of writing this).

martinwoodger · 2018-12-20 10:13:36 UTC

Thanks I did find something online about using raw input but I’m not sure I have set it as syslog anywhere. Is there a setting on the edit input screen? I will revert back to 2.5.1 but was getting desperate I should have tried this first.

jan · 2018-12-20 10:31:42 UTC

http://docs.graylog.org/en/2.5/pages/sending_data.html#raw-plaintext-inputs

RAW is like netcat - just receive anything that is send.

martinwoodger · 2018-12-20 10:56:42 UTC

Thanks I recreated the input as raw and it does look healthier I get an equal number of messages in and out but the search screen is still displaying loading?

martinwoodger · 2018-12-20 11:42:23 UTC

I restarted the graylog and elastic search services and for the first time it has stopped loading and I have an error:
cannot GET http://192.168.224.172:12900/search/universal/relative?query=*&range=300&limit=150&sort=timestamp%3Adesc (500)

It also says to check the logs:

/var/log/elasticsearch/graylog.log

[2018-12-20 11:37:04,502][DEBUG][action.search ] [N’Garai] All shards failed for phase: [query]
RemoteTransportException[[N’Garai][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: IllegalArgumentException[Expected numeric type on field [timestamp], but got [string]];
Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [timestamp], but got [string]
at org.elasticsearch.search.aggregations.support.AggregationContext.numericField(AggregationContext.java:161)
at org.elasticsearch.search.aggregations.support.AggregationContext.originalValuesSource(AggregationContext.java:145)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:85)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:60)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:87)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:85)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

The server.log in the graylog-server directory seems happy.

jan · 2018-12-20 13:05:13 UTC

What Elasticsearch Version did you use?

martinwoodger · 2018-12-20 13:08:43 UTC

Hi Jan, elasticsearch verson: 5.5.4

system · 2019-01-03 13:09:01 UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.