Can not search and does not display input and sources statistic with pipeline

I have a pipeline with a little high speed message throughput. but when i create a pipeline with some rules and stages before start to send messages to the graylog cluster, it does output to ES, but cannot search from graylog web interface and also there is no input statistic and sources statistic available.

but if I delete some stages to left a very simple stage, the statistics and search comes back. and then i add those stages back is also OK. but if i rotate the index set, it will fail again. currently my throughtput should can be supported by my graylog cluster as following buffer usage statistic:

what should i investigated in?

when i looking into server.log of graylog, there is a error report when rotate the index set:

2018-04-08T11:11:50.228+08:00 ERROR [SystemJobManager] Unhandled error while running SystemJob <8289ca70-3ada-11e8-9b6b-525400ac7795> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
java.lang.IllegalArgumentException: Cat response did not contain a JSON Array
	at io.searchbox.core.Cat.parseResponseBody(Cat.java:61) ~[graylog.jar:?]
	at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:212) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.catIndices(Indices.java:534) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:498) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.isClosed(Indices.java:519) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob.execute(SetIndexReadOnlyAndCalculateRangeJob.java:58) ~[graylog.jar:?]
	at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

Which version of Elasticsearch are you using?

5.6.8-1 on centos 7.4 with search guard 5.6.8-19 community feature.

Does it work without SearchGuard?

never try it as i have enable TLS between Graylog and ES/MongoDB. Anyway I can try it without TLS between Graylog and ES…

did you have any outputs running (not asking for the default to Elasticsearch Output)

No. only to index set

when using without search guard, the problem is same.

in the graylog server.log, there remain has the following error reported:

2018-04-10T15:23:06.123+08:00 ERROR [SystemJobManager] Unhandled error while running SystemJob <f14c00c0-3c8f-11e8-a7aa-525400447578> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
java.lang.IllegalArgumentException: Cat response did not contain a JSON Array
	at io.searchbox.core.Cat.parseResponseBody(Cat.java:61) ~[graylog.jar:?]
	at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:212) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.catIndices(Indices.java:534) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:498) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.isClosed(Indices.java:519) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob.execute(SetIndexReadOnlyAndCalculateRangeJob.java:58) ~[graylog.jar:?]
	at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

in the ES there are some error reported:

cat graylog_deprecation.log
[2018-04-10T15:14:36,963][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-10T15:14:36,965][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-10T15:14:36,965][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]

What ES Version did you have running?

ES 5.6.8-1 on CentOS 7.4

more tests shows that it was the to_ip function lead to the problem of not to display the input and source statistic info and cannot search.

when i using a rule like:

rule "IPv6 address is well-formatted"
when
	has_field("ipv6_input") &&
    (	
		regex("^((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){7}([0-9a-fA-F]{1,4})$",to_string($message.ipv6_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,5}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,4}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}){1,3}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){5}(:[0-9a-fA-F]{1,4}){1,2}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}(:[0-9a-fA-F]{1,4})$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,6}:$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){0,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}):((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,2}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,3}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,5}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
	)
then
	set_field("ipv6_address",to_ip($message.ipv6_address_input));
end

this problem always happen after manually rotate the index set if comments out the set_field with to_ip function the statistic and search comes again…i.e.

rule "IPv6 address is well-formatted"
when
	has_field("ipv6_input") &&
    (	
		regex("^((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){7}([0-9a-fA-F]{1,4})$",to_string($message.ipv6_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,5}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,4}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}){1,3}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){5}(:[0-9a-fA-F]{1,4}){1,2}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}(:[0-9a-fA-F]{1,4})$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,6}:$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){0,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}):((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,2}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,3}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,5}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
	)
then
//	set_field("ipv6_address",to_ip($message.ipv6_address_input));
end

as a workaround, can we just set the field to a string which including a well-formed IPv4 or IPv6 address notation, and let Graylog output it to a ES ip datatype field?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.