Can not search and does not display input and sources statistic with pipeline


(Charles Deng) #1

I have a pipeline with a little high speed message throughput. but when i create a pipeline with some rules and stages before start to send messages to the graylog cluster, it does output to ES, but cannot search from graylog web interface and also there is no input statistic and sources statistic available.

but if I delete some stages to left a very simple stage, the statistics and search comes back. and then i add those stages back is also OK. but if i rotate the index set, it will fail again. currently my throughtput should can be supported by my graylog cluster as following buffer usage statistic:

what should i investigated in?


Errors report when rotate index set
(Charles Deng) #2

when i looking into server.log of graylog, there is a error report when rotate the index set:

2018-04-08T11:11:50.228+08:00 ERROR [SystemJobManager] Unhandled error while running SystemJob <8289ca70-3ada-11e8-9b6b-525400ac7795> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
java.lang.IllegalArgumentException: Cat response did not contain a JSON Array
	at io.searchbox.core.Cat.parseResponseBody(Cat.java:61) ~[graylog.jar:?]
	at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:212) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.catIndices(Indices.java:534) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:498) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.isClosed(Indices.java:519) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob.execute(SetIndexReadOnlyAndCalculateRangeJob.java:58) ~[graylog.jar:?]
	at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

(Jochen) #3

Which version of Elasticsearch are you using?


(Charles Deng) #4

5.6.8-1 on centos 7.4 with search guard 5.6.8-19 community feature.


(Jochen) #5

Does it work without SearchGuard?


(Charles Deng) #6

never try it as i have enable TLS between Graylog and ES/MongoDB. Anyway I can try it without TLS between Graylog and ES…


(Jan Doberstein) #7

did you have any outputs running (not asking for the default to Elasticsearch Output)


(Charles Deng) #8

No. only to index set


(Charles Deng) #9

when using without search guard, the problem is same.

in the graylog server.log, there remain has the following error reported:

2018-04-10T15:23:06.123+08:00 ERROR [SystemJobManager] Unhandled error while running SystemJob <f14c00c0-3c8f-11e8-a7aa-525400447578> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
java.lang.IllegalArgumentException: Cat response did not contain a JSON Array
	at io.searchbox.core.Cat.parseResponseBody(Cat.java:61) ~[graylog.jar:?]
	at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
	at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:212) ~[graylog.jar:?]
	at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:46) ~[graylog.jar:?]
	at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.catIndices(Indices.java:534) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:498) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.Indices.isClosed(Indices.java:519) ~[graylog.jar:?]
	at org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob.execute(SetIndexReadOnlyAndCalculateRangeJob.java:58) ~[graylog.jar:?]
	at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:235) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

in the ES there are some error reported:

cat graylog_deprecation.log
[2018-04-10T15:14:36,963][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-10T15:14:36,965][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]
[2018-04-10T15:14:36,965][WARN ][o.e.d.i.m.TypeParsers    ] Expected a boolean [true/false] for property [index] but got [not_analyzed]

(Jan Doberstein) #10

What ES Version did you have running?


(Charles Deng) #11

ES 5.6.8-1 on CentOS 7.4


(Charles Deng) #12

more tests shows that it was the to_ip function lead to the problem of not to display the input and source statistic info and cannot search.

when i using a rule like:

rule "IPv6 address is well-formatted"
when
	has_field("ipv6_input") &&
    (	
		regex("^((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){7}([0-9a-fA-F]{1,4})$",to_string($message.ipv6_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,5}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,4}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}){1,3}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){5}(:[0-9a-fA-F]{1,4}){1,2}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}(:[0-9a-fA-F]{1,4})$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,6}:$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){0,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}):((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,2}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,3}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,5}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
	)
then
	set_field("ipv6_address",to_ip($message.ipv6_address_input));
end

this problem always happen after manually rotate the index set if comments out the set_field with to_ip function the statistic and search comes again…i.e.

rule "IPv6 address is well-formatted"
when
	has_field("ipv6_input") &&
    (	
		regex("^((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){7}([0-9a-fA-F]{1,4})$",to_string($message.ipv6_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,6}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,5}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,4}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}){1,3}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){5}(:[0-9a-fA-F]{1,4}){1,2}$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}(:[0-9a-fA-F]{1,4})$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,6}:$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){6}((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^:(:[0-9a-fA-F]{1,4}){0,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){4}(:[0-9a-fA-F]{1,4}):((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){3}(:[0-9a-fA-F]{1,4}){1,2}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){2}(:[0-9a-fA-F]{1,4}){1,3}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:)(:[0-9a-fA-F]{1,4}){1,4}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
		|| regex("^([0-9a-fA-F]{1,4}:){1,5}:((([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))\\.){3}(([\\d])|([1-9][\\d])|(1[\\d]{2})|(2[0-4][\\d])|(25[0-5]))$",to_string($message.ipv6_address_input)).matches == true
	)
then
//	set_field("ipv6_address",to_ip($message.ipv6_address_input));
end

(Charles Deng) #13

as a workaround, can we just set the field to a string which including a well-formed IPv4 or IPv6 address notation, and let Graylog output it to a ES ip datatype field?


(system) #14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.