Elasticsearch exception reason=all shards failed

Uporaba · September 28, 2021, 7:29am

hi, in the new version 4.1.3, this error appears. How can I understand what this error is and remove it? I don’t see any errors in the logs.

but in graylog version 3.2.4 everything is searched and there are no errors:

UPD: In log i saw:

2021-09-28T07:50:04.217Z ERROR [MessagesAdapterES7] Failed to index [3] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[150]: index [graylog_57], type [_doc], id [b0dd3993-2030-11ec-84e8-001a0000005c], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [upstream_header_time] of type [float] in document with id 'b0dd3993-2030-11ec-84e8-001a0000005c'. Preview of field's value: '-']]; nested: ElasticsearchException[Elasticsearch exception [type=number_format_exception, reason=For input string: "-"]];]
[154]: index [graylog_57], type [_doc], id [b0dd3994-2030-11ec-84e8-001a0000005c], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [upstream_header_time] of type [float] in document with id 'b0dd3994-2030-11ec-84e8-001a0000005c'. Preview of field's value: '-']]; nested: ElasticsearchException[Elasticsearch exception [type=number_format_exception, reason=For input string: "-"]];]
[156]: index [graylog_57], type [_doc], id [b0dd3992-2030-11ec-84e8-001a0000005c], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [upstream_header_time] of type [float] in document with id 'b0dd3992-2030-11ec-84e8-001a0000005c'. Preview of field's value: '-']]; nested: ElasticsearchException[Elasticsearch exception [type=number_format_exception, reason=For input string: "-"]];]
2021-09-28T07:50:04.218Z ERROR [BlockingBatchedESOutput] Unable to flush message buffer
java.lang.ClassCastException: Cannot cast java.lang.String to org.joda.time.DateTime
        at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_302]
        at org.graylog2.plugin.Message.getFieldAs(Message.java:663) ~[graylog.jar:?]
        at org.graylog2.plugin.Message.getTimestamp(Message.java:367) ~[graylog.jar:?]
        at org.graylog2.indexer.messages.Messages$IndexingError.toIndexFailure(Messages.java:326) ~[graylog.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_302]
        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556) ~[?:1.8.0_302]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_302]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_302]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_302]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_302]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_302]
        at org.graylog2.indexer.messages.Messages.propagateFailure(Messages.java:281) ~[graylog.jar:?]
        at org.graylog2.indexer.messages.Messages.bulkIndexRequests(Messages.java:165) ~[graylog.jar:?]
        at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:143) ~[graylog.jar:?]
        at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:123) ~[graylog.jar:?]
        at org.graylog2.outputs.ElasticSearchOutput.writeMessageEntries(ElasticSearchOutput.java:103) ~[graylog.jar:?]
        at org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:117) [graylog.jar:?]
        at org.graylog2.outputs.BlockingBatchedESOutput.writeMessageEntry(BlockingBatchedESOutput.java:98) [graylog.jar:?]
        at org.graylog2.outputs.BlockingBatchedESOutput.write(BlockingBatchedESOutput.java:80) [graylog.jar:?]
        at org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:193) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_302]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_302]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]

gsmith · September 28, 2021, 10:42pm

Hello,

I’m not sure all what’s going on. Your post shows two different version of Graylog. Are these the same server? To help you further I need some more information here
.
It obvious that this is a Elasticsearch/shard issue and there are some changes in the two major version.
Below are a couple commands you can use to find out what is going on with your ES shards/indices.

Check the health of ES

curl -XGET http://localhost:9200/_cluster/health?pretty=true

Check your Shards

curl -XGET http://localhost:9200/_cat/shards

For a better explanation why there are errors you can execute the following.

ES Info

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty

Hope that helps

Uporaba · September 29, 2021, 6:43am

Hi, These are two different services. 4.X version is on another server with a new architecture, 3.X on another with an old one, which will close soon.

{ "cluster_name" : "graylog", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 490, "active_shards" : 490, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

Uporaba · September 29, 2021, 6:46am

dhcp_ind_55        3 p STARTED  127107   38.7mb IP es-node-02
dhcp_ind_55        1 p STARTED  126147   38.4mb IP es-node-01
dhcp_ind_55        2 p STARTED  127209   38.7mb IP es-node-03
dhcp_ind_55        0 p STARTED  127160   38.7mb IP es-node-02
k8s_49             3 p STARTED 3145290    1.2gb IP es-node-02
k8s_49             1 p STARTED 3145936    1.2gb IP es-node-01
k8s_49             2 p STARTED 3142848    1.2gb IP es-node-03
k8s_49             0 p STARTED 3146453    1.2gb IP es-node-02
nolim_13           3 p STARTED       0     208b IP es-node-01
nolim_13           1 p STARTED       0     208b IP es-node-03
nolim_13           2 p STARTED       0     208b IP es-node-02
nolim_13           0 p STARTED       0     208b IP es-node-01
dhcp_ind_51        3 p STARTED  125670   38.2mb IP es-node-03
dhcp_ind_51        1 p STARTED  126019   38.4mb IP es-node-02
dhcp_ind_51        2 p STARTED  125437   38.3mb IP es-node-01
dhcp_ind_51        0 p STARTED  125765   38.3mb IP es-node-03
3mon_2             3 p STARTED       0     208b IP es-node-01
3mon_2             1 p STARTED       0     208b IP es-node-02
3mon_2             2 p STARTED       0     208b IP es-node-01
3mon_2             0 p STARTED       0     208b IP es-node-03
3mon_5             3 p STARTED       0     208b IP es-node-03
3mon_5             1 p STARTED       0     208b IP es-node-01
3mon_5             2 p STARTED       0     208b IP es-node-02
3mon_5             0 p STARTED       0     208b IP es-node-03
gl-system-events_0 3 p STARTED       0     208b IP es-node-02
gl-system-events_0 1 p STARTED       0     208b IP es-node-01
gl-system-events_0 2 p STARTED       0     208b IP es-node-03
gl-system-events_0 0 p STARTED       0     208b IP es-node-02
3mon_0             3 p STARTED       0     208b IP es-node-03
3mon_0             1 p STARTED       0     208b IP es-node-02
3mon_0             2 p STARTED       0     208b IP es-node-01
3mon_0             0 p STARTED       0     208b IP es-node-03
k8s_56             3 p STARTED 2936629    1.2gb IP es-node-02
k8s_56             1 p STARTED 2935415    1.2gb IP es-node-03
k8s_56             2 p STARTED 2931887    1.2gb IP es-node-01
k8s_56             0 p STARTED 2932851    1.2gb IP es-node-03
nolim_30           3 p STARTED       0     208b IP es-node-03
nolim_30           1 p STARTED       0     208b IP es-node-02
nolim_30           2 p STARTED       0     208b IP es-node-01
nolim_30           0 p STARTED       0     208b IP es-node-03
3mon_16            3 p STARTED  539521  104.9mb IP es-node-03
3mon_16            1 p STARTED  538706  104.7mb IP es-node-01
3mon_16            2 p STARTED  538236  104.7mb IP es-node-02
3mon_16            0 p STARTED  538204  104.6mb IP es-node-03
dhcp_pv_52         1 p STARTED       0     208b IP es-node-03
dhcp_pv_52         2 p STARTED       0     208b IP es-node-02
dhcp_pv_52         0 p STARTED       0     208b IP es-node-01
3mon_30            3 p STARTED      80   11.9mb IP es-node-01
3mon_30            1 p STARTED      84   11.8mb IP es-node-03
3mon_30            2 p STARTED      78   11.9mb IP es-node-02
3mon_30            0 p STARTED      76   11.9mb IP es-node-01
k8s_60             3 p STARTED 2978918    1.2gb IP es-node-03
k8s_60             1 p STARTED 2980083    1.2gb IP es-node-02
k8s_60             2 p STARTED 2978351    1.2gb IP es-node-01
k8s_60             0 p STARTED 2981757    1.2gb IP es-node-03
www_61             3 p STARTED    1524  974.1kb IP es-node-01
www_61             1 p STARTED    1658      1mb IP es-node-03
www_61             2 p STARTED    1532    986kb IP es-node-02
www_61             0 p STARTED    1589 1019.8kb IP es-node-01
nolim_12           3 p STARTED       0     208b IP es-node-01
nolim_12           1 p STARTED       0     208b IP es-node-03
nolim_12           2 p STARTED       0     208b IP es-node-02
nolim_12           0 p STARTED       0     208b IP es-node-01
3mon_25            3 p STARTED  511553   99.5mb IP es-node-01
3mon_25            1 p STARTED  512114   99.6mb IP es-node-02
3mon_25            2 p STARTED  512843   99.7mb IP es-node-01
3mon_25            0 p STARTED  512849   99.7mb IP es-node-03
nolim_20           3 p STARTED       0     208b IP es-node-01
nolim_20           1 p STARTED       0     208b IP es-node-03
nolim_20           2 p STARTED       0     208b IP es-node-02
nolim_20           0 p STARTED       0     208b IP es-node-01
k8s_58             3 p STARTED 2229641  960.9mb IP es-node-01
k8s_58             1 p STARTED 2231253    962mb IP es-node-03
k8s_58             2 p STARTED 2230052  961.3mb IP es-node-02
k8s_58             0 p STARTED 2229054  960.7mb IP es-node-01
nolim_19           3 p STARTED       0     208b IP es-node-02
nolim_19           1 p STARTED       0     208b IP es-node-02
nolim_19           2 p STARTED       0     208b IP es-node-01
nolim_19           0 p STARTED       0     208b IP es-node-03
3mon_20            3 p STARTED  511586   98.3mb IP es-node-03
3mon_20            1 p STARTED  512609   98.4mb IP es-node-02
3mon_20            2 p STARTED  511416   98.2mb IP es-node-01
3mon_20            0 p STARTED  512523   98.4mb IP es-node-03
nolim_22           3 p STARTED       0     208b IP es-node-03
nolim_22           1 p STARTED       0     208b IP es-node-02
nolim_22           2 p STARTED       0     208b IP es-node-01
nolim_22           0 p STARTED       0     208b IP es-node-03
k8s_43             3 p STARTED     643  127.1kb IP es-node-03
k8s_43             1 p STARTED     623  127.1kb IP es-node-02
k8s_43             2 p STARTED     666  130.3kb IP es-node-01
k8s_43             0 p STARTED     612  121.1kb IP es-node-03
k8s_41             3 p STARTED     370   85.5kb IP es-node-02
k8s_41             1 p STARTED     376   87.8kb IP es-node-01
k8s_41             2 p STARTED     379   86.7kb IP es-node-03
k8s_41             0 p STARTED     376   87.5kb IP es-node-02
k8s_48             3 p STARTED 3085238    1.2gb IP es-node-02
k8s_48             1 p STARTED 3084854    1.2gb IP es-node-01
k8s_48             2 p STARTED 3085246    1.2gb IP es-node-03
k8s_48             0 p STARTED 3085924    1.2gb IP es-node-02
dhcp_ind_58        3 p STARTED  106151   32.5mb IP es-node-02
dhcp_ind_58        1 p STARTED  105711   32.3mb IP es-node-01
dhcp_ind_58        2 p STARTED  105742   32.4mb IP es-node-03
dhcp_ind_58        0 p STARTED  105575   32.3mb IP es-node-02
dhcp_ind_61        3 p STARTED   44640   14.1mb IP es-node-03
dhcp_ind_61        1 p STARTED   44482     14mb IP es-node-02
dhcp_ind_61        2 p STARTED   44429     14mb IP es-node-01
dhcp_ind_61        0 p STARTED   44568   14.1mb IP es-node-03
3mon_10            3 p STARTED  543682  105.8mb IP es-node-03
3mon_10            1 p STARTED  543225  105.7mb IP es-node-01
3mon_10            2 p STARTED  544351    106mb IP es-node-02
3mon_10            0 p STARTED  544258    106mb IP es-node-03
k8s_59             3 p STARTED 2618572    1.1gb IP es-node-03
k8s_59             1 p STARTED 2615842    1.1gb IP es-node-02
k8s_59             2 p STARTED 2619900    1.1gb IP es-node-01
k8s_59             0 p STARTED 2619081    1.1gb IP es-node-03
nolim_21           3 p STARTED       0     208b IP es-node-02
nolim_21           1 p STARTED       0     208b IP es-node-01
nolim_21           2 p STARTED       0     208b IP es-node-03
nolim_21           0 p STARTED       0     208b IP es-node-02
3mon_18            3 p STARTED  511414   98.8mb IP es-node-02
3mon_18            1 p STARTED  512250     99mb IP es-node-01
3mon_18            2 p STARTED  511369   98.8mb IP es-node-03
3mon_18            0 p STARTED  511194   98.7mb IP es-node-02
3mon_4             3 p STARTED       0     208b IP es-node-01
3mon_4             1 p STARTED       0     208b IP es-node-01
3mon_4             2 p STARTED       0     208b IP es-node-03
3mon_4             0 p STARTED       0     208b IP es-node-02
nolim_26           3 p STARTED       0     208b IP es-node-03
nolim_26           1 p STARTED       0     208b IP es-node-02
nolim_26           2 p STARTED       0     208b IP es-node-01
nolim_26           0 p STARTED       0     208b IP es-node-03
nolim_24           3 p STARTED       0     208b IP es-node-01
nolim_24           1 p STARTED       0     208b IP es-node-03
nolim_24           2 p STARTED       0     208b IP es-node-02
nolim_24           0 p STARTED       0     208b IP es-node-01
k8s_47             3 p STARTED 2697487    1.1gb IP es-node-01
k8s_47             1 p STARTED 2697846    1.1gb IP es-node-03
k8s_47             2 p STARTED 2700150    1.1gb IP es-node-02
k8s_47             0 p STARTED 2700105    1.1gb IP es-node-01
nolim_8            3 p STARTED       0     208b IP es-node-02
nolim_8            1 p STARTED       0     208b IP es-node-01
nolim_8            2 p STARTED       0     208b IP es-node-03
nolim_8            0 p STARTED       0     208b IP es-node-02
dhcp_pv_56         1 p STARTED       0     208b IP es-node-02
dhcp_pv_56         2 p STARTED       0     208b IP es-node-01
dhcp_pv_56         0 p STARTED       0     208b IP es-node-03
3mon_12            3 p STARTED  544275  105.3mb IP es-node-02
3mon_12            1 p STARTED  543073    105mb IP es-node-01
3mon_12            2 p STARTED  545140  105.4mb IP es-node-03
3mon_12            0 p STARTED  545338  105.4mb IP es-node-02
graylog_57         6 p STARTED 5003320      3gb IP es-node-01
graylog_57         1 p STARTED 4997518    2.9gb IP es-node-03
graylog_57         2 p STARTED 4999913      3gb IP es-node-02
graylog_57         4 p STARTED 5006677    2.9gb IP es-node-03
graylog_57         5 p STARTED 5002309      3gb IP es-node-02
graylog_57         7 p STARTED 5003034      3gb IP es-node-02
graylog_57         3 p STARTED 4999835      3gb IP es-node-01
graylog_57         0 p STARTED 5002807    2.9gb IP es-node-01
3mon_27            3 p STARTED  511680   98.7mb IP es-node-02
3mon_27            1 p STARTED  511434   98.6mb IP es-node-02
3mon_27            2 p STARTED  513373   98.9mb IP es-node-01
3mon_27            0 p STARTED  512896   98.9mb IP es-node-03
nolim_23           3 p STARTED       0     208b IP es-node-01
nolim_23           1 p STARTED       0     208b IP es-node-03
nolim_23           2 p STARTED       0     208b IP es-node-02
nolim_23           0 p STARTED       0     208b IP es-node-01
dhcp_pv_63         1 p STARTED       0     208b IP es-node-03
dhcp_pv_63         2 p STARTED       0     208b IP es-node-02
dhcp_pv_63         0 p STARTED       0     208b IP es-node-01
nolim_25           3 p STARTED       0     208b IP es-node-02
nolim_25           1 p STARTED       0     208b IP es-node-01
nolim_25           2 p STARTED       0     208b IP es-node-03
nolim_25           0 p STARTED       0     208b IP es-node-02
gl-events_0        3 p STARTED      26   19.2kb IP es-node-02
gl-events_0        1 p STARTED      17   17.2kb IP es-node-02
gl-events_0        2 p STARTED      18   17.4kb IP es-node-01
gl-events_0        0 p STARTED      19   17.7kb IP es-node-03
dhcp_ind_52        3 p STARTED  103521   31.7mb IP es-node-01
dhcp_ind_52        1 p STARTED  103520   31.7mb IP es-node-03
dhcp_ind_52        2 p STARTED  103316   31.6mb IP es-node-02
dhcp_ind_52        0 p STARTED  103589   31.7mb IP es-node-01
dhcp_ind_60        3 p STARTED  120914   36.9mb IP es-node-01
dhcp_ind_60        1 p STARTED  120942   36.9mb IP es-node-03
dhcp_ind_60        2 p STARTED  121064   36.9mb IP es-node-02
dhcp_ind_60        0 p STARTED  120843   36.9mb IP es-node-01
nolim_27           3 p STARTED       0     208b IP es-node-01
nolim_27           1 p STARTED       0     208b IP es-node-03
nolim_27           2 p STARTED       0     208b IP es-node-02
nolim_27           0 p STARTED       0     208b IP es-node-01
nolim_10           3 p STARTED       0     208b IP es-node-01
nolim_10           1 p STARTED       0     208b IP es-node-03
nolim_10           2 p STARTED       0     208b IP es-node-02
nolim_10           0 p STARTED       0     208b IP es-node-01
dhcp_pv_58         1 p STARTED       0     208b IP es-node-03
dhcp_pv_58         2 p STARTED       0     208b IP es-node-01
dhcp_pv_58         0 p STARTED       0     208b IP es-node-02
3mon_21            3 p STARTED  512638   99.4mb IP es-node-03
3mon_21            1 p STARTED  512739   99.4mb IP es-node-02
3mon_21            2 p STARTED  512474   99.3mb IP es-node-01
3mon_21            0 p STARTED  511244   99.2mb IP es-node-03
nolim_7            3 p STARTED       0     208b IP es-node-02
nolim_7            1 p STARTED       0     208b IP es-node-01
nolim_7            2 p STARTED       0     208b IP es-node-03
nolim_7            0 p STARTED       0     208b IP es-node-02
k8s_45             3 p STARTED 3967285    1.6gb IP es-node-01
k8s_45             1 p STARTED 3966958    1.6gb IP es-node-03
k8s_45             2 p STARTED 3967330    1.6gb IP es-node-02
k8s_45             0 p STARTED 3966582    1.6gb IP es-node-01
gl-events_1        3 p STARTED     189  131.3kb IP es-node-01
gl-events_1        1 p STARTED     177   93.2kb IP es-node-03
gl-events_1        2 p STARTED     207  135.4kb IP es-node-02
gl-events_1        0 p STARTED     194   97.3kb IP es-node-01
gl-system-events_1 3 p STARTED       0     208b IP es-node-03
gl-system-events_1 1 p STARTED       0     208b IP es-node-02
gl-system-events_1 2 p STARTED       0     208b IP es-node-01
gl-system-events_1 0 p STARTED       0     208b IP es-node-03
nolim_4            3 p STARTED       0     208b IP es-node-01
nolim_4            1 p STARTED       0     208b IP es-node-03
nolim_4            2 p STARTED       0     208b IP es-node-02
nolim_4            0 p STARTED       0     208b IP es-node-01
3mon_11            3 p STARTED  544772  105.9mb IP es-node-03
3mon_11            1 p STARTED  544072  105.8mb IP es-node-01
3mon_11            2 p STARTED  543742  105.7mb IP es-node-02
3mon_11            0 p STARTED  544775  105.8mb IP es-node-03
k8s_44             3 p STARTED 2358398  973.1mb IP es-node-01
k8s_44             1 p STARTED 2359066  973.2mb IP es-node-03
k8s_44             2 p STARTED 2361309  974.1mb IP es-node-02
k8s_44             0 p STARTED 2358908  973.4mb IP es-node-01
dhcp_pv_55         1 p STARTED       0     208b IP es-node-02
dhcp_pv_55         2 p STARTED       0     208b IP es-node-01
dhcp_pv_55         0 p STARTED       0     208b IP es-node-03
nolim_1            3 p STARTED       0     208b IP es-node-01
nolim_1            1 p STARTED       0     208b IP es-node-03
nolim_1            2 p STARTED       0     208b IP es-node-02
nolim_1            0 p STARTED       0     208b IP es-node-01
dhcp_pv_65         1 p STARTED       0     208b IP es-node-02
dhcp_pv_65         2 p STARTED       0     208b IP es-node-01
dhcp_pv_65         0 p STARTED       0     208b IP es-node-03
dhcp_ind_49        3 p STARTED  125385   38.2mb IP es-node-02
dhcp_ind_49        1 p STARTED  125589   38.2mb IP es-node-03
dhcp_ind_49        2 p STARTED  125593   38.2mb IP es-node-01
dhcp_ind_49        0 p STARTED  125206   38.1mb IP es-node-02
graylog_56         6 p STARTED 3646333    2.2gb IP es-node-01
graylog_56         3 p STARTED 3648650    2.2gb IP es-node-03
graylog_56         5 p STARTED 3651768    2.2gb IP es-node-01
graylog_56         1 p STARTED 3650232    2.2gb IP es-node-02
graylog_56         2 p STARTED 3650905    2.2gb IP es-node-01
graylog_56         7 p STARTED 3652341    2.2gb IP es-node-02
graylog_56         4 p STARTED 3651581    2.2gb IP es-node-02
graylog_56         0 p STARTED 3652719    2.2gb IP es-node-03
3mon_28            3 p STARTED  511655   99.6mb IP es-node-02
3mon_28            1 p STARTED  510545   99.4mb IP es-node-03
3mon_28            2 p STARTED  510740   99.4mb IP es-node-01
3mon_28            0 p STARTED  511115   99.4mb IP es-node-03
3mon_3             3 p STARTED       0     208b IP es-node-02
3mon_3             1 p STARTED       0     208b IP es-node-01
3mon_3             2 p STARTED       0     208b IP es-node-03
3mon_3             0 p STARTED       0     208b IP es-node-02
dhcp_pv_60         1 p STARTED       0     208b IP es-node-02
dhcp_pv_60         2 p STARTED       0     208b IP es-node-01
dhcp_pv_60         0 p STARTED       0     208b IP es-node-03
3mon_29            3 p STARTED  513629    100mb IP es-node-03
3mon_29            1 p STARTED  514000  100.1mb IP es-node-02
3mon_29            2 p STARTED  514459  100.2mb IP es-node-01
3mon_29            0 p STARTED  514395  100.2mb IP es-node-03
dhcp_ind_54        3 p STARTED  106379   32.5mb IP es-node-02
dhcp_ind_54        1 p STARTED  106788   32.6mb IP es-node-02
dhcp_ind_54        2 p STARTED  106616   32.6mb IP es-node-01
dhcp_ind_54        0 p STARTED  106541   32.5mb IP es-node-03
dhcp_pv_54         1 p STARTED       0     208b IP es-node-01
dhcp_pv_54         2 p STARTED       0     208b IP es-node-03
dhcp_pv_54         0 p STARTED       0     208b IP es-node-02
dhcp_ind_50        3 p STARTED   99105   30.5mb IP es-node-03
dhcp_ind_50        1 p STARTED  100145   30.7mb IP es-node-02
dhcp_ind_50        2 p STARTED   99232   30.4mb IP es-node-01
dhcp_ind_50        0 p STARTED  100103   30.7mb IP es-node-03
nolim_3            3 p STARTED       0     208b IP es-node-03
nolim_3            1 p STARTED       0     208b IP es-node-02
nolim_3            2 p STARTED       0     208b IP es-node-01
nolim_3            0 p STARTED       0     208b IP es-node-03
nolim_0            3 p STARTED       0     208b IP es-node-01
nolim_0            1 p STARTED       0     208b IP es-node-03
nolim_0            2 p STARTED       0     208b IP es-node-02
nolim_0            0 p STARTED       0     208b IP es-node-01
nolim_18           3 p STARTED       0     208b IP es-node-01
nolim_18           1 p STARTED       0     208b IP es-node-03
nolim_18           2 p STARTED       0     208b IP es-node-02
nolim_18           0 p STARTED       0     208b IP es-node-01
3mon_8             3 p STARTED  368188   71.8mb IP es-node-03
3mon_8             1 p STARTED  368697   71.9mb IP es-node-02
3mon_8             2 p STARTED  369253     72mb IP es-node-01
3mon_8             0 p STARTED  368821   71.9mb IP es-node-03
dhcp_pv_59         1 p STARTED       0     208b IP es-node-01
dhcp_pv_59         2 p STARTED       0     208b IP es-node-03
dhcp_pv_59         0 p STARTED       0     208b IP es-node-02
nolim_2            3 p STARTED       0     208b IP es-node-01
nolim_2            1 p STARTED       0     208b IP es-node-03
nolim_2            2 p STARTED       0     208b IP es-node-02
nolim_2            0 p STARTED       0     208b IP es-node-01
dhcp_pv_64         1 p STARTED       0     208b IP es-node-02
dhcp_pv_64         2 p STARTED       0     208b IP es-node-01
dhcp_pv_64         0 p STARTED       0     208b IP es-node-03
www_62             3 p STARTED     731    1.1mb IP es-node-02
www_62             1 p STARTED     730    1.2mb IP es-node-01
www_62             2 p STARTED     745    1.1mb IP es-node-03
www_62             0 p STARTED     762      1mb IP es-node-02
3mon_24            3 p STARTED  512753   99.4mb IP es-node-03
3mon_24            1 p STARTED  511576   99.3mb IP es-node-02
3mon_24            2 p STARTED  513171   99.5mb IP es-node-01
3mon_24            0 p STARTED  512113   99.4mb IP es-node-03
k8s_55             3 p STARTED 2694033    1.1gb IP es-node-01
k8s_55             1 p STARTED 2690355    1.1gb IP es-node-02
k8s_55             2 p STARTED 2691899    1.1gb IP es-node-01
k8s_55             0 p STARTED 2695772    1.1gb IP es-node-03
dhcp_pv_61         1 p STARTED       0     208b IP es-node-01
dhcp_pv_61         2 p STARTED       0     208b IP es-node-02
dhcp_pv_61         0 p STARTED       0     208b IP es-node-03
nolim_6            3 p STARTED       0     208b IP es-node-02
nolim_6            1 p STARTED       0     208b IP es-node-01
nolim_6            2 p STARTED       0     208b IP es-node-03
nolim_6            0 p STARTED       0     208b IP es-node-02
k8s_51             3 p STARTED 2883659    1.2gb IP es-node-03
k8s_51             1 p STARTED 2882611    1.2gb IP es-node-02
k8s_51             2 p STARTED 2886472    1.2gb IP es-node-01
k8s_51             0 p STARTED 2884205    1.2gb IP es-node-03
graylog_59         4 p STARTED 2380682    1.5gb IP es-node-01
graylog_59         1 p STARTED 2384689    1.4gb IP es-node-01
graylog_59         6 p STARTED 2380306    1.6gb IP es-node-02
graylog_59         5 p STARTED 2382791    1.9gb IP es-node-03
graylog_59         2 p STARTED 2380295    1.5gb IP es-node-03
graylog_59         3 p STARTED 2385432    1.5gb IP es-node-02
graylog_59         7 p STARTED 2383996    1.6gb IP es-node-01
graylog_59         0 p STARTED 2382540    1.5gb IP es-node-02
3mon_17            3 p STARTED  512670   99.2mb IP es-node-03
3mon_17            1 p STARTED  511819   98.9mb IP es-node-01
3mon_17            2 p STARTED  512303   99.1mb IP es-node-02
3mon_17            0 p STARTED  511912     99mb IP es-node-03
dhcp_pv_57         1 p STARTED       0     208b IP es-node-03
dhcp_pv_57         2 p STARTED       0     208b IP es-node-02
dhcp_pv_57         0 p STARTED       0     208b IP es-node-01
3mon_1             3 p STARTED       0     208b IP es-node-02
3mon_1             1 p STARTED       0     208b IP es-node-02
3mon_1             2 p STARTED       0     208b IP es-node-01
3mon_1             0 p STARTED       0     208b IP es-node-03
dhcp_pv_53         1 p STARTED       0     208b IP es-node-01
dhcp_pv_53         2 p STARTED       0     208b IP es-node-03
dhcp_pv_53         0 p STARTED       0     208b IP es-node-02
k8s_61             3 p STARTED 1546811  816.6mb IP es-node-01
k8s_61             1 p STARTED 1548154  722.2mb IP es-node-03
k8s_61             2 p STARTED 1546928  715.4mb IP es-node-02
k8s_61             0 p STARTED 1545834  758.3mb IP es-node-01
k8s_52             3 p STARTED 2857538    1.2gb IP es-node-01
k8s_52             1 p STARTED 2856882    1.2gb IP es-node-03
k8s_52             2 p STARTED 2860519    1.2gb IP es-node-02
k8s_52             0 p STARTED 2857702    1.2gb IP es-node-01
3mon_19            3 p STARTED  510893   98.1mb IP es-node-03
3mon_19            1 p STARTED  512550   98.5mb IP es-node-01
3mon_19            2 p STARTED  511649   98.3mb IP es-node-02
3mon_19            0 p STARTED  512901   98.6mb IP es-node-03
nolim_29           3 p STARTED       0     208b IP es-node-02
nolim_29           1 p STARTED       0     208b IP es-node-01
nolim_29           2 p STARTED       0     208b IP es-node-03
nolim_29           0 p STARTED       0     208b IP es-node-02
k8s_50             3 p STARTED 3018134    1.3gb IP es-node-03
k8s_50             1 p STARTED 3016650    1.3gb IP es-node-02
k8s_50             2 p STARTED 3018206    1.3gb IP es-node-01
k8s_50             0 p STARTED 3018919    1.3gb IP es-node-03
3mon_7             3 p STARTED  362500   70.5mb IP es-node-03
3mon_7             1 p STARTED  360690   70.1mb IP es-node-02
3mon_7             2 p STARTED  360933   70.1mb IP es-node-01
3mon_7             0 p STARTED  361584   70.3mb IP es-node-03
nolim_11           3 p STARTED       0     208b IP es-node-01
nolim_11           1 p STARTED       0     208b IP es-node-03
nolim_11           2 p STARTED       0     208b IP es-node-02
nolim_11           0 p STARTED       0     208b IP es-node-01
nolim_14           3 p STARTED       0     208b IP es-node-01
nolim_14           1 p STARTED       0     208b IP es-node-03
nolim_14           2 p STARTED       0     208b IP es-node-02
nolim_14           0 p STARTED       0     208b IP es-node-01
www_60             3 p STARTED    2599    1.6mb IP es-node-01
www_60             1 p STARTED    2538    1.6mb IP es-node-03
www_60             2 p STARTED    2512    1.6mb IP es-node-02
www_60             0 p STARTED    2646    1.7mb IP es-node-01
3mon_6             3 p STARTED       0     208b IP es-node-03
3mon_6             1 p STARTED       0     208b IP es-node-02
3mon_6             2 p STARTED       0     208b IP es-node-01
3mon_6             0 p STARTED       0     208b IP es-node-03
nolim_15           3 p STARTED       0     208b IP es-node-01
nolim_15           1 p STARTED       0     208b IP es-node-03
nolim_15           2 p STARTED       0     208b IP es-node-02
nolim_15           0 p STARTED       0     208b IP es-node-01
k8s_46             3 p STARTED 2951652    1.2gb IP es-node-02
k8s_46             1 p STARTED 2955851    1.2gb IP es-node-01
k8s_46             2 p STARTED 2951303    1.2gb IP es-node-03
k8s_46             0 p STARTED 2954493    1.2gb IP es-node-02
3mon_23            3 p STARTED  512322   99.6mb IP es-node-03
3mon_23            1 p STARTED  513515   99.9mb IP es-node-02
3mon_23            2 p STARTED  511589   99.5mb IP es-node-01
3mon_23            0 p STARTED  512221   99.6mb IP es-node-03
k8s_54             3 p STARTED 2690039    1.1gb IP es-node-02
k8s_54             1 p STARTED 2689798    1.1gb IP es-node-01
k8s_54             2 p STARTED 2690440    1.1gb IP es-node-03
k8s_54             0 p STARTED 2691941    1.1gb IP es-node-02
3mon_22            3 p STARTED  511373  100.3mb IP es-node-03
3mon_22            1 p STARTED  511970  100.4mb IP es-node-02
3mon_22            2 p STARTED  512007  100.3mb IP es-node-01
3mon_22            0 p STARTED  512388  100.4mb IP es-node-03
3mon_9             3 p STARTED  545230  106.2mb IP es-node-02
3mon_9             1 p STARTED  544644  106.1mb IP es-node-03
3mon_9             2 p STARTED  544151    106mb IP es-node-01
3mon_9             0 p STARTED  544679  106.1mb IP es-node-02
k8s_53             3 p STARTED 2654191    1.1gb IP es-node-02
k8s_53             1 p STARTED 2653612    1.1gb IP es-node-01
k8s_53             2 p STARTED 2654340    1.1gb IP es-node-03
k8s_53             0 p STARTED 2655516    1.1gb IP es-node-02
graylog_58         3 p STARTED 4805837    2.9gb IP es-node-03
graylog_58         6 p STARTED 4808681    2.9gb IP es-node-03
graylog_58         1 p STARTED 4810224    2.9gb IP es-node-02
graylog_58         7 p STARTED 4805985    2.9gb IP es-node-02
graylog_58         4 p STARTED 4804286    2.9gb IP es-node-02
graylog_58         2 p STARTED 4806822    2.9gb IP es-node-01
graylog_58         5 p STARTED 4803674    2.9gb IP es-node-01
graylog_58         0 p STARTED 4806478    2.9gb IP es-node-03
dhcp_ind_56        3 p STARTED  102617   31.4mb IP es-node-02
dhcp_ind_56        1 p STARTED  101873   31.2mb IP es-node-02
dhcp_ind_56        2 p STARTED  102320   31.3mb IP es-node-01
dhcp_ind_56        0 p STARTED  102168   31.3mb IP es-node-03
nolim_5            3 p STARTED       0     208b IP es-node-02
nolim_5            1 p STARTED       0     208b IP es-node-01
nolim_5            2 p STARTED       0     208b IP es-node-03
nolim_5            0 p STARTED       0     208b IP es-node-02
nolim_17           3 p STARTED       0     208b IP es-node-01
nolim_17           1 p STARTED       0     208b IP es-node-03
nolim_17           2 p STARTED       0     208b IP es-node-02
nolim_17           0 p STARTED       0     208b IP es-node-01
dhcp_ind_47        3 p STARTED  124823     38mb IP es-node-03
dhcp_ind_47        1 p STARTED  124270   37.8mb IP es-node-02
dhcp_ind_47        2 p STARTED  124778     38mb IP es-node-01
dhcp_ind_47        0 p STARTED  124426   37.9mb IP es-node-03
nolim_16           3 p STARTED       0     208b IP es-node-01
nolim_16           1 p STARTED       0     208b IP es-node-03
nolim_16           2 p STARTED       0     208b IP es-node-02
nolim_16           0 p STARTED       0     208b IP es-node-01
k8s_42             3 p STARTED     890  200.9kb IP es-node-02
k8s_42             1 p STARTED     933  210.1kb IP es-node-02
k8s_42             2 p STARTED     892  200.1kb IP es-node-01
k8s_42             0 p STARTED     961  220.6kb IP es-node-03
dhcp_pv_66         1 p STARTED       0     208b IP es-node-03
dhcp_pv_66         2 p STARTED       0     208b IP es-node-02
dhcp_pv_66         0 p STARTED       0     208b IP es-node-01
dhcp_ind_59        3 p STARTED   97544     30mb IP es-node-02
dhcp_ind_59        1 p STARTED   97038   29.9mb IP es-node-01
dhcp_ind_59        2 p STARTED   96967   29.8mb IP es-node-03
dhcp_ind_59        0 p STARTED   96687   29.7mb IP es-node-02
dhcp_ind_48        3 p STARTED   99774   30.6mb IP es-node-02
dhcp_ind_48        1 p STARTED  100670   30.9mb IP es-node-01
dhcp_ind_48        2 p STARTED  100438   30.8mb IP es-node-03
dhcp_ind_48        0 p STARTED   99795   30.6mb IP es-node-02
www_59             3 p STARTED     184  146.9kb IP es-node-03
www_59             1 p STARTED     172  142.4kb IP es-node-01
www_59             2 p STARTED     173  143.3kb IP es-node-02
www_59             0 p STARTED     168  141.2kb IP es-node-02
3mon_13            3 p STARTED  544612  105.3mb IP es-node-02
3mon_13            1 p STARTED  544247  105.1mb IP es-node-01
3mon_13            2 p STARTED  543423    105mb IP es-node-03
3mon_13            0 p STARTED  545592  105.4mb IP es-node-02
3mon_14            3 p STARTED  544215  105.7mb IP es-node-01
3mon_14            1 p STARTED  545216    106mb IP es-node-01
3mon_14            2 p STARTED  544176  105.7mb IP es-node-03
3mon_14            0 p STARTED  544458  105.9mb IP es-node-02
3mon_26            3 p STARTED  513186   99.2mb IP es-node-03
3mon_26            1 p STARTED  511720   98.9mb IP es-node-01
3mon_26            2 p STARTED  512166     99mb IP es-node-02
3mon_26            0 p STARTED  512228   99.1mb IP es-node-03
dhcp_ind_53        3 p STARTED  122152   37.2mb IP es-node-03
dhcp_ind_53        1 p STARTED  121790   37.1mb IP es-node-02
dhcp_ind_53        2 p STARTED  123117   37.5mb IP es-node-01
dhcp_ind_53        0 p STARTED  122407   37.3mb IP es-node-03
3mon_15            3 p STARTED  544231  105.7mb IP es-node-01
3mon_15            1 p STARTED  544645  105.8mb IP es-node-01
3mon_15            2 p STARTED  544183  105.7mb IP es-node-03
3mon_15            0 p STARTED  543571  105.6mb IP es-node-02
nolim_9            3 p STARTED       0     208b IP es-node-03
nolim_9            1 p STARTED       0     208b IP es-node-02
nolim_9            2 p STARTED       0     208b IP es-node-01
nolim_9            0 p STARTED       0     208b IP es-node-03
nolim_28           3 p STARTED       0     208b IP es-node-02
nolim_28           1 p STARTED       0     208b IP es-node-03
nolim_28           2 p STARTED       0     208b IP es-node-01
nolim_28           0 p STARTED       0     208b IP es-node-03

Uporaba · September 29, 2021, 6:46am

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
  },
  "status" : 400
}

Uporaba · September 29, 2021, 6:51am

Operating system information

CentOS Linux release 7.9.2009 (Core)

Package versions

Graylog: 4.1.3
MongoDB: db version v4.2.15
Elasticsearch: “number” : “7.10.2”

Uporaba · September 29, 2021, 7:46am

I also ran into this problem:

custom mapping contains:

{
  "template": "graylog_*",
  "mappings": {
    "properties": {
        "request_time" : {
            "type" : "float"
        },
        "upstream_connect_time" : {
            "type" : "float"
        },
        "upstream_header_time" : {
            "type" : "float"
        },
        "upstream_response_time" : {
            "type" : "float"
        },
        "response_time" : {
            "type" : "float"
        },
        "api_errorcode" : {
            "type" : "text",
            "fielddata" : true
      }
    }
  }
}

But if you look at this field in the search, it shows that it has 2 data types:

How i can fix it?

gsmith · September 29, 2021, 10:15pm

Hello,

I fix you post so its easier to read.

Uporaba:

"cluster_name" : "graylog", 
"status" : "green", 
"timed_out" : false, 
"number_of_nodes" : 3, 
"number_of_data_nodes" : 3,
"active_primary_shards" : 490, 
"active_shards" : 490, 
"relocating_shards" : 0, 
"initializing_shards" : 0,
"unassigned_shards" : 0, 
"delayed_unassigned_shards" : 0, 
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0, 
"task_max_waiting_in_queue_millis" : 0, 
"active_shards_percent_as_number" : 100.0 }

So you have a 3 node cluster? Are all these nodes same version (i.e. elasticsearch, MongoDb, Graylog) Knowing that probably would have been helpful.
.
Do you have a custom template for your Index?
Do you have any pipelines or extractors on your INPUT/s?

From what you posted so far and I’m assuming your using a custom template or you made some type of changes to index mapping in your template.

Widget Error Break Down.

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default.
Please use a keyword field instead.
Alternatively, set fielddata=true on [responce_time]
in order to load field data by uninverting the inverted index. Note that this can use significant memory.

Next section also shows you what’s wrong, this would be what you need to fix.

Illegal_argument_exception, reason=field [responce_time] of type [Keyword] is not supported for aggregation [avg]]
Illegal_argument_exception, reason=field [responce_time] of type[text] is not supported for aggregation [avg]

I believe that your field responce_time has the incorrect data type in your Index template (elasticsearch) and this is the reason your receiving "Elasticsearch exception reason=all shards failed" and your widget error show this also.

To sum it up, looks like you field responce_time is not supported for Average on your widget. Might need to recreate the widget or adjust your field mapping in you index template.

Hope that helps

Uporaba · September 30, 2021, 6:05am

Yes all nodes same version.

Yea, i use custom mapping:

{
  "template": "graylog_*",
  "mappings": {
    "properties": {
        "request_time" : {
            "type" : "float"
        },
        "upstream_connect_time" : {
            "type" : "float"
        },
        "upstream_header_time" : {
            "type" : "float"
        },
        "upstream_response_time" : {
            "type" : "float"
        },
        "response_time" : {
            "type" : "float"
        },
        "api_errorcode" : {
            "type" : "text",
            "fielddata" : true
      }
    }
  }
}

Then what type of data do I need to put in order for everything to work?

and why does it show 2 types of field data?

gsmith · September 30, 2021, 10:06pm

Hello

Unfortunately I do not know how you configured you environment but I would assume it has something to do with your index custom template. Just an idea, If you have more them one elasticsearch node make sure the custom index template on each Elasticsearch node is the same.
Try executing this on each ES node see if the field response_time has more then one data type.

curl 'http://es_node1:9200/_mapping?pretty'

EXAMPLE:
Below is from my lab widget using average function.

Index mapping for the widget.

This is shown in a All messages.

It seems that you have something wrong with your custom index template. Some where, elasticsearch is trying to use two field data types which is throwing the error.

Edit:

Sorry I missed you question. Take a look at this link below for the type of data type you need.

Since your using a custom template use the PUT command. When you find what data type you do need add it to your index template.

curl -X PUT "localhost:9200/my-index-000001/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    " response_time ": {
      "type": "long"
    }
  }
}
'

Hope that helps

Uporaba · October 1, 2021, 7:21am

Well, the problem with the data type seems to be solved, the error is gone. Now this problem remains. When I make a request like this: url_without_param: *my_foo_app_api* AND NOT source: *testing*
I get this error:

This request is an example, there are many such requests that give this error
How can I check why this is happening? And how can I fix it?

P.S-In the old version of graylog everything worked fine

gsmith · October 1, 2021, 9:40pm

Hello,

Can you explain what you did to solve this problem? This would give use a better idea what’s going on in your environment.

First, I would execute the command below. Just to be sure.

curl -XGET http://localhost:9200/_cluster/health?pretty=true

Next, I would check out my log files ( i.e. Elasticsearch, Graylog, etc…) on the Graylog Server to get a better clarity . Maybe it can explain whats happening.

This documentation will show the location of those log files.
https://docs.graylog.org/en/4.1/pages/configuration/file_location.html

Sometimes this happens because the string type of grouping aggregate query. This would be found in your index template mapping.

Here is something similar to your issue.

With your custom template, those errors seam to arise from that. That would be the first place I would look. Check out your mapping for url_without_param && source make sure you have the right data type.

If you feel that this is a bug you can post it here.

Hope that helps.

Uporaba · October 2, 2021, 2:34pm

I followed what you wrote to me latter

is good

{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 93,
  "active_shards" : 278,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Graylog logs: https://pastebin.com/KAGKT8Fw
Elastic logs is clear
Mongo is clear too

How can I figure out what type of data I need?

Uporaba · October 2, 2021, 2:50pm

This unfortunately does not solve my problem

Uporaba · October 4, 2021, 3:26pm

Hello. I found another problem besides the fact that all indexes failed

aaronsachs · October 4, 2021, 9:37pm

That error is due to the fact that leading wildcards are disabled by default. See server.conf — Graylog 4.1.0 documentation You’ll need to enable them, though I would exercise caution, as that can cause some performance issues when searching. You can use allow_leading_wildcard_searches = true to change this in your server config.

Uporaba · October 4, 2021, 10:26pm

dude, thx. i think what is solved all my problems

gsmith · October 5, 2021, 12:33am

Hello,

Sorry for the late reply, weekend issues.
Nice, that solved your issue/s. Also note what @aaronsachs said, there could be some performance issues.

Uporaba · October 5, 2021, 4:06am

is what? lack of resources? or something else? Slower operation?

gsmith · October 5, 2021, 4:08am

Topic		Replies	Views
Issues with Elasticsearch 7.10.1 Graylog Central (peer support)	8	2064	April 7, 2021
Getting mapper_parsing_exception Graylog Central (peer support)	2	1288	October 20, 2018
Error messages - Index Failures Graylog Central (peer support)	2	495	August 21, 2018
Indexer Errors Because of mapper_parsing_exception Graylog Central (peer support) filebeat-linux , elastic	3	623	January 1, 2024
Lots of Indexer failures recently Graylog Central (peer support) pipeline-rules , debuggingpl	6	1207	April 16, 2021

Elasticsearch exception reason=all shards failed

Operating system information

Package versions

Related topics