Why are the same logs appearing?

Graylog Central (peer support)
, , ,
1

Hi all,
I find there are two same search result in the graph, only difference between them is the timestamp field. But I find only one message in log file.

image
image.png999×441 24.1 KB

In this graph,in fact it’s just two logs, but search four result.I am very confused.
Please help me, thank you!

2

How exactly are you sending these logs to Graylog?
What’s the complete configuration of all involved components?

3

I use collector sidecar filebeat to send logs to graylog.
collector sidecar configuration is

server_url: http://XXXXXXXXX/api/
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
node_id: collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - hybris-debug
    - hybris-error
    - hybris-info
    - hybris-tomcat-access
    - hybris-tomcat-console
    - inv-full
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

filebeat configuraiton is:

filebeat:
  prospectors:
  - encoding: utf-8
    exclude_files: []
    fields:
      collector_node_id: collector-sidecar-job04-p
      flag: hybris-tomcat-console
      gl2_source_collector: cc6b9ac5-a9c1-4a59-abaa-923893da8229
      type: log
    ignore_older: 0
    paths:
    - /home/prodlog/hybrislog/tomcat/console-*.log
    scan_frequency: 10s
    tail_files: true
    type: log
  - encoding: utf-8
    exclude_files: []
    fields:
      collector_node_id: collector-sidecar-job04-p
      flag: hybris-debug
      gl2_source_collector: cc6b9ac5-a9c1-4a59-abaa-923893da8229
      type: log
    ignore_older: 0
    paths:
    - /home/prodlog/hybrislog/frpf/debug/hybris-debug.log
    scan_frequency: 10s
    tail_files: true
    type: log
  - encoding: utf-8
    exclude_files: []
    fields:
      collector_node_id: collector-sidecar-job04-p
      flag: hybris-info
      gl2_source_collector: cc6b9ac5-a9c1-4a59-abaa-923893da8229
      type: log
    ignore_older: 0
    multiline:
      match: after
      negate: true
      pattern: ^.*[0-9]{2}m[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}
    paths:
    - /home/prodlog/hybrislog/frpf/info/hybris-info.log
    scan_frequency: 10s
    tail_files: true
    type: log
  - encoding: utf-8
    exclude_files: []
    fields:
      collector_node_id: collector-sidecar-job04-p
      flag: hybris-error
      gl2_source_collector: cc6b9ac5-a9c1-4a59-abaa-923893da8229
      type: log
    ignore_older: 0
    multiline:
      match: after
      negate: true
      pattern: ^.*[0-9]{2}m[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}
    paths:
    - /home/prodlog/hybrislog/frpf/error/hybris-error.log
    scan_frequency: 10s
    tail_files: true
    type: log
  - encoding: utf-8
    exclude_files: []
    fields:
      collector_node_id: collector-sidecar-job04-p
      flag: hybris-tomcat-access
      gl2_source_collector: cc6b9ac5-a9c1-4a59-abaa-923893da8229
      type: log
    ignore_older: 0
    paths:
    - /home/prodlog/hybrislog/tomcat/access.*.log
    scan_frequency: 10s
    tail_files: true
    type: log
  - encoding: utf-8
    exclude_files: []
    fields:
      collector_node_id: collector-sidecar-job04-p
      flag: inv-full
      gl2_source_collector: cc6b9ac5-a9c1-4a59-abaa-923893da8229
      type: log
    ignore_older: 0
    paths:
    - /u01/prod/app/hybris6200/hybris/log/frpf/inv/full/1.log
    scan_frequency: 10s
    tail_files: true
    type: log
output:
  logstash:
    hosts:
    - xxxxxxxxx5:5044
    - xxxxxxxxx6:5044
    - xxxxxxxxx1:5044
    - xxxxxxxxx2:5044
    - xxxxxxxxx3:5044
    - xxxxxxxxx4:5044
    loadbalance: true
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- hybris-debug
- hybris-error
- hybris-info
- hybris-tomcat-access
- hybris-tomcat-console
- inv-full

And I find this two message ID are same.

image
image.png981×289 16.8 KB

image

4

Are you using any 3rd party plugins in Graylog?

If the message ID is identical, then it’s the same message and is “only” displayed twice.

5

I don’t use any 3rd party plugin in graylog.I install graylog with default plugin.

image
image.png1305×482 40.9 KB

6

By the way, If I click stream same message in search result display once, but click search button in home page,it display twice:confused:Then two same message appeared in alert notification.Is this problem because the same message is in two streams?

7

It depends. If these two streams have different backing index sets, then the message is being stored twice in Elasticsearch. If all streams are using the same backing index set, then the message is being stored only once.

See http://docs.graylog.org/en/2.4/pages/configuration/index_model.html for details.

8

En…but I configure alert with only one stream , it appeared twice in notification.
Do I need to ensure that only one log of all messages in elasticsearch appear once？

9

As I wrote before, that depends on your use case and how you’ve configured your system.

10

Sorry,I not very clear about it.:sob:The current situation is two streams have different backing index sets. One stream(named stream1) include another stream(named stream2) message, stream1 has more message than stream2. Then I configured alert on stream2,but in notification appear two same message.So I need to change stream2 backing index to stream1 backing index set and problem will be solved? I do not know whether this understanding is correct.
Thank you so much for responding to my questions so quickly.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.