Graylog Illuminate, need feedback

[benoitp]

Hello,

I’m trying for days to get Graylog Illuminate working without any success.

• Graylog 5.0.5
• opensearch-2.6.0
• mongodb-org-6.0.5
• Illuminate 3.2.0

In Illuminate Processing Packs, we have activated :
• Illuminate Core v3.2.0:GIM Enforcement Add-on
• Illuminate v3.2.0:Microsoft Windows Security
• Illuminate v3.2.0:Apache HTTPD

We tried to add some Apache logs with Filebeat 7, and a input Beat without any setting, on the Graylog side.

The message are routed in the new stream « Illuminate:Apache2 Device Messages»

But without any modification.
So I’m wondering what Graylog Illuminate is supposed to do, and if someone is using it and how.

Note :
I disables ALL my Pipelines
I’m NOT using Sidecar., so i added this to filebeat to see if it was better with it :

fields.collector_node_id: dirv-monitoring-centreon-02
fields.gl2_source_collector: 1111

filebeat.yml

filebeat.inputs:

- type: filestream
  paths:
    - "/var/log/httpd/*access_log"
    - "/var/log/httpd24/*access_log"
  fields_under_root: true
  fields:
    event_source_product: apache_httpd

- type: filestream
  paths:
    - "/var/log/httpd/*error_log"
    - "/var/log/httpd24/*error_log"
  fields_under_root: true
  fields:
    event_source_product: apache_httpd

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

output.logstash:
  hosts: ["pirv-siem-es-graylog-01:5044", "pirv-siem-es-graylog-02:5044", "pirv-siem-es-graylog-03:5044"]

Message in Graylog :

{
  "filebeat_log_offset": 2422350,
  "filebeat_agent_name": "dirv-monitoring-centreon-02.",
  "gl2_remote_ip": "10.128.10.5",
  "gl2_remote_port": 55132,
  "filebeat_fields_collector_node_id": "dirv-monitoring-centreon-02",
  "filebeat_event_source_product": "apache_httpd",
  "source": "dirv-monitoring-centreon-02.",
  "filebeat_agent_hostname": "dirv-monitoring-centreon-02.",
  "beats_type": "filebeat",
  "gl2_source_input": "642c26a81547815d600a39b8",
  "filebeat_@metadata_beat": "filebeat",
  "filebeat_@timestamp": "2023-04-05T07:39:31.004Z",
  "filebeat_agent_type": "filebeat",
  "filebeat_@metadata_version": "7.17.9",
  "filebeat_host_name": "dirv-monitoring-centreon-02.",
  "gl2_source_node": "91c8f29c-a136-4651-919f-318e38fbf955",
  "filebeat_agent_version": "7.17.9",
  "timestamp": "2023-04-05T07:39:31.004Z",
  "filebeat_agent_ephemeral_id": "493daebd-c566-436f-a875-0181d8398152",
  "event_source_product": "apache_httpd",
  "gl2_accounted_message_size": 1243,
  "filebeat_input_type": "filestream",
  "filebeat_fields_gl2_source_collector": 1111,
  "streams": [
    "642c3a051547815d600a6248"
  ],
  "gl2_message_id": "01GX85PHS78D0EHSQAT5MYRPQ6",
  "message": "178.237.98.45 - - [05/Apr/2023:09:39:30 +0200] \"GET /centreon/api/internal.php?object=centreon_keepalive&action=keepAlive HTTP/1.1\" 200 24 \"http://dirv-monitoring-centreon-02/centreon/administration/extensions/manager\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0\"",
  "filebeat_ecs_version": "1.12.0",
  "filebeat_@metadata_type": "_doc",
  "filebeat_agent_id": "52ccc1d6-85f5-4d0b-8487-a5ca1d7f240e",
  "_id": "010c7c61-d385-11ed-bfd2-005056ba6fc2",
  "filebeat_log_file_path": "/var/log/httpd/access_log"
}

[benoitp]

Solution found, with help from the support because all Illuminate rules are hidden, so you cannot debug anything, and the documentation don’t give the complete information about the requirements.

The access log MUST be ending with : “/access.log”

To bad when you know that the standard name under RedHat family is “access_log”

The only solution is to change the name of the log files , on all host, all VirtualHost

But when you have thousands of servers, appliances, applications from vendors, etc. It’s nearly impossible !

Hey @Graylog a great feature of Filebeat is tags
tags: ["apache", "access"]