Graylog Illuminate, need feedback


I’m trying for days to get Graylog Illuminate working without any success.

• Graylog 5.0.5
• opensearch-2.6.0
• mongodb-org-6.0.5
• Illuminate 3.2.0

In Illuminate Processing Packs, we have activated :
• Illuminate Core v3.2.0:GIM Enforcement Add-on
• Illuminate v3.2.0:Microsoft Windows Security
• Illuminate v3.2.0:Apache HTTPD

We tried to add some Apache logs with Filebeat 7, and a input Beat without any setting, on the Graylog side.

The message are routed in the new stream « Illuminate:Apache2 Device Messages»

But without any modification.
So I’m wondering what Graylog Illuminate is supposed to do, and if someone is using it and how.

Note :
I disables ALL my Pipelines
I’m NOT using Sidecar., so i added this to filebeat to see if it was better with it :

fields.collector_node_id: dirv-monitoring-centreon-02
fields.gl2_source_collector: 1111



- type: filestream
    - "/var/log/httpd/*access_log"
    - "/var/log/httpd24/*access_log"
  fields_under_root: true
    event_source_product: apache_httpd

- type: filestream
    - "/var/log/httpd/*error_log"
    - "/var/log/httpd24/*error_log"
  fields_under_root: true
    event_source_product: apache_httpd

  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

  hosts: ["pirv-siem-es-graylog-01:5044", "pirv-siem-es-graylog-02:5044", "pirv-siem-es-graylog-03:5044"]

Message in Graylog :

  "filebeat_log_offset": 2422350,
  "filebeat_agent_name": "dirv-monitoring-centreon-02.",
  "gl2_remote_ip": "",
  "gl2_remote_port": 55132,
  "filebeat_fields_collector_node_id": "dirv-monitoring-centreon-02",
  "filebeat_event_source_product": "apache_httpd",
  "source": "dirv-monitoring-centreon-02.",
  "filebeat_agent_hostname": "dirv-monitoring-centreon-02.",
  "beats_type": "filebeat",
  "gl2_source_input": "642c26a81547815d600a39b8",
  "filebeat_@metadata_beat": "filebeat",
  "filebeat_@timestamp": "2023-04-05T07:39:31.004Z",
  "filebeat_agent_type": "filebeat",
  "filebeat_@metadata_version": "7.17.9",
  "filebeat_host_name": "dirv-monitoring-centreon-02.",
  "gl2_source_node": "91c8f29c-a136-4651-919f-318e38fbf955",
  "filebeat_agent_version": "7.17.9",
  "timestamp": "2023-04-05T07:39:31.004Z",
  "filebeat_agent_ephemeral_id": "493daebd-c566-436f-a875-0181d8398152",
  "event_source_product": "apache_httpd",
  "gl2_accounted_message_size": 1243,
  "filebeat_input_type": "filestream",
  "filebeat_fields_gl2_source_collector": 1111,
  "streams": [
  "gl2_message_id": "01GX85PHS78D0EHSQAT5MYRPQ6",
  "message": " - - [05/Apr/2023:09:39:30 +0200] \"GET /centreon/api/internal.php?object=centreon_keepalive&action=keepAlive HTTP/1.1\" 200 24 \"http://dirv-monitoring-centreon-02/centreon/administration/extensions/manager\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0\"",
  "filebeat_ecs_version": "1.12.0",
  "filebeat_@metadata_type": "_doc",
  "filebeat_agent_id": "52ccc1d6-85f5-4d0b-8487-a5ca1d7f240e",
  "_id": "010c7c61-d385-11ed-bfd2-005056ba6fc2",
  "filebeat_log_file_path": "/var/log/httpd/access_log"

Solution found, with help from the support because all Illuminate rules are hidden, so you cannot debug anything, and the documentation don’t give the complete information about the requirements.

The access log MUST be ending with : “/access.log”

To bad when you know that the standard name under RedHat family is “access_log”

The only solution is to change the name of the log files , on all host, all VirtualHost :unamused:

But when you have thousands of servers, appliances, applications from vendors, etc. It’s nearly impossible !

Hey @Graylog a great feature of Filebeat is tags :wink:
tags: ["apache", "access"]


This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.