Graylog messaggges

Graylog Central (peer support)
1

Hi i installed grayloy side car
And i made possible changes in filebeat.yml and sidecar.yml and i configured in ui as well and i could see its running but then i am not able to get messages can anyone please help me in this

2

Did you apply the configuration to the sidecar in the Graylog GUI? It would help if you posted related yml files and configurations - obfuscated where necessary of courseā€¦ The sidecar installation has a logs directory that you can look at that is very expressive of status. Also - read these on posting questions:

3

yes I applied sharing you the config below and i am completely new and i wasnā€™t able to understand so please help me @tmacgbay
The GUI

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
  type: log
output.logstash:
   hosts: ["10.81.1.112:12201"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log
4

sidecar.yml

# The URL to the Graylog server API.

#server_url: "http://127.0.0.1:9000/api/"
i mentioned but removed the service

# The API token to use to authenticate against the Graylog server API.

# This field is mandatory

server_api_token: "1l57rapqidp30ii144es05rr5p15925kl0ehnjo6cis81ph0ejpk"

# The node ID of the sidecar. This can be a path to a file or an ID string.

# If set to a file and the file doesn't exist, the sidecar will generate an

# unique ID and writes it to the configured path.

#

# Example file path: "file:/etc/graylog/sidecar/node-id"

# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"

#

# ATTENTION: Every sidecar instance needs a unique ID!

#

#node_id: "file:/etc/graylog/sidecar/node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the

# hostname of the host it is running on.

#node_name: ""

# The update interval in seconds. This configures how often the sidecar will

# contact the Graylog server for keep-alive and configuration update requests.

#update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.

# Default: false

# tls_skip_verify: true

# This enables/disables the transmission of detailed sidecar information like

# collector statues, metrics and log file lists. It can be disabled to reduce

# load on the Graylog server if needed. (disables some features in the server UI)

#send_status: true

# A list of directories to scan for log files. The sidecar will scan each

# directory for log files and submits them to the server on each update.

#

# Example:

# list_log_files:

# - "/var/log/nginx"

# - "/opt/app/logs"

#

# Default: empty list

#list_log_files: []

# Directory where the sidecar stores internal data.

#cache_path: "/var/cache/graylog-sidecar"

# Directory where the sidecar stores logs for collectors and the sidecar itself.

#log_path: "/var/log/graylog-sidecar"

# The maximum size of the log file before it gets rotated.

#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.

#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.

#collector_configuration_directory: "/var/lib/graylog-sidecar/generated"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.

# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match

# Example:

#collector_binaries_whitelist:

# - "/usr/bin/filebeat"

# - "/opt/collectors/*"

#

# Example disable whitelisting:

# collector_binaries_whitelist: []

#

# Default:

collector_binaries_accesslist:

# - "/opt/filebeat-8.3.2-linux-x86_64/filebeat"

- "/usr/bin/filebeat"

- "/usr/share/filebeat/bin/filebeat"

# - "/usr/bin/filebeat"

# - "/usr/bin/packetbeat"

# - "/usr/bin/metricbeat"

# - "/usr/bin/heartbeat"

# - "/usr/bin/auditbeat"

# - "/usr/bin/journalbeat"

# - "/usr/share/filebeat/bin/filebeat"

# - "/usr/share/packetbeat/bin/packetbeat"

# - "/usr/share/metricbeat/bin/metricbeat"

# - "/usr/share/heartbeat/bin/heartbeat"

# - "/usr/share/auditbeat/bin/auditbeat"

# - "/usr/share/filebeat/bin/filebeat"

# - "/usr/bin/nxlog"

# - "/opt/nxlog/bin/nxlog"

#sudo graylog-sidecar -service install

#sudo systemctl enable graylog-sidecar

#sudo systemctl start graylog-sidecar
5 
###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The  file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:


# For more available modules and options, please see the  sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/audit/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.


# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud 

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  removed host 

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
6

Please edit your posts and use the </> forum tool on your code (highlight the code/log and click on </>

I did this on your first post with code to show you what it looks likeā€¦ it makes it a LOT more readable.
image

7 
I am really really sorry
[quote="kalyan, post:4, topic:25198"]
# The URL to the Graylog server API.

#server_url: ā€œhttp://127.0.0.1:9000/api/ā€
i mentioned but removed the service

# The API token to use to authenticate against the Graylog server API.

# This field is mandatory

server_api_token: ā€œ1l57rapqidp30ii144es05rr5p15925kl0ehnjo6cis81ph0ejpkā€

# The node ID of the sidecar. This can be a path to a file or an ID string.

# If set to a file and the file doesnā€™t exist, the sidecar will generate an

# unique ID and writes it to the configured path.

# Example file path: ā€œfile:/etc/graylog/sidecar/node-idā€

# Example ID string: ā€œ6033137e-d56b-47fc-9762-cd699c11a5a9ā€

# ATTENTION: Every sidecar instance needs a unique ID!

#node_id: ā€œfile:/etc/graylog/sidecar/node-idā€

# The node name of the sidecar. If this is empty, the sidecar will use the

# hostname of the host it is running on.

#node_name: ā€œā€

# The update interval in seconds. This configures how often the sidecar will

# contact the Graylog server for keep-alive and configuration update requests.

#update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.

# Default: false

# tls_skip_verify: true

# This enables/disables the transmission of detailed sidecar information like

# collector statues, metrics and log file lists. It can be disabled to reduce

# load on the Graylog server if needed. (disables some features in the server UI)

#send_status: true

# A list of directories to scan for log files. The sidecar will scan each

# directory for log files and submits them to the server on each update.

# Example:

# list_log_files:

# - ā€œ/var/log/nginxā€

# - ā€œ/opt/app/logsā€

# Default: empty list

#list_log_files:

# Directory where the sidecar stores internal data.

#cache_path: ā€œ/var/cache/graylog-sidecarā€

# Directory where the sidecar stores logs for collectors and the sidecar itself.

#log_path: ā€œ/var/log/graylog-sidecarā€

# The maximum size of the log file before it gets rotated.

#log_rotate_max_file_size: ā€œ10MiBā€

# The maximum number of old log files to retain.

#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.

#collector_configuration_directory: ā€œ/var/lib/graylog-sidecar/generatedā€

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.

# Wildcards can be used, for a full pattern description see [filepath package - path/filepath - Go Packages](https://golang.org/pkg/path/filepath/#Match)

# Example:

#collector_binaries_whitelist:

# - ā€œ/usr/bin/filebeatā€

# - ā€œ/opt/collectors/*ā€

# Example disable whitelisting:

# collector_binaries_whitelist:

# Default:

collector_binaries_accesslist:

# - ā€œ/opt/filebeat-8.3.2-linux-x86_64/filebeatā€

* ā€œ/usr/bin/filebeatā€
* ā€œ/usr/share/filebeat/bin/filebeatā€

# - ā€œ/usr/bin/filebeatā€

# - ā€œ/usr/bin/packetbeatā€

# - ā€œ/usr/bin/metricbeatā€

# - ā€œ/usr/bin/heartbeatā€

# - ā€œ/usr/bin/auditbeatā€

# - ā€œ/usr/bin/journalbeatā€

# - ā€œ/usr/share/filebeat/bin/filebeatā€

# - ā€œ/usr/share/packetbeat/bin/packetbeatā€

# - ā€œ/usr/share/metricbeat/bin/metricbeatā€

# - ā€œ/usr/share/heartbeat/bin/heartbeatā€

# - ā€œ/usr/share/auditbeat/bin/auditbeatā€

# - ā€œ/usr/share/filebeat/bin/filebeatā€

# - ā€œ/usr/bin/nxlogā€

# - ā€œ/opt/nxlog/bin/nxlogā€

#sudo graylog-sidecar -service install

#sudo systemctl enable graylog-sidecar

#sudo systemctl start graylog-sidecar
[/quote]
8 
# This file is an example configuration file highlighting only the most common

# options. The file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

# You can find the full configuration reference here:

# For more available modules and options, please see the sample

# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.

* type: filestream

# Unique ID among all inputs, an ID is required.

id: my-filestream-id

# Change to true to enable this input configuration.

enabled: false

# Paths that should be crawled and fetched. Glob based paths.

paths:
  * /var/log/audit/*.log
#- c:\programdata\elasticsearch\logs*

# Exclude lines. A list of regular expressions to match. It drops the lines that are

# matching any regular expression from the list.

#exclude_lines: [ā€˜^DBGā€™]

# Include lines. A list of regular expressions to match. It exports the lines that are

# matching any regular expression from the list.

#include_lines: [ā€˜^ERRā€™, ā€˜^WARNā€™]

# Exclude files. A list of regular expressions to match. Filebeat drops the files that

# are matching any regular expression from the list. By default, no files are dropped.

#prospector.scanner.exclude_files: [ā€˜.gz$ā€™]

# Optional additional fields. These fields can be freely picked

# to add additional information to the crawled log files for filtering

#fields:

# level: debug

# review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:

# Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

# Set to true to enable config reloading

reload.enabled: false

# Period on which files under path should be checked for changes

#reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

#name:

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: [ā€œservice-Xā€, ā€œweb-tierā€]

# Optional fields that you can specify to add additional information to the

# output.

#fields:

# env: staging

# ================================= Dashboards =================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here or by using the `setup` command.

#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the [artifacts.elastic.co](http://artifacts.elastic.co/)

# website.

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

# Kibana Host

# Scheme and port can be left out and will be set to the default (http and 5601)

# In case you specify and additional path, the scheme is required: http://localhost:5601/path

# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: ā€œlocalhost:5601ā€

# Kibana Space ID

# ID of the Kibana Space into which the dashboards should be loaded. By default,

# the Default Space will be used.

#space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------

#output.elasticsearch:

# Array of hosts to connect to.

#hosts: [ā€œlocalhost:9200ā€]

# Protocol - either `http` (default) or `https`.

#protocol: ā€œhttpsā€

# Authentication credentials - either API key or username/password.

#api_key: ā€œid:api_keyā€
#username: ā€œelasticā€
#password: ā€œchangemeā€

# ------------------------------ Logstash Output -------------------------------

output.logstash:

# The Logstash hosts

#hosts: [ā€œlocalhost:5044ā€]
removed host

# Optional SSL. By default is off.

# List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: [ā€œ/etc/pki/root/ca.pemā€]

# Certificate for SSL client authentication

#ssl.certificate: ā€œ/etc/pki/client/cert.pemā€

# Client Certificate Key

#ssl.key: ā€œ/etc/pki/client/cert.keyā€

# ================================= Processors =================================

processors:

* add_host_metadata:
when.not.contains.tags: forwarded
* add_cloud_metadata: ~
* add_docker_metadata: ~
* add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use [ā€œ*ā€]. Examples of other selectors are ā€œbeatā€,

# ā€œpublisherā€, ā€œserviceā€.

#logging.selectors: [ā€œ*ā€]

# ============================= X-Pack Monitoring ==============================

# Filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this

# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch

# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.

#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well.

# Note that the settings should point to your Elasticsearch *monitoring* cluster.

# Any setting that is not set is automatically inherited from the Elasticsearch

# output configuration, so if you have the Elasticsearch output configured such

# that it is pointing to your Elasticsearch monitoring cluster, you can simply

# uncomment the following line.

#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.

#instrumentation:
# Set to true to enable instrumentation of filebeat.
#enabled: false


# Environment in which filebeat is running on (eg: staging, production, etc.)
#environment: ""

# APM Server hosts to report instrumentation results to.
#hosts:
#  - http://localhost:8200

# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:

# Secret token for the APM Server(s).
#secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases

#migration.6_to_7.enabled: true This file is an example configuration file highlighting only the most common

# options. The file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

# You can find the full configuration reference here:

# For more available modules and options, please see the sample

# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.

* type: filestream

# Unique ID among all inputs, an ID is required.

id: my-filestream-id

# Change to true to enable this input configuration.

enabled: false

# Paths that should be crawled and fetched. Glob based paths.

paths:
  * /var/log/audit/*.log
#- c:\programdata\elasticsearch\logs*

# Exclude lines. A list of regular expressions to match. It drops the lines that are

# matching any regular expression from the list.

#exclude_lines: [ā€˜^DBGā€™]

# Include lines. A list of regular expressions to match. It exports the lines that are

# matching any regular expression from the list.

#include_lines: [ā€˜^ERRā€™, ā€˜^WARNā€™]

# Exclude files. A list of regular expressions to match. Filebeat drops the files that

# are matching any regular expression from the list. By default, no files are dropped.

#prospector.scanner.exclude_files: [ā€˜.gz$ā€™]

# Optional additional fields. These fields can be freely picked

# to add additional information to the crawled log files for filtering

#fields:

# level: debug

# review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:

# Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

# Set to true to enable config reloading

reload.enabled: false

# Period on which files under path should be checked for changes

#reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

#name:

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: [ā€œservice-Xā€, ā€œweb-tierā€]

# Optional fields that you can specify to add additional information to the

# output.

#fields:

# env: staging

# ================================= Dashboards =================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here or by using the `setup` command.

#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the [artifacts.elastic.co](http://artifacts.elastic.co/)

# website.

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

# Kibana Host

# Scheme and port can be left out and will be set to the default (http and 5601)

# In case you specify and additional path, the scheme is required: http://localhost:5601/path

# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: ā€œlocalhost:5601ā€

# Kibana Space ID

# ID of the Kibana Space into which the dashboards should be loaded. By default,

# the Default Space will be used.

#space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------

#output.elasticsearch:

# Array of hosts to connect to.

#hosts: [ā€œlocalhost:9200ā€]

# Protocol - either `http` (default) or `https`.

#protocol: ā€œhttpsā€

# Authentication credentials - either API key or username/password.

#api_key: ā€œid:api_keyā€
#username: ā€œelasticā€
#password: ā€œchangemeā€

# ------------------------------ Logstash Output -------------------------------

output.logstash:

# The Logstash hosts

#hosts: [ā€œlocalhost:5044ā€]
removed host

# Optional SSL. By default is off.

# List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: [ā€œ/etc/pki/root/ca.pemā€]

# Certificate for SSL client authentication

#ssl.certificate: ā€œ/etc/pki/client/cert.pemā€

# Client Certificate Key

#ssl.key: ā€œ/etc/pki/client/cert.keyā€

# ================================= Processors =================================

processors:

* add_host_metadata:
when.not.contains.tags: forwarded
* add_cloud_metadata: ~
* add_docker_metadata: ~
* add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use [ā€œ*ā€]. Examples of other selectors are ā€œbeatā€,

# ā€œpublisherā€, ā€œserviceā€.

#logging.selectors: [ā€œ*ā€]

# ============================= X-Pack Monitoring ==============================

# Filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this

# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch

# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.

#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well.

# Note that the settings should point to your Elasticsearch *monitoring* cluster.

# Any setting that is not set is automatically inherited from the Elasticsearch

# output configuration, so if you have the Elasticsearch output configured such

# that it is pointing to your Elasticsearch monitoring cluster, you can simply

# uncomment the following line.

#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.

#instrumentation:
# Set to true to enable instrumentation of filebeat.
#enabled: false


# Environment in which filebeat is running on (eg: staging, production, etc.)
#environment: ""

# APM Server hosts to report instrumentation results to.
#hosts:
#  - http://localhost:8200

# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:

# Secret token for the APM Server(s).
#secret_token:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases

#migration.6_to_7.enabled: true
9

Yml files require specific indentation and will not work without it - you can do rudimentary yml validation here but it doesnā€™t check the code.

It is still very difficult to read your posts, they are all over the place. Use the egrep tips in here to get file detail without comments (I donā€™t need all the code comments!) , then use the </> only on the code.

We all are donating spare time to answer questions in the forum, I donā€™t want to spend my spare time cleaning up posts! :smiley:

10

I am really sorry i am completely new to the stuff so tbh i wasnā€™t aware how this whole stuff works
can you please help me @tmacgbay

11

the only changes i did in sidecar.yml are

I added server url and server api token

collector_binaries_accesslist:
 # - "/opt/filebeat-8.3.2-linux-x86_64/filebeat"
   - "/usr/bin/filebeat"
   - "/usr/share/filebeat/bin/filebeat"

in the filebeat.yml

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/audit/*.log

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false


setup.template.settings:
  index.number_of_shards: 1
12

Hello @kalyan

First there are couple things I wanted to mention,
I corrected all your post/s so its readable,:+1: ,

Second, I moved this post to Graylog Central because I believe this is a issue not a development. I have also read that you new and require help, please keep in mind that how/what you post will depended on the help you get.

Third, this may also help specially when posting configuration files.

root#  cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

This will make it neat :+1:

Example here is my sidecar configuration

root@keycloak:/etc/graylog/sidecar# cat /etc/graylog/sidecar/sidecar.yml | egrep -v "^\s*(#|$)"
server_url: "http://10.10.10.10:9000/api/"
server_api_token: "1i1like1hamburgers1and1milkshakes1withmycoffeee"
node_id: "file:/etc/graylog/sidecar/node-id"
node_name: "keycloak101"
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
    - "/var/log/"
collector_binaries_whitelist:
  - "/usr/share/filebeat/bin/filebeat"
  - "/usr/bin/packetbeat"
root@keycloak:/etc/graylog/sidecar#

That is basic configuration.

1 Like
13

Next: here is my FileBeat configuration, remind you this is just basic stuff and you need to replace the IP Address with your Graylog server, that is if this a a remote device sending log to Graylog, this should be enough to get you going.

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
  type: log
output.logstash:
   hosts: ["18.8.8.8:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

This is what it looks like on the Web UI :smiley:

image
image1039Ɨ855 41.4 KB

Also take note That Iā€™m not configuring FileBeat configuration file, Only in the Web UI :point_up:
Hope that helps

1 Like
14

Thanks a lot i will try to modify N see if it works

1 Like
16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.