Multiple inputs for single sidecar collector

[oletts]

Hi All,

I currently have a templated collector sidecar config to collect system logs that is applied to all our servers. I now have a server where I want to get both the system logs and the web logs from and via two different inputs because I want to use different extractors for each type of log.

I found this previous post that says seems to say it is possible to have different inputs from the same server but I was wondering if they can be to different collector configurations so that we can extract the logs differently? Also if anyone has an example config file that would allow for this?

Thanks in advance for any replies.

[Totally_Not_A_Robot (Tess)]

Isn’t this just a simple case of defining a new collector configuration? This defines a new tag and the desired input sources (files, eventlog) and the desired output targets (BEATS receiver) for the config. You can then simply add the tag to the host’s Sidecar config file. And presto! You’re done!

[jan (Jan Doberstein)]

with the collector sidecar 0.1.7 it is not possible - with the upcoming 3.0 release this will work.

[oletts]

I was getting slightly mixed up between graylog inputs/outputs and beats inputs/outputs.

I’ve tried defining different inputs using nxlogs rather than beats but that doesn’t appear to be sending any logs at all now.

In order to get multiple inputs do I need to use nxlogs or beats?

[oletts]

Does this mean that the collector sidecar can’t have multiple configurations at the same time?

[v_2nas (Nav)]

Yes, you can have different collectors or you can do the both in the same config.
For simplicity, you can create another collector filebeat and, configure it to read the web logs and send it to graylog.

I don’t have a sample, but search on the forums. I am sure you will get some examples.

[Totally_Not_A_Robot (Tess)]

@oletts

Does this mean that the collector sidecar can’t have multiple configurations at the same time?

It depends on what you specifically mean by “configuration”. Because as @v_2nas says: one Sidecar can be subscribed to many Graylog tags, with each tag providing specific FileBeat and WinLogBeat configurations.

[oletts]

I don’t think I’ve been very precise with my language so far so I’m going to write out the components I have at the moment and then hopefully you’ll be able to see where I’m getting confused:

• 1 Web Server with a single sidecar

• 2 Graylog Server inputs each with their own port associated (1 for system logs, 1 for web logs)

  • There are two separate inputs so that each input can have its own extractor definition for differing analysis

• 2 Collector Sidecar Configurations each with their own tag associated (1 for the system logs, 1 for web logs)

  • Each configuration has an output linked with the relevant graylog input above
  • Each configuration also has an input pointing to the logs location of the web server

I tried doing this with two beats inputs but I don’t get any logs appearing from the input for the web logs. All the logs for both system and web appear under the system logs input, which means I can’t use a separate extractor for the web logs.

I may have something incorrectly configured but I’m fairly certain everything is fine.

I’ve also tried using NXLog instead of beats to see if that works but now I don’t get the web logs at all even on the system graylog input stream.

Thanks for the help with this and I apologise if I’m not using the correct names for everything.

[Totally_Not_A_Robot (Tess)]

The use-case you’re describing really should work. It is definitely possible to define a config that does what you want. So now we’re down to troubleshooting

[oletts]

Hmmm, I wonder what I’m doing wrong then. Would the version of the sidecar make a big difference?

Are there any sidecar logs which would indicate what the issue could be?

Just in case that isn’t the issue though, here are some sanitised config files which might shine some light on the situation:

collector_sidecar.yml

server_url: http://my_graylog_server.domain.com:9000/api/
update_interval: 5
tls_skip_verify: false
send_status: true
list_log_files:
node_id: webserver.domain.com
collector_id: file:/etc/graylog/collector-sidecar/collector-id
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
  - system
  - web
backends:
    - name: nxlog
      enabled: true
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf

nxlog.conf

define ROOT /usr/bin

<Extension gelf>
  Module xm_gelf
</Extension>



User nxlog
Group nxlog


Moduledir /usr/libexec/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid

define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
LogFile %LOGFILE%
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%LOGFILE%', 7);
     </Schedule>
</Extension>




<Input INPUT_GUID1>
        Module im_file
        File '/var/log/web/web_*.log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
</Input>
<Input INPUT_GUID2>
        Module im_file
        File '/var/log/web/different_logs/different.log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
</Input>
<Input INPUT_GUID3>
        Module im_file
        File '/var/log/messages'
        PollInterval 10
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
</Input>




<Output OUTPUT_GUID1>
        Module om_udp
        Host <graylog_server_IP>
        Port 5050
        OutputType  GELF
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = 'collector_GUID';
        Exec $collector_node_id = 'webserver.domain.com';
        Exec $Hostname = hostname_fqdn();
</Output>
<Output OUTPUT_GUID2>
        Module om_udp
        Host <graylog_server_IP>
        Port 5045
        OutputType  GELF
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = 'collector_GUID';
        Exec $collector_node_id = 'webserver.domain.com';
        Exec $Hostname = hostname_fqdn();
</Output>



<Route route-2>
  Path INPUT_GUID1 => OUTPUT_GUID1
</Route>
<Route route-3>
  Path INPUT_GUID2 => OUTPUT_GUID1
</Route>
<Route route-5>
  Path INPUT_GUID3 => OUTPUT_GUID2
</Route>

[oletts]

Just as a side note, I found the reference documentation that says that filebeat can’t send to multiple outputs which is why we are using NXlog now:

Configure the output | Filebeat Reference [8.11] | Elastic

You configure Filebeat to write to a specific output by setting options in the output section of the filebeat.yml config file. Only a single output may be defined.

[Totally_Not_A_Robot (Tess)]

EDIT:
Well, it could be that each tag starts its own FileBeat collector on a host. So if a host has three tags, it could be running three FileBeats…

@Jan, am I on the right track?

[jan (Jan Doberstein)]

@Jan, am I on the right track?

@oletts right filebeat can have currently onyl one output. The upcoming release will work around this.

With the current stable release you will habe all files send to one output

[oletts]

@jan if I want to be able to send to multiple outputs so that I can do different extractors for each set of logs I should use nxlog instead of filebeat?