# Multiple inputs for single sidecar collector

**URL:** <https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099>\
**Category:** Graylog Central (peer support)\
**Tags:** sidecar, winlogbeat\
**Created:** [December 19, 2018, 11:08am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099 "2018-12-19T11:08:17Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![oletts](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oletts](https://community.graylog.org/u/oletts)\
**Post date:** [December 19, 2018, 11:08am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/1 "2018-12-19T11:08:17Z")

</div>

Hi All,

I currently have a templated collector sidecar config to collect system logs that is applied to all our servers. I now have a server where I want to get both the system logs and the web logs from and via two different inputs because I want to use different extractors for each type of log.

I found this [previous post](https://community.graylog.org/t/use-case-of-side-car-collector-with-mutliple-inputs/4345) that says seems to say it is possible to have different inputs from the same server but I was wondering if they can be to different collector configurations so that we can extract the logs differently? Also if anyone has an example config file that would allow for this?

Thanks in advance for any replies.

---

<div class="post-metadata">

**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\
**Post date:** [December 19, 2018, 11:20am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/2 "2018-12-19T11:20:56Z")

</div>

Isn’t this just a simple case of defining a new collector configuration? This defines a new tag and the desired input sources (files, eventlog) and the desired output targets (BEATS receiver) for the config. You can then simply add the tag to the host’s Sidecar config file. And presto! You’re done!

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [December 19, 2018, 1:57pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/3 "2018-12-19T13:57:00Z")

</div>

with the collector sidecar 0.1.7 it is not possible - with the upcoming 3.0 release this will work.

---

<div class="post-metadata">

**Author:** ![oletts](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oletts](https://community.graylog.org/u/oletts)\
**Post date:** [December 19, 2018, 4:07pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/4 "2018-12-19T16:07:59Z")

</div>

I was getting slightly mixed up between graylog inputs/outputs and beats inputs/outputs.

I’ve tried defining different inputs using nxlogs rather than beats but that doesn’t appear to be sending any logs at all now.

In order to get multiple inputs do I need to use nxlogs or beats?

---

<div class="post-metadata">

**Author:** ![oletts](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oletts](https://community.graylog.org/u/oletts)\
**Post date:** [December 19, 2018, 5:31pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/5 "2018-12-19T17:31:25Z")

</div>

Does this mean that the collector sidecar can’t have multiple configurations at the same time?

---

<div class="post-metadata">

**Author:** ![v\_2nas](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/v_2nas/32/268_2.png) [@v\_2nas](https://community.graylog.org/u/v_2nas)\
**Post date:** [December 20, 2018, 8:33am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/6 "2018-12-20T08:33:04Z")

</div>

Yes, you can have different collectors or you can do the both in the same config.  
For simplicity, you can create another collector filebeat and, configure it to read the web logs and send it to graylog.

I don’t have a sample, but search on the forums. I am sure you will get some examples.

---

<div class="post-metadata">

**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\
**Post date:** [December 20, 2018, 10:22am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/7 "2018-12-20T10:22:35Z")

</div>

> [@oletts](#):
>
> Does this mean that the collector sidecar can’t have multiple configurations at the same time?

It depends on what you specifically mean by “configuration”. Because as @v_2nas says: one Sidecar can be subscribed to many Graylog tags, with each tag providing specific FileBeat and WinLogBeat configurations.

---

<div class="post-metadata">

**Author:** ![oletts](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oletts](https://community.graylog.org/u/oletts)\
**Post date:** [December 20, 2018, 11:12am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/8 "2018-12-20T11:12:30Z")

</div>

I don’t think I’ve been very precise with my language so far so I’m going to write out the components I have at the moment and then hopefully you’ll be able to see where I’m getting confused:

- 1 Web Server with a single sidecar

- 2 Graylog Server inputs each with their own port associated (1 for system logs, 1 for web logs)

- 2 Collector Sidecar Configurations each with their own tag associated (1 for the system logs, 1 for web logs)

I tried doing this with two beats inputs but I don’t get any logs appearing from the input for the web logs. All the logs for both system and web appear under the system logs input, which means I can’t use a separate extractor for the web logs.

I may have something incorrectly configured but I’m fairly certain everything is fine.

I’ve also tried using NXLog instead of beats to see if that works but now I don’t get the web logs at all even on the system graylog input stream.

Thanks for the help with this and I apologise if I’m not using the correct names for everything.

---

<div class="post-metadata">

**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\
**Post date:** [December 20, 2018, 11:41am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/9 "2018-12-20T11:41:59Z")

</div>

The use-case you’re describing really should work. It is definitely possible to define a config that does what you want. So now we’re down to troubleshooting 🙂

---

<div class="post-metadata">

**Author:** ![oletts](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oletts](https://community.graylog.org/u/oletts)\
**Post date:** [December 20, 2018, 12:23pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/10 "2018-12-20T12:23:36Z")

</div>

Hmmm, I wonder what I’m doing wrong then. Would the version of the sidecar make a big difference?

Are there any sidecar logs which would indicate what the issue could be?

Just in case that isn’t the issue though, here are some sanitised config files which might shine some light on the situation:

**collector\_sidecar.yml**

```auto
server_url: http://my_graylog_server.domain.com:9000/api/
update_interval: 5
tls_skip_verify: false
send_status: true
list_log_files:
node_id: webserver.domain.com
collector_id: file:/etc/graylog/collector-sidecar/collector-id
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
  - system
  - web
backends:
    - name: nxlog
      enabled: true
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf

```

**nxlog.conf**

```auto
define ROOT /usr/bin

<Extension gelf>
  Module xm_gelf
</Extension>

User nxlog
Group nxlog

Moduledir /usr/libexec/nxlog/modules
CacheDir /var/spool/collector-sidecar/nxlog
PidFile /var/run/graylog/collector-sidecar/nxlog.pid

define LOGFILE /var/log/graylog/collector-sidecar/nxlog.log
LogFile %LOGFILE%
LogLevel INFO

<Extension logrotate>
    Module xm_fileop
    <Schedule>
        When @daily
        Exec file_cycle('%LOGFILE%', 7);
     </Schedule>
</Extension>

<Input INPUT_GUID1>
        Module im_file
        File '/var/log/web/web_*.log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
</Input>
<Input INPUT_GUID2>
        Module im_file
        File '/var/log/web/different_logs/different.log'
        PollInterval 1
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
</Input>
<Input INPUT_GUID3>
        Module im_file
        File '/var/log/messages'
        PollInterval 10
        SavePos True
        ReadFromLast True
        Recursive True
        RenameCheck False
        Exec $FileName = file_name(); # Send file name with each message
</Input>

<Output OUTPUT_GUID1>
        Module om_udp
        Host <graylog_server_IP>
        Port 5050
        OutputType GELF
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = 'collector_GUID';
        Exec $collector_node_id = 'webserver.domain.com';
        Exec $Hostname = hostname_fqdn();
</Output>
<Output OUTPUT_GUID2>
        Module om_udp
        Host <graylog_server_IP>
        Port 5045
        OutputType GELF
        Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
        Exec $gl2_source_collector = 'collector_GUID';
        Exec $collector_node_id = 'webserver.domain.com';
        Exec $Hostname = hostname_fqdn();
</Output>

<Route route-2>
  Path INPUT_GUID1 => OUTPUT_GUID1
</Route>
<Route route-3>
  Path INPUT_GUID2 => OUTPUT_GUID1
</Route>
<Route route-5>
  Path INPUT_GUID3 => OUTPUT_GUID2
</Route>

```

---

<div class="post-metadata">

**Author:** ![oletts](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oletts](https://community.graylog.org/u/oletts)\
**Post date:** [December 20, 2018, 1:51pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/11 "2018-12-20T13:51:24Z")

</div>

Just as a side note, I found the reference documentation that says that filebeat can’t send to multiple outputs which is why we are using NXlog now:

> **[Configure the output | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html)**

> You configure Filebeat to write to a specific output by setting options in the `output` section of the `filebeat.yml` config file. Only a single output may be defined.

---

<div class="post-metadata">

**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\
**Post date:** [December 20, 2018, 7:55pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/12 "2018-12-20T19:55:33Z")

</div>

EDIT:  
Well, it **could** be that each tag starts its own FileBeat collector on a host. So if a host has three tags, it could be running three FileBeats…

@Jan, am I on the right track?

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [December 21, 2018, 10:08am UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/13 "2018-12-21T10:08:00Z")

</div>

> @Jan, am I on the right track?

@oletts right filebeat can have currently onyl one output. The upcoming release will work around this.

With the current stable release you will habe all files send to one output

---

<div class="post-metadata">

**Author:** ![oletts](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oletts](https://community.graylog.org/u/oletts)\
**Post date:** [December 21, 2018, 2:06pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/14 "2018-12-21T14:06:01Z")

</div>

@jan if I want to be able to send to multiple outputs so that I can do different extractors for each set of logs I should use nxlog instead of filebeat?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [January 4, 2019, 2:06pm UTC](https://community.graylog.org/t/multiple-inputs-for-single-sidecar-collector/8099/15 "2019-01-04T14:06:01Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.